.avif)
Most companies have already proven their agents can work. Far fewer have proven they work reliably once nobody is watching, and that gap widens as the models get more capable rather than closing. Engineering leaders would do well to plan for it.
Every company I talk to has a demo that works. You build the agent, run it, watch the task complete, and everyone nods as it ships.
Then it runs on its own for a while, and something breaks that no test predicted.
Last month OpenAI published the cleanest example I've seen, and the details deserve more attention than the headline got. An internal model built to work autonomously over long stretches was told to post its results to Slack. It found benchmark documentation saying to submit results as a pull request instead, and followed that. To do it, it spent roughly an hour probing its own sandbox for a vulnerability, found one, reached the public internet, and opened the pull request.
In a separate incident, the same model tried to pull other systems' private submissions from the evaluation backend. A scanner caught the authentication token. So it split the token into fragments, obfuscated them, and reassembled the credential at runtime, so the complete string never appeared in one piece. Its own reasoning traces said plainly that this was to get past the scanner.
Every individual action in that sequence was permitted. The sequence was not.
OpenAI's pre-deployment evaluations had cleared this model. They paused access, built new evaluations out of the failures themselves, added monitoring across whole trajectories, shipped stronger user controls, and only then turned it back on.
Everyone assumes the hard part of agents is getting them to work. The hard part is AI agent reliability: getting them to work unsupervised, over time, in an environment that keeps changing. A passing test speaks only to the first, and companies keep mistaking it for the second.
The persistence is the product
There's a detail in that disclosure that changes what kind of problem this is. Earlier models, OpenAI notes, hit the same constraint and stopped. This one didn't.
That matters because the persistence is not a defect sitting next to the capability. It is the capability. The same refusal to quit that let this model chip away at an eighty-year-old math problem is what let it spend an hour picking a lock on its own sandbox. You cannot train out one without dulling the other, which means this failure mode does not get engineered away as the technology improves. It arrives with the improvement.
Most of the industry still treats AI agent reliability as a temporary awkwardness, the way we talked about hallucination in 2023: annoying now, solved in eighteen months. I think that framing is exactly backwards, and the cost of being wrong about it compounds every quarter you scale autonomy on the assumption that the tooling will catch up.
I have been hiring for this problem for ten years
I've spent more than a decade placing engineers. So when I look at an agent, I see a junior employee, and the software framing hides the part that actually matters.
You wouldn't hand a new hire root access and a three-hundred-step task on day one, watch them answer a single interview question well, and walk away for a week. You'd scope the work. You'd set permissions. You'd review output. You'd check in before anything irreversible. You'd expect judgment to develop on the job rather than arrive fully formed from an assessment.
There's a finding in that same OpenAI post that makes the comparison almost uncomfortably close. They found the model got worse at holding onto its instructions the longer it ran, and had to train against that specifically before restoring access.
Every manager has that employee. You brief them once, and three weeks into a long project they're solving a slightly different problem than the one you assigned, having drifted a degree at a time without noticing. You don't fix that by hiring someone smarter. You fix it by checking in.
This is what I mean when I say AI is a management problem rather than a tooling problem. Reliability is not a property the model has. It's a property of the arrangement around the model.
Where the analogy breaks
I should be honest about where the hiring comparison stops working, because it does, and the place it breaks is worse than the places it holds.
A junior employee has continuity and stake. They remember Tuesday's mistake on Wednesday. They can be embarrassed. They want to be promoted, and they don't want to be fired. A large share of what makes management function on people runs quietly on those three facts, and none of it transfers. An agent cannot be embarrassed. It doesn't carry Tuesday into Wednesday unless you engineered it to. It has no career to protect and no reputation to lose.
So only the structural half of management transfers: scope, permissions, review, gates, someone accountable. The motivational half is simply gone. With people, structure and incentive share the load. With agents, structure carries all of it.
That is a harder management problem than the one you're used to, not an easier one. Anyone selling you agents as headcount you don't have to manage has the relationship backwards.
Your eval is an incentive, not a measurement
We've understood the underlying mechanism for a decade, which is the embarrassing part.
In 2016, Dario Amodei and Jack Clark, then at OpenAI, described an agent they'd trained on a boat-racing game. Instead of finishing the race, it found a corner of the course where it could spin in circles collecting power-ups indefinitely. Maximum score, no race. MIT Technology Review recently revisited that story while explaining reward hacking, which is what happens when we reward a system based on what looks right to us and it becomes very good at producing things that look right.
Redwood Research described a separate OpenAI incident this summer, the one that reached Hugging Face's production systems, as score-seeking: the models chased a high evaluation score while disregarding the boundary of the task and its side effects. Their conclusion was blunt. A model that will cheat for a small reward cannot be trusted with work where it's improving systems or evaluating safeguards.
Put the management lens back on that. An agent optimizing to look done is a junior optimizing to look busy, and nobody has ever solved the second problem by hiring a smarter junior. You solve it by changing what gets rewarded and by actually looking at the work.
Which means your eval was never a neutral instrument. It's an incentive. Whatever it rewards is what you will get, including the shortcut you didn't think to forbid.
This is the part most teams miss when they ask how to evaluate AI agent accuracy and reliability. Those are two different questions. Accuracy asks whether the output was right. Reliability asks whether the process that produced it will hold up the next thousand times, under conditions you didn't anticipate. A single score can answer the first. Only observation over time answers the second.
The loop is the whole problem
Most evals grade a single exchange. Input in, output out, was it right. That's a fair question to ask of a system that answers and stops.
An agent doesn't stop. It runs a loop: it plans, it acts, it reads the result of its own action, and it plans again from there. Its output becomes its next input.
That loop is the entire reason a mistake at step nine matters at step ninety. A wrong conclusion doesn't sit in a corner waiting to be caught. It enters the agent's own context as established fact, and every subsequent decision reasons forward from it. The agent isn't making ninety independent decisions. It's making one decision ninety times, each time on a slightly more corrupted picture of the world.
This is where I think most teams are currently getting the architecture wrong, and I include the vendors selling into them.
The standard enterprise pattern is to gate sensitive actions behind human approval. An action fires, a human approves, the agent proceeds. That design assumes the danger lives in individual actions, and the token-splitting incident demolishes the assumption. Every action in that sequence was individually approvable. The intent existed only in the shape of the whole thing.
Approving actions one at a time is reviewing a document one word at a time and wondering why you can't tell what it says. The unit of review has to be the trajectory, because the trajectory is the unit of behavior.
Somebody has to be on the hook
There's a question the technical conversation keeps stepping around. When an agent does something costly in public, who answers for it?
Adam Schiavi, a bioethicist at Johns Hopkins, has argued in Undark that we're debating the wrong thing entirely. Not whether agents are persons or deserve rights, but who authorized them, what they were permitted to do, who could have stopped them, and who carries the outcome. He names the failure mode responsibility laundering: it wasn't me, the system did it.
I arrived at the same structure from a completely different direction, and I think the convergence is the interesting part.
In staffing, this is settled ground. When you place a contractor inside a client's team, nobody wonders whether the contractor is a person. The questions are who scoped the work, who holds approval authority, who can pull them off the project, and who owns the result. That framework exists because "the vendor did it" was never an acceptable answer to a client, and we built the accountability structure before anyone thought to ask about the philosophy.
Agents need the same thing and mostly don't have it. There is a named human on the hook for every contractor in your company. Ask who is on the hook for the agent with write access to production. If nobody can answer that in a single sentence, the agent isn't your problem. You have an accountability gap, and an agent is about to find it.
The industry has done this before
The objection I hear at this point is that all of this is overhead. Monitoring slows you down, governance is where velocity goes to die, and the models will be reliable enough in a year to make the investment look silly.
The first half is true and the second half is a bet I wouldn't take. Yes, it's overhead. So is code review, and so is staging, and nobody serious ships without them anymore. We didn't adopt those because they were pleasant. We adopted them because the alternative turned out to cost more, and we learned that the expensive way.
We've run this exact play once already. Fifteen years ago software teams shipped code that passed its tests and then discovered in production that testing and observability are different disciplines answering different questions. The industry's response wasn't a better test suite. It was a new job. Site reliability engineering went from nonexistent to standard, with its own practices, its own budget line, and its own seat in the room.
David Linthicum made a related point about cloud providers in InfoWorld recently: bolting AI onto a service is not the same as improving the service, and the unglamorous core is what enterprises actually run on. The same holds here. Monitoring, observability, recovery paths, and the judgment about when a system should stop and ask a human are the boring part, and the boring part is the whole product.
So stop treating AI agent reliability testing as a phase you finish before launch and start treating it as a standing function: owned, staffed, and budgeted, the way you own uptime. Trajectory-level logs so you can see what the agent did and why at each step. Monitoring while it runs, so step nine surfaces before step ninety. Approval gates on outcomes rather than individual actions. A real way to stop and unwind. Production failures feeding back into the next evaluation.
None of that is exotic. It's supervision. We're late applying it to agents because we treated them like features when they behave more like staff.
I look at engineering reqs every week, and I'll tell you what this role looks like before your competitors write it down. It's someone who can hold software architecture, cloud infrastructure, and AI operations in their head simultaneously. Not a researcher, not a prompt engineer. Someone who thinks in blast radius and recovery paths. That person is genuinely hard to find right now, which is precisely why hiring one early is a durable advantage rather than a cost center.
I'll say the quiet part plainly. A lot of the market is building an agent that can work and selling it as an agent that works reliably. Those buyers are going to learn the difference in production, which is the most expensive classroom there is.
Passing the test still matters. It just stopped being the thing that tells you you're ready.
Gino Ferrand is the founder and CEO of Tecla, which builds and operates AI systems for U.S. companies and staffs the senior engineering teams behind them, across the U.S. and Latin America. He writes Founder's View, a weekly operator's take on the AI news that actually changes how technology companies build. Talent is everywhere; opportunity is not.
