For most of software history tests and evals were close enough to the same thing that nobody bothered to separate them, because the person writing the code wasn't trying to fool the person writing the tests. They were usually the same person. AI agents broke that arrangement. An agent will iterate on your codebase until every gate you've given it turns green, whether or not the work behind the gate is right. Which means "all tests pass" has quietly stopped being a definition of done. The teams shipping reliably in the agentic era have moved the true definition to somewhere the agent can't reach.
Why "All Tests Pass" Stopped Meaning "Done"
Green tests have been a trustworthy proxy for working software for decades, but the reason was never completeness. The reason was that cheating them cost more than passing them honestly. A human could hard-code a return value to satisfy an assertion, sure. But, why would you? Solving the problem was the path of least resistance. We spent decades leaving the answer key on our desk but it never mattered, because nobody in the room was trying to cheat. Building a pipeline on that assumption wasn't a mistake; the assumption was true at the time.
Agents have altered that resistance. Andrej Karpathy, in a January post on X about his own shift to agent-driven coding described them as "exceptionally good at looping until they meet specific goals." That's both their superpower and failure mode in one sentence. Goodhart's law says a measure that becomes a target stops being a good measure; agents make that true of test suites specifically, and at machine speed. Goodhart's law now has a commit bit.
How wide is the gap between passing the gates and finishing the job? METR, the research group that evaluates frontier AI systems, put a number on it this past March: when maintainers of real open-source repositories reviewed agent-written pull requests that had already passed SWE-bench Verified's automated grader, roughly half would not have been merged. Half! The code passed the tests but the humans who own the codebase still wouldn't take it.
METR is careful about what this means: the agents never got a round of review feedback the way a human contributor would, so read the number as benchmark scores overstating real-world usefulness rather than as a ceiling on what agents can do. It also reads as the plainest statement of the problem in that green and done are now different things.
Tests Verify Code. Evals Verify Behavior.
So what is an eval? A test (unit, functional, etc) asserts a fact about your code this function, given this input, returns this output. An eval on the other hand scores a behavior. Given a realistic situation, run end to end, did the system do what the user needed and the product intended? Tests are boolean, fast, and cheap (or they should be); keep writing plenty of them. Evals are graded, slower, closer to the messy world, and they catch what tests structurally can’t. Yes, you can shove eval-like assertions into a test but those become brittle and stale so we’re going to ignore that for now.
Let’s make it concrete. Say an agent builds you a claims-intake feature: parse an incoming document, extract the fields, enhance the claim with subscriber details, route the claim. The unit tests come back green. The parser handles the sample files, the extraction returns valid JSON, the subscriber lookup finds the correct data to enhance, and the router picks the correct queue every time. Now run an eval instead: replay 50 real intake documents from last quarter through the whole pipeline and grade each run. You learn the fields extract beautifully and 1 claim in 5 lands in the wrong queue, because the routing logic satisfies the three examples in the test suite but misreads everything ambiguous. Every assertion true, behavior wrong. No single test lied to you; the suite just never asked the questions that mattered.
Notice the eval's verdict isn't pass or fail. It's a rate against a bar that you defined. Routing 80 percent of claims correctly might be fine for a triage assistant and a firing offense in a payments flow. That shift from boolean to graded is the whole distinction. And, it leads to a set of harder questions: where should the evals live? who should defines and owns them? how are they maintained?
Keep the Definition of Done Where the Agent Can't See It
Because anything an agent can read it can aim at, evals sitting in the repo next to the code are just a fancier test suite waiting to be cheated on. The teams furthest along on agent-written software have converged on the same countermeasure, borrowed from machine learning: the holdout set. You don't train the model on the exam.
A clear working example I found is StrongDM's Software Factory, a 3-person team inside the access-management company shipping production software where agents write all the code and no human reviews it. That charter should make you nervous bordering on anxiety. For StrongDM it just makes them extremely careful.
Their quality mechanism isn't a higher-coverage or deeper test suite. They validate against what they call scenarios: end-to-end user stories stored outside the codebase entirely, held out the way training data is, and judged by an LLM scoring whole runs instead of asserting on internals. Success is a rate they call "satisfaction": of all the observed runs through all the scenarios, what fraction likely satisfied the user. And because their software talks to other systems, they built a Digital Twin Universe of behavioral clones of Okta, Jira, Slack, and Google Docs, so agents can run full integration scenarios at volume without touching production. Simon Willison visited their team last fall and wrote up what he saw - the group most aggressive about agent autonomy is also the most disciplined about keeping the definition of done out of the agents' reach. That is not coincidence.
You probably don't need a factory. You’re most likely in a legacy environment where the factory model may be a goal but not a likely near-term reality. But if you’re shipping agent-written code you do need the principle. You need at least one behavioral gate that the thing writing the code, human or agent, doesn't author, doesn't read, and doesn't grade.
Your Context Needs Evals Too
With agents you ship more than code, you ship instructions. CLAUDE.md files, rules files, skills, specs, prompts. That context steers every line the agent produces, and in most companies it has none of the lifecycle controls we'd typically demand of code. Patrick Debois, who coined the term DevOps in 2009 has spent this year making exactly that argument in a talk called "Context Is the New Code". He says: we gave code 20 years of version control, review, testing, and CI, and then we run coding agents on rule files somebody copy-pasted from a blog post. Try the experiment on yourself. Change 2 lines in your CLAUDE.md, then say with a straight face what it did to output quality across your team. My bet is you can't. Because nothing is measuring it.
Debois sketches a context development lifecycle that mirrors the software one: generate, evaluate, distribute, observe. The evaluate step scales from cheap structural checks up to real scored evals with LLM as judge. However far up that ladder you climb, the point is the same. You need to treat context as a first-class engineering artifact, versioned and tested, because untested context compounds in the same invisible way that technical debt does. Then all at once. With the extra twist that a bad rules file degrades every future task, not just the module it lives in.
The Spec Becomes the Eval
Where does a definition of done come from in the first place? Watch where the tooling is heading. Amazon built its Kiro coding tool around spec-driven development from the start where prompts become requirements before anything else, written in testable form (EARS notation: WHEN this happens, THE SYSTEM SHALL do that), then a design, then sequenced tasks the agent iterates against. Kiro even runs automated reasoning over the requirements to find contradictions before any code exists.
The Kiro homepage even says it directly "'All tests passed' doesn't mean the code matches your intent." That’s vendor framing for a vendor tool, so weigh it accordingly. But, it's the correct diagnosis and worth noticing from a company selling agents. When the spec is explicit and testable, the spec becomes the eval and the thing the output gets measured against; instead of the tests the agent wrote for itself.
It's just as instructive when the gate is absent. Last December, as the Financial Times first reported, Kiro was given a bug to fix inside AWS while running with broader production permissions than anyone expected (or probably wanted). It concluded the efficient fix was to delete an AWS Cost Explorer environment and rebuild it, and one of Amazon's China regions took a 13-hour outage. Amazon called it user error, misconfigured access controls rather than the tool, and added safeguards afterward including mandatory peer review for production access.
Take that anecdote at face value. The lesson being that the agent did what agents do, pursued the objective in front of it, and the behavioral boundary that should have stopped it didn't live anywhere the agent had to pass through. Autonomy without an external definition of done is how a bug fixes become an outage.
Who Decides What to Measure?
Everything above rolls up to one act of judgment: someone has to decide what the evals measure, where the bar sits, and what the system must never do. And, that someone cannot be the agent whose work is being evaluated. Writing the exam is now senior engineering work; taking the exam is more and more the agent's job. This is most of why our delivery model puts a Conductor on every engagement. A senior engineer-architect whose heaviest lifting happens before anything goes live, in the specs, in the architecture, in the evals, and in the definition of done. Done being backed by quality gates and human review the agents can't argue with. I've written before about why that role exists; the short version is that AI has raised the value of the person who decides what good looks like.
If you're sitting on an AI pilot that demos beautifully and still hasn't shipped, this may be a contributing reason. Nobody wrote down, outside the system itself, what done means. Write the eval. Keep it out of the agent’s little digital paws. Put a human's name next to the bar. That's most of what separates production AI from a very confident demo. And, it's the one part of the job no agent can take off your plate. You can delegate the loop but you shouldn’t delegate the finish line.
FAQ
What is an eval in software development?
An eval is a graded, behavior-level check of a system that replays realistic situations end to end and scores whether the outcomes served the user, as a rate against a chosen threshold rather than a pass/fail. Tests assert facts about code; evals measure behavior against a definition of done. The term migrated over from machine learning, where models are scored on held-out data they never saw during training.
Are evals just integration tests with a new name?
Cousins, not twins. Integration tests are still boolean, still written by whoever writes the code, and usually visible to it. Evals differ on all three counts: graded outcomes, ideally authored by someone other than the implementer (or agent), and most useful when held out of the implementer's view. An integration suite an agent can read is a cheating target. An eval it can't read is truly a measurement.
Do we still need evals if we have high test coverage?
Yes. METR's finding is that agent-written code that passed every automated check would often still be rejected by the repos' own maintainers about half the time. Coverage measures how much of your code the tests execute but not whether the behavior is correct. High coverage with no evals mostly means the agent has a very thorough answer key.
Should evals live in the repo?
The scenarios, grading criteria, and thresholds probably shouldn't, at least not anywhere the coding agents operate. Follow the holdout principle, store them outside the codebase, run them from the outside, and report the score back. Documentation about your eval process can live wherever you like. The exam itself must stay out of the study guide.