Why Ground Truth Testing Fails for AI Agents
Ground truth testing is the default approach for evaluating AI agents. Write the expected output. Run the agent. Compare. If the output matches, the test passes.
It works for deterministic cases. "What is the capital of France" has one right answer. "Calculate the sum of column B" has one right answer.
It stops working the moment a user asks something that doesn't have one right answer. And that happens all the time.
The clean input problem
Every ground truth test starts with someone writing a test case. That person is usually the engineer who built the agent. They write the inputs they expect users to send.
The issue is straightforward: the person who built the agent writes the tests. They test what they designed for. They never test what they didn't design for because it didn't occur to them.
An engineer building a CRM copilot writes test cases like "how many deals closed last quarter" and "show me open opportunities over $50K" and "what is our average deal size."
Clean inputs. Clear answers. The agent passes every one.
Here is what a real user asks:
"Show me the big deals." Big is not defined. "How many deals closed last quarter. Actually make that this quarter. No wait, show me both." Three intent changes in one message. "Why did revenue drop." Revenue didn't drop.
There is no ground truth for these because there is no single correct answer. "Show me the big deals" could mean deals over $10K or $50K or $100K depending on who is asking. A good agent clarifies. A bad agent guesses. Ground truth testing can't distinguish between the two because the test case was never written.
The multi-turn problem
Ground truth testing evaluates single input-output pairs. Question in, answer out, compare.
AI agents don't work that way. They have conversations. Turn one sets context. Turn two builds on it. Turn three might contradict turn two. Turn four references something from turn one.
We tested 150+ agents and the most common failure isn't a bad answer on turn one. It's what happens at turn four when the user changes intent. The agent keeps responding to the original intent because it doesn't track accumulated context across the conversation.
Ground truth testing can't catch this because it doesn't test turns in sequence. Each test case is independent. There is no test case that says "ask about pricing, then ask about integrations, then go back to pricing with a new constraint, and check if the agent remembers the original pricing conversation."
You could write that test case for that one scenario. But covering the combinations of intent changes, contradictions, and context dependencies that real users create naturally would require hundreds of them. No team has the time to write and maintain all of those.
The non-determinism problem
An AI agent running on an LLM does not produce the same output every time. Ask the same question twice and you might get two different answers.
Ground truth testing assumes determinism. The expected output is fixed. If the agent's answer is slightly different in wording but correct in substance, the test fails. If the agent's answer matches the expected output word for word but is actually wrong in context, the test passes.
This forces teams into one of two bad options. They either make the test so loose that any reasonable answer passes, which catches nothing because the agent could hallucinate a specific number and still match the format. Or they make the test so strict that only the exact expected phrasing passes, which produces false failures where the agent answered correctly but used different words.
Most teams go with the loose option because false failures waste engineering time. The result is a test suite that passes everything and catches nothing meaningful.
The maintenance problem
Ground truth tests require maintenance. Every time the product changes, the expected outputs change. New features mean new test cases. Updated pricing means updated expected answers. A new integration means new scenarios to cover.
One engineering team we spoke with has 8 people maintaining their evaluation framework. Eight engineers writing test cases, updating expected outputs, and reviewing results instead of improving the agent itself.
The test infrastructure costs more than the product it tests. That ratio doesn't make sense.
What works better
The alternative isn't getting rid of all testing. It's adding a layer that generates the inputs instead of requiring you to write them.
Adversarial testing generates the scenarios. Instead of an engineer writing "how many deals closed last quarter," the system generates "show me the big deals" and "why did revenue drop" and "show me Sarah's pipeline, actually just the ones closing this month, and compare them to last month."
For data agents, the alternative is generating the test environment itself. Instead of building a test dataset by hand, generate the dataset from a schema description. Because the system generated the data, it knows the correct answer to every question. Ground truth is computed, not hand-written.
The ground truth approach covers the deterministic cases and it should keep doing that. But it only covers the cases someone thought to write. The agents that score highest on our tests are the ones that handle the cases nobody wrote. That's the layer ground truth testing doesn't reach.