Three Layers of AI Agent Evaluation
Most teams evaluating AI agents are doing one of two things: scoring the outputs the agent produces, or monitoring what the agent does in production. Both are necessary. Both have a gap.
There is a third layer that most teams haven't implemented yet, and it's the one that catches the failures the other two miss.
Layer 1: Output scoring
This is the most common approach. You provide a set of inputs, run the agent, and score the outputs against some criteria. Faithfulness, relevance, hallucination, coherence, toxicity. The metrics are well-established and the tooling is mature.
Output scoring answers the question "given this input, did the agent produce a good output?" And for the inputs you test, it works well. If you write 50 test cases and the agent passes all 50, you know it handles those 50 scenarios correctly.
The limitation is in the inputs themselves. Someone has to write them. That someone is usually the engineer who built the agent, which means the test cases reflect what the builder expects users to say. The scenarios the builder didn't imagine are the scenarios that never get tested.
Output scoring also works best on single input-output pairs. Most evaluation frameworks test one question and one answer at a time. But agents have conversations. Turn one sets context that should influence turn five. A test that evaluates turns in isolation misses the failures that only appear across a conversation arc.
This layer is essential. Every team should have it. But it only covers the ground you thought to test.
Layer 2: Production monitoring
Once the agent is live, you need to know what it's doing. Production monitoring traces every interaction in real time. Every LLM call, every tool use, every retrieval step gets logged. When a user has a bad experience, you can inspect the trace and see exactly where the agent went wrong.
Production monitoring answers the question "what happened in this specific interaction?" It's the debugging layer. When something breaks, you can trace it back to the exact decision point where the agent made a mistake.
The limitation is timing. You find the problem after a real user experienced it. The trace tells you what went wrong, but only after the damage is done. A data agent that returned the wrong number to a VP who made a budget decision based on it, a support bot that gave incorrect policy information to a frustrated customer. The monitoring shows you the failure. It doesn't prevent it.
In practice, most teams also don't have someone reviewing traces continuously. Issues sit in logs until a customer complains or until someone runs a periodic review. The failures that don't generate complaints are the ones that persist the longest.
This layer is also essential. Every team deploying agents in production should have visibility into what those agents are doing. But it's reactive by design.
Layer 3: Adversarial testing
This is the layer most teams are missing. Instead of scoring outputs on inputs you provided, or monitoring what happens with real users, adversarial testing generates the inputs. The system creates users and scenarios specifically designed to push the agent past the boundaries it was built for.
A synthetic user who changes what they want mid-conversation. A query that's deliberately ambiguous and could mean two different things. A request for data the user shouldn't have access to. A question built on a false assumption. A follow-up that references something from three turns ago.
These are the scenarios that real users create naturally but that no engineer writes as test cases because they didn't think of them. The builder tests "what is your pricing" and the agent passes. The real user asks "I talked to your competitor and they're cheaper, why should I pay more for a team my size in healthcare." Same topic. Completely different test.
Adversarial testing answers the question "what happens when someone tries to break this agent?" And the answer is usually surprising. We've tested over 150 agents and the average succeeds in about half of adversarial conversations. The failures are things the team had no idea about because the scenarios were never in their test suite.
For data agents, the adversarial layer extends to generating the test environment itself. Instead of an engineer building test datasets by hand and computing expected answers manually, the system generates a synthetic dataset from a schema description and creates adversarial queries with computed ground truth. The questions include ambiguous inputs, contradictory requests, scope boundary violations, and false-premise questions that no engineer would think to write because they know their own data too well.
Why all three layers matter
Each layer catches a different class of failure.
Output scoring catches quality issues on known inputs. If the agent's response to "what is your refund policy" is inaccurate, output scoring finds it. This is the baseline. Without it, you don't know if the agent handles even the expected scenarios correctly.
Production monitoring catches issues in live traffic. If a specific user interaction produces a bad result, monitoring surfaces it. This is the safety net. Without it, you're blind to what your agent does in the real world.
Adversarial testing catches failures on inputs nobody anticipated. If the agent breaks when a user changes intent at turn four or asks an ambiguous question about their data, adversarial testing finds it before a real user does. This is the prevention layer. Without it, your first indication of these failures is a customer complaint or a bad business decision based on wrong data.
Teams that only do output scoring ship agents that work on expected inputs and fail on unexpected ones. Teams that only do monitoring find failures after users experience them. Teams that add adversarial testing find failures before they reach users, on scenarios they never would have written themselves.
The three layers are complementary. Output scoring validates what you built. Monitoring watches how it performs. Adversarial testing finds what you missed.
Most teams have the first layer. Many are adding the second. The third is the one where the tooling is newest and adoption is just starting. But it's the layer that catches the failures the other two are structurally unable to find, because you can't score an output for an input that was never generated, and you can't monitor a failure that hasn't happened yet.
Where to start
If you have nothing, start with output scoring. Get a baseline on whether your agent handles expected inputs correctly.
If you have output scoring but your agent is in production, add monitoring. You need visibility into what real users are experiencing.
If you have both and you're still seeing unexpected failures in production, the gap is adversarial testing. The failures that surprise you are the ones that were never in your test suite. Generating those scenarios automatically is how you stop being surprised.
For data agents specifically, if your team is spending weeks building test datasets by hand and still can't cover every scenario, generating the test environment automatically saves that time and tests the inputs your team wouldn't have written.