Why our data agent tests don't use an AI judge
When a question has a right answer you can compute, code should grade it, not another model. That is how we test data agents, the ones that answer questions about a company's data, like "what was Q3 revenue from closed-won deals?" There is a correct answer. We compute it by running SQL on the test data, and plain code checks the agent against it. No AI judge in the loop. Here is why we built it that way, and how we proved to ourselves it works.
A quick note on the term. A data agent answers questions about your data: your CRM, your tickets, your orders. Ask it something and there is a fact at the other end. That is the whole reason this works, because a fact is something you can check with code.
We tested our own grader first
Before we trusted our grader, we tried to break it.
For every question in our test set, we wrote three answers by hand. Two were correct, written in different styles, because a real agent might give you a tidy table, a sentence, or a rounded number.
The third was a near miss. Right shape, wrong underneath. One wrong number. An extra row that should not be there. A made-up record that never existed in the data.
Then we ran all three through the grader and watched what it did. A good grader passes both correct answers and catches the near miss. Ours let the near misses through about 40% of the time.
Why it was so wrong
The first grader worked on keywords. It scanned the answer for the values it expected, and passed if it found them. Reasonable on paper, until you watch it fail.
The number 20 matches inside the year 2026. So an answer that should have said 20 of something could pass because the text had 2026 in it somewhere else. The grader saw the characters 2 and 0 sitting together and called it a match.
Partial matches like that were all over the place. A keyword check does not know that a value has to mean something and land in the right spot. It just looks for characters.
What we built instead
We took the AI out of the grading step. No model runs when we grade a data agent. This is the grader inside ClientCoded, the automated testing system we build for AI agents, and for data tests it is plain code from start to finish.
The setup is simple. Every test runs against a real set of data. Before grading anything, we run SQL on that data to compute the correct answer. That is the answer key, and it comes straight from the numbers.
Then the grader checks the agent's answer against that key:
- Each required value is actually in the answer.
- Each value sits next to the right name, so a correct number on the wrong account does not count.
- The agent did not invent extra rows or totals that were never in the data.
And it accepts the normal ways an agent writes. Rounding is fine. Dollars instead of cents is fine. A date in any common format is fine. A table or a sentence, both fine. What it rejects is a wrong value, a missing one, or a row that was never there.
Did it work
We reran the self-test. Two numbers moved.
Correct answers that got marked wrong dropped from about 4 or 5% to about 1%. We stopped failing agents for writing a right answer in a reasonable style.
Wrong answers that slipped through dropped from about 40% to a low single digit.
Making sure we did not cheat
There is a trap with this kind of tuning. If you keep adjusting a grader against your own test set until the numbers look great, you have really just taught it to pass that one set, and it can still fall apart on anything new.
So we hold one test environment completely out. The grader never gets tuned on it, and we only use it to check ourselves. Its scores there stayed in line with the rest, which told us the grader holds up on data it has never seen.
Why plain code is the right call here
Three things come out of grading with code instead of a model.
The same answer always gets the same grade. Run it today or next week, you get the identical result. For the part of the system that is supposed to be the source of truth, that steadiness is the point.
There is no cost per test. We run a lot of tests, and plain code does not bill us for each one.
And every grade shows its reason. A pass or a fail points at a specific value and a specific check, so if you ever disagree with a grade, you can open it and see exactly why.
The one place we allow judgment
Not every question has one clean answer. "Show me the top deals" is vague. Top by what? How many?
An ambiguous question is still fair, as long as the agent deals with the ambiguity out loud. If it says how it read the question, like "top 5 by dollar value," it passes. If it asks a clarifying question, it passes. If it just hands over an answer and hopes, it fails, because a real user would have no idea what they were looking at.
Where we do use an AI judge
Now the honest part. Not all of our testing works like this, and it would be wrong to pretend it does.
We also run conversation tests, where a sales or support agent talks with a simulated customer over a full back and forth. There is no SQL query for whether the agent handled an objection well, held the right tone, or escalated at the right moment. Those take judgment.
So we grade those with an AI model, and we keep it on a short leash. The model scores against published, written rubrics, one dimension at a time, so it answers a narrow, defined question rather than handing back a vibe. The rubrics are written down and visible, so you can see what each score is measuring.
That is the line we draw. Use code where there is a right answer you can compute. Use a tightly constrained rubric where there is not. The mistake we wanted to avoid was reaching for a model on the first kind, where a few lines of code are faster, cheaper, and steadier.
The takeaway
If you are building your own automated testing system, that split is where I would start. Grade with code wherever the answer is computable, and keep any model that does the grading on a tight, written rubric. And whichever side a test falls on, try to break your grader before you trust it. The payoff is a grader you can rely on: the same result every time, with a visible reason behind every grade.
Frequently asked questions
Why don't your data agent tests use an AI judge?
Because a data agent question has a right answer you can compute. We run SQL on the test data to get the correct answer, then plain code checks the agent against it. Code grades the same way every time, adds no cost per test, and can show the exact reason for every pass and fail. An AI judge would add cost, could grade the same answer differently on different runs, and tends to share the same blind spots as the agent it is grading.
How do you test whether a grader is accurate?
Write answers by hand and run them through the grader. For each question we wrote two correct answers in different styles and one near miss that was wrong in a specific way, like one wrong number or an extra row. A good grader passes both correct answers and catches the near miss. When we did this, our first grader passed about 40 percent of the wrong answers.
How does the grader handle answers written in different styles?
It computes the correct answer from the data, then checks that each required value is present and sits next to the right name. It accepts rounding, dollars instead of cents, dates in any common format, and answers written as a table or as prose. It only fails an answer for a wrong value, a missing value, or an invented row or total.
Do you ever use an AI model to grade agents?
Yes, for conversation tests. When a sales or support agent talks with a simulated customer, judging tone, objection handling, and escalation timing takes judgment, so those are scored by an AI model against published written rubrics, one dimension at a time. The rule we follow is simple: use code where there is a right answer you can compute, and a tightly constrained rubric where there is not.