How to QA a Data Agent Without Building Test Datasets by Hand
Every company deploying a data agent hits the same wall. The agent works in demos. It answers clean questions correctly. Then a real user asks something messy and nobody knows if the answer is right.
The QA process for data agents today looks like this: an engineer writes 20-30 test questions, manually computes the expected answer for each one, runs the agent against those questions, and compares. This takes weeks. And every time the schema changes, they start over.
One team we spoke with has 8 engineers maintaining their evaluation infrastructure. Their assessment: "It's not possible to have data sets that cover every possible scenario."
They're right about the manual approach. But automated generation can cover far more than anyone could write by hand.
The core problem with data agent QA
A data agent returns a wrong answer. The answer is fluent, specific, and confident. The user believes it. A business decision gets made based on incorrect data.
Unlike a conversational agent where a bad tone or missed objection is noticeable, a data agent failure is invisible. "You have 14 overdue tickets" sounds perfectly fine unless the real number is 23.
To catch this, you need to know the right answer. Traditional LLM evaluation metrics like faithfulness, relevance, and coherence don't help here. The response is faithful to what the model generated. It's relevant to the question. It's coherent. It's also wrong.
You need ground truth. And getting ground truth for data agents is where every team gets stuck.
Generate the data, own the ground truth
The insight is simple: if you generate the dataset, you know every answer.
A customer describes their data schema. "We have a CRM with deals, contacts, companies, and activities. Deals have stages: prospecting, discovery, proposal, negotiation, closed won, closed lost."
From that description, the system generates 5,000 deals with realistic amounts, stages, close dates, and owners. 2,000 contacts linked to 500 companies. 10,000 activities across those deals.
Every row is synthetic. No real customer data is involved. But the patterns are realistic: deal amounts follow a normal distribution, close dates span the last 18 months, some deals are stalling, some contacts have no recent activity.
Because the system generated every row, it can compute the correct answer to any question about the data. "How many deals closed last quarter" is a count against a known dataset, not an approximation.
Seven categories of adversarial questions
Clean inputs are the baseline. Every agent handles "how many deals closed last quarter" correctly. The value is in the questions nobody writes by hand.
We generate questions across seven categories:
Clean queries (20%). Straightforward questions with clear answers. These establish whether the agent works at all. If it can't handle these, nothing else matters.
Ambiguous queries (20%). "Show me the big deals." What does big mean? A good agent asks for clarification. A bad agent picks a threshold and hopes for the best.
Multi-step queries (15%). "Show me deals over $50K, then filter by the ones closing this month, sort by amount, and tell me who owns the top three." This tests whether the agent can chain operations correctly without losing track of the constraints along the way.
Scope boundary (15%). "Show me everyone's commission numbers." Should the agent return this data? These questions test whether the agent respects permission limits and handles requests for unauthorized or destructive actions safely.
Contradictory queries (10%). "Show me Q1 numbers. Actually make that Q3. No wait, show me both." The user changed their mind three times. Can the agent track the latest intent without confusing it with the earlier ones?
Invalid assumptions (10%). "Why did revenue drop last month?" Revenue actually went up. Does the agent correct the false premise, or does it generate an explanation for something that didn't happen? This is where confident hallucination is most dangerous because the agent is explaining a trend that doesn't exist.
Context-dependent (10%). "Show me more details on that last one." This requires the agent to track what "that last one" refers to from the previous exchange. Sequential questions that build on each other test whether the agent maintains context or treats each question as a fresh start.
What the scorecard shows
After running 50 questions against the agent, the result is a breakdown by category.
"42/50 correct. Clean queries: 10/10. Ambiguous: 6/10. Scope boundary: 9/10. Contradictory: 5/5. Invalid assumptions: 3/5. Context-dependent: 4/5. Multi-step: 5/10."
Now the team knows exactly where the agent breaks. Multi-step queries and ambiguous inputs are the weak spots. Clean inputs and scope boundaries are solid. The fix is targeted instead of a general "make it better."
Each wrong answer includes the question, the expected answer, the agent's response, and an explanation of what went wrong. Something like "the agent returned 14 overdue tickets but the correct count is 23 because the agent filtered by status but missed tickets where the due date had passed without a status update."
Five domains, same pattern
The same approach works for any agent that queries structured data:
CRM agents. Pipeline questions, deal aggregations, permission boundaries, forecast accuracy.
Ticketing agents. Sprint status, dependency tracking, assignment and reassignment with permission checks.
Knowledge base agents. Document retrieval, conflicting sources where an outdated article contradicts a current one, out-of-scope questions, multi-document synthesis.
System log agents. Root cause analysis, error correlation across services, uptime calculations, cascading failure diagnosis.
Messaging agents. Cross-channel search, thread summarization, restricted channel boundaries, action item extraction.
Each domain has its own failure modes. A CRM agent needs to handle ambiguous deal-size thresholds. A ticketing agent needs to walk dependency chains. A knowledge base agent needs to detect when two sources contradict each other. The question generator is aware of these domain-specific patterns.
How to start
Describe your schema. Table names, column names, relationships, business rules. The system generates everything else.
You never share real customer data. You share structure. The system creates a synthetic replica with realistic patterns and adversarial questions designed to push the agent past the inputs it was built for.