Field notes on AI agent quality.

When a question has a right answer you can compute, code should grade it, not another model. How we built and tested our data agent grader, and where we still use an AI model against written rubrics.

A pre-launch test suite is a snapshot that goes stale the day you ship. How to close the loop: monitor production, turn the real failures into new adversarial tests, and retest so coverage keeps up with real users.

The wrong answers that hurt are the fluent, confident ones that read as correct. Catch them in two places: test against known answers before launch, and score every production conversation in real time.

Your own testing misses the failures because you test the happy path. How to test an agent with adversarial scenarios across 7 categories, scored against known answers, with a pre-launch checklist.

An AI SDR fails in a prospect's inbox with your name on it, and the later turns are the ones nobody tests. How to test an AI SDR against synthetic prospects across multi-turn threads, scored on 10 dimensions, before a single real send.

Manual spot checks miss the inputs you never tested, and every retrain is a potential regression. How to test a Rasa agent with adversarial scenarios across 7 categories, scored on 10 dimensions.

Tracing shows what happened, not whether the outcome was right. Why observability alone misses the fluent, confident, wrong answer, and what closes the loop from behavioral test to root cause to verified fix.

OpenClaw makes deploying an agent easy. Testing it is another matter. How adversarial testing catches the multi-turn, fabrication, scope, and unauthorized-action failures before your users do.

The average agent fails half its adversarial conversations, and breaks between turn three and turn five. Here are the three most common failures, and what the agents that score well do differently.

Ground truth testing breaks the moment a user asks something without one right answer. The clean-input, multi-turn, non-determinism, and maintenance problems, and what works better.

If you generate the dataset, you own the ground truth. The seven categories of adversarial questions, what the scorecard shows, and how the pattern covers CRM, ticketing, knowledge base, logs, and messaging.

Output scoring, production monitoring, and adversarial testing. Each catches a different class of failure, and the third layer is the one most teams are missing.

Every agent works in the demo. Turn one is fine, turn three is where cracks appear, turn five is where failures compound. Why the builder is the worst person to test the agent.