What Happens When AI Agents Face Real Users
Every AI agent works in the demo. The question is what happens when someone who didn't build it starts using it.
I spent 5 years in sales at WeWork working with over 1,000 startups. I watched the same pattern hundreds of times: a team builds a product, tests it internally, and launches it. The first real user does something nobody expected and the product breaks in a way nobody anticipated.
With traditional software, that break is a bug report. With AI agents, that break is a conversation with a customer, a query against company data, or an email to a real prospect. The user doesn't file a bug report. They just leave. And the team never finds out what happened.
The gap between testing and reality
Engineering teams test their agents with clean inputs. "What is your pricing." "How do I reset my password." "Show me last quarter's revenue."
Real users don't ask clean questions. They ask something like:
"I saw something on your website about pricing but I can't remember what it was, can you just give me a ballpark for a team of maybe 10 or 15, we're in healthcare so there might be compliance stuff"
That is one sentence with five pieces of information, two hedges, one reference to something the user can't remember, and an implied question about compliance. No test case covers this. The agent has to interpret intent, handle ambiguity, address the compliance mention, and provide a useful answer all at once.
What we've observed across 150+ agent tests
We generate adversarial synthetic users and run them against live agents. Full multi-turn conversations where the user pushes the agent into territory nobody designed for.
Here is what we see consistently.
Turn one is always fine. The user asks a reasonable opening question. The agent responds well. If you only tested turn one, every agent passes.
Turn three is where cracks appear. The user has shared some context by this point. Maybe their team size, their industry, their current tools. The agent should reference this information going forward. Most agents treat turn three as if turns one and two never happened. They ask for information the user already provided. They pitch features that don't match what the user described.
Turn five is where failures compound. By turn five, a bad agent has asked for the user's email twice, ignored a stated constraint, and repeated the same call to action three times. A good agent has built on every piece of shared context and adapted its approach based on how the user is responding.
The gap between good and bad agents is not the model. It's the system around the model. The harness that tells the agent what to remember, when to adapt, and when to stop talking.
The data agent problem is worse
Conversational agents have visible failures. A user sees a bad response and can push back or ask again. The failure is at least detectable in the conversation.
Data agents have invisible failures. The agent says "you have 14 overdue tickets" and the user believes it. Why wouldn't they? The response is clear, specific, and confident. The user makes a decision based on that number. Nobody checks if 14 is correct.
We tested data agents across five domains: CRM, ticketing, knowledge base, system logs, and messaging. The most common failure mode is that the agent returns a confident, fluent, wrong answer and nobody notices.
A CRM agent that miscounts closed deals by 15%. An analytics copilot that calculates churn rate using the wrong time window. A knowledge base agent that cites an outdated article when a newer one contradicts it.
These failures don't trigger error alerts. They don't show up in observability dashboards. The output looks correct. The data is just wrong.
Why internal testing misses this
The person who built the agent is the worst person to test it. Not because they're bad at testing but because they think like the builder.
A builder asks "does the agent handle pricing questions?" and then types "what is your pricing" and verifies the response. Test passed.
A real user asks "I'm comparing you against two other tools and honestly the other one is cheaper, can you justify the price difference for a team our size."
That is a pricing question. But it's also an objection, a competitive mention, a sizing question, and an implied request for ROI justification all in one message. The builder never writes this test case because they think of pricing as a single-topic question rather than a conversation that branches in multiple directions.
The same gap exists for data agents. The builder tests "how many deals closed last quarter" and verifies the number. The real user asks "why is our close rate dropping" when the close rate isn't actually dropping. The builder never writes a test for a question built on a false premise.
Testing differently
The answer is not to write more test cases. More ground truth with more expected outputs is the same approach with more volume. It still only covers scenarios someone imagined ahead of time.
Testing differently means generating the scenarios. Let a system create the adversarial inputs, the ambiguous queries, the contradictory conversations, and the false-premise questions. Test the things the builder didn't think of because the system is specifically designed to generate the inputs that builders miss.
For data agents, testing differently also means generating the test environment. Instead of building test datasets by hand, generate them from a schema description. When the system builds the data, it knows every answer. Ground truth is manufactured automatically rather than hand-written over weeks.
The agents that perform best in production are the ones that were tested against inputs nobody expected. The teams behind those agents didn't necessarily think of better test cases. They used a system that generated the test cases they couldn't have written themselves.