How to Test Your Rasa Agent Beyond the Happy Path
If you built a conversational agent on Rasa, you probably tested it the same way everyone does: you typed in the 10 inputs you expected users to send and verified the responses looked right. Then you shipped it.
The problem isn't what happens when users say what you expect. It's what happens when they don't.
The testing gap in Rasa projects
Rasa gives you tools to train your NLU model and define conversation flows. What it doesn't give you is a structured way to test what happens when real users do unexpected things.
Most Rasa teams we've talked to describe their QA process as one of three things:
- Manual spot checks after every model retrain
- A handful of end-to-end tests they wrote once and haven't updated since
- Nothing
The manual approach breaks down fast. You retrain your NLU model, update a few responses, and run through your 10 test inputs. They all pass. You ship. Two days later a customer reports the agent gave them completely wrong pricing information because your new model shifted the intent classification for a phrase you never tested.
What actually breaks in production
After testing 150+ agents across different frameworks, the same failure patterns show up repeatedly. These aren't Rasa-specific but they hit Rasa agents hard because of how tightly NLU classification and dialogue policies interact.
Confident wrong answers
Your agent classifies an intent correctly but pulls the wrong entity or slot value. The response reads perfectly. It's just wrong. Nobody catches it because the conversation looks normal.
Example: a user asks about their order status. The agent classifies the intent correctly as check_order but fills the order_id slot with a number from a previous conversation turn. The response says "Your order #4521 is being shipped" when their actual order is #4528. Confident, helpful, wrong.
Scope boundary failures
The agent is designed to handle customer support but a user asks about job openings. Instead of saying "I can't help with that," the agent tries to answer because the NLU model found a vaguely similar intent. The user gets a response that sounds authoritative but is completely fabricated from the training data.
Entity ambiguity
A user says "I want to cancel" and the agent asks "Which subscription?" But the user only has one subscription. The agent doesn't check. It asks a clarifying question that makes no sense because the dialogue policy doesn't consider the user's actual data.
Or worse: a user says "John's account" and there are 5 Johns in the system. The agent picks one without asking.
Post-retrain regression
You improve the model's handling of refund requests. Unintentionally, the intent boundary between refund_request and order_complaint shifts. Now 15% of complaints get classified as refund requests and the agent starts processing refunds nobody asked for.
This is the most common failure we see. Every retrain is a potential regression that your existing tests don't cover because they were written for the old model.
Long conversation drift
Your agent handles turns 1 through 5 perfectly. By turn 12, it's lost track of the original context, the persona instructions have faded, and it starts responding generically. Users who need help with complex multi-step issues get the worst experience because they interact the longest.
How to test for these failures
The answer isn't writing more test cases manually. You'll always think of the inputs your agent handles well because you built it. The failures come from inputs you never considered.
Adversarial testing
Instead of testing with expected inputs, test with inputs designed to break things:
Ambiguous inputs. Send your agent "I want to check on that thing from last week" with no prior context. Does it ask for clarification or does it guess?
Out-of-scope inputs. Ask your support agent about the weather, about your stock price, about something completely unrelated. Does it refuse gracefully or does it try to answer?
Contradictory inputs. Tell the agent "I want to cancel my subscription" then immediately say "Actually, how do I upgrade?" Does it handle the pivot or does it get confused?
Adversarial personas. Have someone pretend to be an angry customer who gives incomplete information, changes their mind, and asks irrelevant questions in the middle of a support flow. That's what real users do.
Edge case entities. Names with special characters. Email addresses with unusual domains. Phone numbers in different formats. Order IDs that don't exist. Amounts of $0. Dates in the past.
Structured categories
Don't just throw random inputs at your agent. Categorize your tests so you can measure improvement:
- Clean inputs (baseline): standard expected queries to verify basic functionality
- Ambiguous inputs: questions that could be interpreted multiple ways
- Multi-step queries: requests requiring the agent to maintain context across turns
- Scope boundary: inputs outside the agent's designed domain
- Contradictory inputs: requests with impossible or conflicting conditions
- Invalid assumptions: inputs that assume data exists when it doesn't
- Context-dependent: questions that require information from earlier in the conversation
Run 20 to 30 tests per category. That gives you a statistical sample large enough to measure whether your agent is improving or regressing.
Scoring beyond pass/fail
A simple pass/fail doesn't tell you much. Score each conversation across multiple dimensions:
- Accuracy: did the agent give correct information?
- Scope handling: did it stay within its defined boundaries?
- Clarity: was the response understandable?
- Escalation: did it know when to hand off to a human?
- Persona consistency: did it maintain the right tone throughout?
When you can see that your agent scores 90% on accuracy but 40% on scope handling, you know exactly where to focus your next iteration.
Making this repeatable
The biggest problem with manual testing isn't the quality. It's that nobody does it consistently. After the first launch, testing degrades from "thorough manual review" to "I tried a few things and it seemed fine."
What you need is a process that runs automatically after every model retrain or response update:
- Run the same adversarial test suite against the new version
- Compare scores against the previous baseline
- Flag any dimension where the score dropped
- Block the deploy if critical dimensions regressed
This is basically CI/CD for your conversational agent. The same way you wouldn't ship code without running tests, you shouldn't ship an agent update without running adversarial scenarios.
The tooling question
You can build this yourself. Write a script that sends messages to your Rasa agent's REST endpoint, collects responses, and scores them against expected behaviors. That works until you have 200 test cases to maintain and nobody wants to update them.
We built ClientCoded specifically for this problem. We generate the adversarial scenarios automatically across all 7 categories, score every conversation across 10 dimensions, and flag regressions when you push an update. If you want to see what it catches on your agent, the first audit is free.
But regardless of whether you use a tool or build your own, the approach matters more than the tooling. Test with adversarial inputs, categorize your tests, score across dimensions, and make it repeatable. Your users are already doing the adversarial testing for you in production. The question is whether you find the failures before they do.