AI Testing scores conversations against criteria you write, and it works two ways. Simulate Conversations invents the customer: you describe a situation, say what kind of person raises it and what a good outcome looks like, and the platform holds that conversation with your agents as many times as you ask. Test with Historical Data takes the other route and evaluates real production conversations that already happened, drawn from a dataset you define. Either way each conversation is scored one criterion at a time, with the transcript kept beside the score.
Everything lives on the AI Testing page of a project: the scenarios, the history of what has been run, and the evaluation of each run.
What it is good for
The everyday use is checking a change before it ships, with simulated conversations. Run the scenarios that cover the work an agent does, read the per-criterion scores, and see whether the change did what you expected without breaking something beside it.
The other is measuring what really happened. A dataset samples past production conversations by date, channel, duration and what your AI Insights and custom attributes recorded about them, and the same success criteria are applied to those. It is how a claim about live quality gets a number behind it.
The second is finding where a conversation goes wrong rather than that it went wrong. A run that scores well overall and zero on one criterion points at the exact turn to read, and the transcript is one click away.
The third is watching a number move. Runs are kept in a chronological history with their scores and token consumption, so successive runs can be compared rather than judged one at a time.
What it is not
- The two test types answer different questions, and neither answers both. A historical test reads conversations that already happened, so it tells you how the project did perform and can say nothing about a change you have not shipped. A simulated test is the one that can, because the conversation is held against the agents as they are configured now.
- A simulated run is not repeatable to the letter. Conversations are generated fresh on each run and the scoring is a model's judgment, so the same scenario run twice gives different transcripts and scores that move. Read a single run as a measurement with noise in it, and put weight on the trend across several runs rather than on one figure. Precise success criteria narrow the noise; vague ones widen it.
- It does not change your agents on its own. A run produces evidence and, through AI Agents Optimization, proposals. Nothing reaches an agent's configuration until somebody confirms it.
Before you start
Two things have to exist, and both sit outside the AI Testing page:
- At least one AI Agent in the project. Scenarios are written and simulated against the agents' own configuration, so a project without one has nothing to test. See Creating an AI Agent.
- An LLM configuration selected for AI Testing, under Administration > Settings > AI Testing. The same setting serves the simulated conversations, the evaluation judge and the scenario generator.
Writing the scenarios for you, with Generate with AI, is available in the Premium edition.
Each run reports the tokens it consumed. A test is a conversation like any other as far as cost is concerned, so the number of simulated conversations and the message cap are worth setting deliberately rather than high.
Next
- Creating a Test: the configuration form, both test types, personas, simulation parameters and success criteria.
- Generating Test Scenarios with AI: having the scenarios written from your agents' configuration, steering what they cover, and reviewing what comes back.
- Creating a Dataset of Past Conversations: selecting which production conversations a historical test evaluates, by date, channel, duration and what was recorded about them.
- Reading Test Results: the run history, the criteria breakdown, the per-conversation scores and the transcripts.
- AI Agents Optimization: turning a run's evaluation into proposed configuration changes, reviewed and approved one by one.