Creating a Test

A test is a scenario, the criteria a good conversation has to meet, and the settings that decide how it is run. This article covers building one on the AI Testing page of a project, both kinds: a test that simulates conversations and a test that evaluates conversations that already happened.

Reading what a run produced is covered in Reading Test Results. Having the scenarios written for you instead of writing them by hand is covered in Generating Test Scenarios with AI. What the module measures overall, and the two things that must exist before any of it runs, are on AI Testing.

The AI Testing page

The page lists every test in the project as a card.

1. Create New Test: The first card, with two ways in. The + button opens the empty configuration form, described below. Generate with AI writes scenarios from the AI Agents already configured in the project, for you to review and keep.

2. Test cards: One per test, carrying a badge naming its type, Simulated conversations or Historical data, its name, its description and the result of its last run: the date, how many conversations were evaluated, and a colour-coded success rate. Run Test at the foot starts a new run without opening the form, and the button beside it schedules one. The Show filter at the top right narrows the page to one test type.

Choosing the test type

The configuration form opens on the Test tab, and the first choice is the type. It decides what the rest of the form asks for.

Success criteria are asked for either way, because they are what the evaluation scores against.

Setting up a simulated test

Three things describe the conversation to hold: what happens in it, who the customer is, and how many times to hold it.

The scenario

Test Name is required and is what the card shows.

Scenario Description is required, and it is the instruction the simulation follows, so write the conversation you expect rather than the outcome you want: for a booking agent, that the details are collected, then options offered, then the choice confirmed back.

Additional Info is optional and is where concrete data belongs: the cities, the dates, the account numbers the simulated customer should use. Keeping values here rather than in the scenario is what lets one scenario be reused with different data.

Personas

A persona is who the simulated customer is, not what they know. + Add Persona opens an empty description, + Add From Templates offers ready-made ones you can then edit.

Describe behaviour: how patient they are, how much they explain, whether they want options before deciding. Data values do not belong here, they belong in Additional Info. A test can carry several personas, and each conversation uses one of them, which is how one scenario gets tried against different kinds of customer.

Simulation parameters

Set the message cap to fit the whole flow you are testing. A cap below what the scenario needs cuts conversations off partway, and they are then scored on a conversation that never finished.

Setting up a historical test

Selecting Test with Historical Data shortens the form to three things: a Test Name, a Dataset, and the success criteria. There is no scenario, no persona and no simulation parameters, because nothing is being simulated. The conversations already exist and the dataset is what selects them.

The dataset is the whole configuration of this kind of test: which period to draw from, which channels, how long a conversation has to be, and what its AI Insights or custom attributes have to say. Building one is covered in Creating a Dataset of Past Conversations, and a test cannot be saved without one.

A historical run is also much cheaper and much faster than a simulated one, because no conversation has to be held.

Success criteria

Criteria are what the evaluation judges each conversation against. Add them with + Add Criteria, or start from + Add From Templates.

Each one has a Name, which becomes a column when you read the results, and a Description, which is the text the judge is given. The description is doing real work: it is read by a model, one conversation at a time, and a vague one produces scores that move for no reason. "The agent confirmed the booking reference back to the customer" scores consistently. "The agent was helpful" does not.

Type decides how the criterion is scored.

+ Add From Templates offers three ready-made criteria, Language Adherence, Language Quality and Semantic Coherence, each arriving with its description already written. They judge how the agent speaks rather than what it achieved, so they pair well with one or two criteria of your own about the outcome.

The model that runs and judges the test

Both the simulated customer and the evaluation are a model, set once for the tenant under Administration > Settings > AI Testing rather than per test, so changing it moves the baseline for every project at once. It is worth a strong reasoning model: judging a transcript against your criteria is a harder task than holding the conversation was.

Saving and running

Save & Run becomes available once the test carries everything its type asks for: a scenario, at least one persona and at least one success criterion for a simulated test, a dataset and at least one success criterion for a historical one. It saves the test and queues a run, then moves to the Evaluations tab where the run appears.

One project runs one test at a time. While a run is in progress the next is queued rather than started.

Running a test on a schedule

The button beside Run Test on a test's card opens Schedule Test Run.

Run At has to be in the future: a time that has already passed is refused. Set the time with the arrows in the hour and minute boxes, then click a day in the calendar before saving, so the field shows the value that will actually be stored.

Leave Recurring off for a single run, and the dialog asks for nothing else. Turning it on adds a Repeat of Daily, Weekly or Monthly and an End Date, and the last two then ask which days: On Days for Weekly, where you pick any combination of Mo to Su, and a day of the month for Monthly.

Where a schedule is visible

In one place only. The card's schedule button turns green, and that is the entire surface: nothing on the test's own page, nothing on the Evaluations tab, and no list of upcoming runs anywhere. The card is where you look.

Reopening the dialog on a scheduled test shows the saved time, with Update to change it and Delete Schedule to remove it.

When it runs, the run appears on the Evaluations tab like any other and carries no mark saying it was scheduled, so afterwards it is indistinguishable from one started by hand. A single, non-recurring schedule is used up when it fires and the button goes back to its plain state.

Moving a test to another project

Export and Import, at the top right of the AI Testing page, carry tests between projects and environments.

Export opens a list of the project's tests with a checkbox on each and a count of what is selected, so it exports the ones you choose rather than everything. What comes out is a ZIP named after the project and the moment it was taken, holding one JSON file per exported test.

Import takes that ZIP back. It accepts a .zip only, and the Import button stays disabled until a file is chosen.

Limits