Generating Test Scenarios with AI

Test scenarios can be written for you from the AI Agents already configured in a project, so testing can start without writing each scenario by hand. This article covers the Generate with AI option on the AI Testing page: choosing the agents the scenarios are based on, steering what they are about, reviewing what comes back, and adding the ones you keep.

It does not cover the test configuration form, running a test, or reading evaluation results. Those are in AI Testing, and everything a generated scenario contains can be edited there afterwards.

Scenario generation is available in the Premium edition.

📝 Note: Two things must exist before generation works. The project needs at least one AI Agent, because the scenarios are written from the agents' own configuration (see Creating an AI Agent). The tenant needs an LLM configuration selected for AI Testing under Administration > Settings > AI Testing, which is the same setting the test runs and the evaluation judge use. With no agent selected, the Generate button in the modal stays disabled.

Opening the generator

Open your project and go to AI Testing. The generator is on the first card of the page, beside the scenarios that already exist.

1. Create New Test card: The entry point for a new test scenario, always the first card on the page.

2. + (manual creation): Opens the empty test configuration form, where a scenario is written field by field. This route is documented in AI Testing.

3. Generate with AI: Opens the Generate Scenario with AI modal, covered below.

4. Existing scenario cards: The scenarios already saved in this project, each with its last evaluation results and a Run Test button.

The generation modal

The modal opens on its subtitle, "select the AI Agent(s) this scenario should be based on", with nothing chosen yet.

1. AI Agent selection: The project's agents, listed as a tree with each supervisor's sub-agents nested under it. Tick every agent the scenarios should cover. The footer shows how many you have selected out of the maximum of ten.

2. Notes for the scenario (optional): Free text that steers what the scenarios are about: the customer, the data, the situation, the part of the agent's job you want exercised. Up to 1000 characters.

3. Number of scenarios to generate: How many scenarios come back from one run. Any value from 1 to 10. Default: 3.

4. Generate: Starts the generation. It stays disabled until at least one agent is ticked.

📝 Note: Ticking a supervisor does not tick its sub-agents. Each agent is selected on its own, so tick every agent you want covered, including the specialists under a supervisor you have already ticked.

💡 Tip: Leave the notes field empty and the generator invents its own customer details, which will not match the mock data your tools return. Put the concrete values in the note (customer names, IDs, invoice numbers, cities) and they come back inside the scenarios.

What travels with the request is the selected agents' descriptions, instructions, tools, tool parameters and transfer instructions, together with your note and the number of scenarios you asked for. Scenarios are written from that configuration, not from past conversations, so they cover what the agents are designed to do rather than what customers have actually asked.

Filling it in

The example below uses a project with a CustomerCareSupervisor and two specialists under it, BillingAssistant and TechSupportAssistant.

1. Tick all three agents, one by one.

2. Type into Notes for the scenario (optional):

   Use the customer Emma Collins, customer ID CUST-1001, current invoice INV-2026-07-4412 for 64.90 USD due 2026-08-15, service number SVC-77821, and the active outage in Istanbul Kadikoy.

3. Leave Number of scenarios to generate at 3.

4. Click Generate.

1. Agent tree with the selection made: The supervisor and both specialists are ticked individually.

2. Notes for the scenario: The note above, carrying the data the scenarios should use.

3. Number of scenarios to generate: Left at the default of 3.

Reviewing what comes back

The scenarios arrive in the modal, collapsed, one row each, and every one is already selected. Nothing has been saved yet.

📝 Note: The output is generated, so your scenarios will read differently from the ones in this screenshot, and differently again on a second run with the same input.

1. Result header: How many scenarios were generated in this run.

2. Scenario rows: One row per scenario, each with its own checkbox. Every scenario arrives checked. Uncheck the ones you do not want to keep. At least one must stay checked before the scenarios can be added.

3. Expanded scenario: Click a row to open it and read what it will actually test: the scenario description, the persona the simulated customer will adopt, the additional information, and the success criteria.

4. Additional Info: The concrete detail carried through from your note, such as the customer and the invoice above.

5. Success Criteria: What each simulated conversation will be judged against. These name the project's own tools, because they were written from the selected agents' configuration.

6. Add to test scenarios: Saves the checked scenarios to the AI Testing page.

💡 Tip: Read at least one scenario in full before adding. Everything here is editable afterwards in the test configuration form, but it is faster to drop a scenario that misses the point now and generate again with a sharper note than to rewrite it later.

After the scenarios are added

The scenarios you kept appear on the AI Testing page as cards, immediately.

1. Generated scenario cards: One card per scenario you kept, with the generated name and description.

2. Empty evaluation results: The new cards carry no evaluation results, because generating a scenario does not run it.

3. An earlier test that has been run: For contrast, a scenario with a completed run shows its date, its number of simulated conversations and its success rate.

⚠️ Warning: Generation saves the scenarios, it does not run them. A generated scenario has no evaluation result until somebody runs it. Use Run Test on the card, or open the scenario and use the test configuration form, as described in AI Testing.

A generated scenario is an ordinary test scenario from this point on. Open it to change its name, description, personas, additional information, simulation parameters or success criteria before the first run.

Limits