The AI Agent Optimization feature analyzes the results of an AI Testing run, generates concrete improvement suggestions for the responding agents, and applies the approved changes through the MCP Server. The flow documented here covers Human Approval Mode, in which every change is reviewed and confirmed by the user before it is committed.
📝 Note: There are two ways to improve an agent with AI, and this article covers one of them. AI Agent Optimization starts from a test run: it reads what the evaluation found and proposes changes based on that evidence. Agent Improver starts from a sentence you write, needs no test run first, and is opened from the agent's own card. See Improving an AI Agent with AI. Both propose changes for approval rather than applying them directly.
Overview
AI Agent Optimization closes the loop between AI Testing and agent configuration. After a test run is simulated and evaluated, a summarization step condenses the evaluation results into a structured input, and the Optimization Agent turns that input into a set of proposed configuration changes. In Human Approval Mode, those proposals are surfaced as suggestion cards in the User Review Panel, where the operator confirms, refines, or rejects them. Only confirmed changes are sent to the MCP Server for application, and an optional re-run validates the effect of the changes against the same scenarios.
The process is organized into five sequential phases:
1. Simulation: Test scenarios are executed.
2. Evaluation + Summarization: Conversations are scored against success criteria and condensed.
3. Agent Modifier + User Review: Improvement suggestions are generated and reviewed.
4. Apply Changes: Confirmed changes are committed through the MCP Server.
5. Re-run Test: The updated agents are re-tested on demand.
How it Works
Phase 1: Simulation
- AI Testing executes the configured test scenarios. Each scenario is run using its scenario description together with the associated personas.
Phase 2: Evaluation + Summarization
- The generated conversations are passed to the Evaluation API along with the success criteria defined for the test.
- The Evaluation API returns the criteria results of all conversations to Summarization, which condenses them into a single structured output.
📝 Note: The summarization step receives both passing and failing criteria, not only the failures. This allows the Optimization Agent to identify what should be preserved as well as what should be improved.
Phase 3: Agent Modifier + User Review
- Summarization sends the summarization output together with the responding agent configurations to the Optimization Agent.
- The Optimization Agent invokes its tool to enumerate the proposed changes.
- The planned changes are presented to the User Review Panel as suggestion cards.
- A review loop runs until the operator confirms:
- Selected suggestions are returned to the Optimization Agent as selected captions.
- A refinement may be entered as free-text chat input (for example, a request to shorten the first suggestion), which prompts the agent to revise its proposals.
- The Optimization Agent re-runs its tool to reflect the refinement, and the updated cards are presented again.
💡 Tip: Refinement is conversational. Rather than rejecting a suggestion outright, the user may steer it with natural-language instructions, and the revised proposal is regenerated within the same review loop.
Phase 4: Apply Changes
- Once the operator confirms, the Optimization Agent sends a request to the MCP Server.
- The MCP Server applies the changes through its update endpoint and returns confirmation to the Optimization Agent.
Phase 5: Re-run Test
- If the operator triggers a re-run from the User Review Panel, a re-run test request is sent to AI Testing using the same scenarios and personas as the original run.
- AI Testing runs the simulations again, allowing the effect of the applied changes to be measured against an identical baseline.
Data Flow Diagram
User Interface Walkthrough
Evaluation Details Page
The flow starts from the evaluation results of a completed test run by clicking the Analyze & Suggest Improvements button. Selecting it opens a mode menu from which the optimization run is started.
| Mode | Description | Availability |
|---|---|---|
| Human Approval | Review and approve AI suggestions before applying them. | Available |
| Self-Optimizing | AI automatically applies optimizations without manual review. | Coming soon |
📝 Note: The Self-Optimizing mode is marked Coming soon and is not selectable. In the current release, every optimization run proceeds through the Human Approval flow described below.
Human Approval Panel
Selecting Human Approval opens a side panel anchored to the right edge of the screen.
Refining Suggestions
A refinement instruction may be entered in the typing area (for example, a request to make the suggestions shorter). The instruction is sent as a chat message, and the panel regenerates the suggestion set in response: the summary line updates to reflect the new intent, and a revised list of cards is presented with the same anatomy.
Applying Changes
Selecting Apply Selected commits the checked cards to the responding agent configurations. On completion, the panel displays a confirmation banner indicating that the changes were applied successfully and the agent configurations were updated. A Re-run Test action is offered alongside the banner. It re-runs the test using the same scenarios and personas as the original evaluation, allowing the effect of the applied changes to be measured against an identical baseline.
