Skip to main content
A test set is a collection of test cases, and a test case is one scenario you want to evaluate your agent against — a goal the simulated user pursues, like “get a refund” or “reschedule an appointment.” When you launch a simulation, Coval runs each test case as a conversation and scores the result against your metrics; the test set aggregates the outcomes across all its cases.

​
What test cases let you do

A test case is how you tell Coval what to put your agent through. A good set of them lets you:
  • Cover the scenarios that matter — happy paths, edge cases, and known failure modes, one test case each.
  • Catch regressions — re-run the same test set after every change to confirm nothing broke.
  • Test across user types — run the same test set with a different persona (one per run) to see how your agent holds up with different kinds of users.

Create a test set

Generate test cases with AI, build them manually, or upload a CSV/Excel file.

Input types

Scenario, transcript, audio, script, or image — how tightly the simulated user follows your input.

Expected behavior & attributes

Define how the agent should respond and attach per-test-case data for evaluation.

API

Create, list, update, and delete test sets programmatically.

​
Test case inputs

The Simulation Input is the heart of a test case: it becomes the simulated user’s objective for the run. The input type determines how tightly the simulated user follows it — from improvising toward a high-level goal to delivering exact lines: See Input types for how each behaves and how to configure it. You can also attach an image (WebSocket voice agents only) or a pre-recorded audio file to a test case.
For comprehensive testing, create multiple types of test sets:
  • Regression Set: Contains “happy path” scenarios representing typical successful interactions
  • Adversarial Set: Contains edge cases and scenarios designed to test your agent’s limits and handling of unusual requests

​
Test case vs. persona

A test case defines what the simulated user is trying to accomplish; a persona defines who they are — their tone, accent, and behavior. The same test case can run with any persona, which pursues the test case’s goal in its own style. To make the simulated user attempt a specific (or adversarial) request, put it in the test case’s Simulation Input — the persona only shapes how the user behaves, not what they’re trying to do. A run executes over multiple test cases, but only one persona.

​
Version history

Every config-changing save of a test set is recorded in its version history — capturing both the test-set configuration and a snapshot of its test-case grid — so you can see how the test set changed over time and tell which version a run used. See Versioning for how copy-on-save works and how to pull the history through the v1 API.