Skip to main content
A test set is a collection of test cases, and a test case is one scenario you want to evaluate your agent against — a goal the simulated user pursues, like “get a refund” or “reschedule an appointment.” When you launch a simulation, Coval runs each test case as a conversation and scores the result against your metrics; the test set aggregates the outcomes across all its cases.

​
What test cases let you do

A test case is how you tell Coval what to put your agent through. A good set of them lets you:
  • Cover the scenarios that matter — happy paths, edge cases, and known failure modes, one test case each.
  • Catch regressions — re-run the same test set after every change to confirm nothing broke.
  • Test across user types — run the same test set with a different persona (one per run) to see how your agent holds up with different kinds of users.

Create a test set

Generate test cases with AI, build them manually, or upload a CSV/Excel file.

Input types

Scenario, transcript, audio, script, or image — how tightly the simulated user follows your input.

Expected behavior & attributes

Define how the agent should respond and attach per-test-case data for evaluation.

API

Create, list, update, and delete test sets programmatically.

​
Test case inputs

The Simulation Input is the heart of a test case: it becomes the simulated user’s objective for the run. The input type determines how tightly the simulated user follows it — from improvising toward a high-level goal to delivering exact lines:
TypeWhat it isSimulated user behavior
ScenarioHigh-level intentImprovises freely toward the goal
TranscriptA reference conversationAdapts as needed to match the flow
ScriptExact turnsFollows them precisely, word for word
See Input types for how each behaves and how to configure it. You can also attach an image (WebSocket voice agents only) or a pre-recorded audio file to a test case.
For comprehensive testing, create multiple types of test sets:
  • Regression Set: Contains “happy path” scenarios representing typical successful interactions
  • Adversarial Set: Contains edge cases and scenarios designed to test your agent’s limits and handling of unusual requests

​
Test case vs. persona

A test case defines what the simulated user is trying to accomplish; a persona defines who they are — their tone, accent, and behavior. The same test case can run with any persona, which pursues the test case’s goal in its own style. To make the simulated user attempt a specific (or adversarial) request, put it in the test case’s Simulation Input — the persona only shapes how the user behaves, not what they’re trying to do. A run executes over multiple test cases, but only one persona.

​
Version history

Every config-changing save of a test set is recorded in its version history — capturing both the test-set configuration and a snapshot of its test-case grid — so you can see how the test set changed over time and tell which version a run used. See Versioning for how copy-on-save works and how to pull the history through the v1 API.