Skip to main content
There are three ways to build a test set: build it by hand, use the Coval Test Set Generator, or upload existing test cases from a file.

​
Build manually

Choose Use manual creation mode to start a blank test set and add test cases yourself.

​
Use the Coval Test Set Generator

The Coval Test Set Generator drafts test cases from a description of what you want to test.
1

Enter your test scenario

Type the scenario you want to test into the input box.
2

(Optional) Add context

Attach files (text, JSON, or markdown), choose an agent to evaluate, or pick a suggested category to sharpen the results.
3

(Optional) Add attributes

Define attributes to generate for each test case — e.g. a ticket_number of format X-### yields X-001, or a destination described as “a possible airport code” yields SFO.
4

Submit and review

Submit with the arrow button or Enter, then review and edit the generated cases in the test set editor.
Be specific in your description, and attach agent prompts or documentation for more relevant tests. You can always edit, add, or remove cases after generation.

​
Upload a CSV or Excel file

Import test cases in bulk from a .csv or .xlsx file. The first row must be column headers (case-insensitive), and only input is required.
ColumnRequiredDescription
inputYesThe test case input or prompt
expected_behaviorsNoBehaviors to grade against — a JSON array, a comma-separated string, or a single string
typeNoSCENARIO or TRANSCRIPT
metadataNoA JSON object of test-case attributes
agent_idsNoAgent IDs for the whole test set — a JSON array or comma-separated string (first non-empty value wins)
knowledge_base_entriesNoKnowledge base entries to attach — see formats below
any other columnNoTreated as an attribute
Knowledge base entries accept any of these formats:
  • JSON array of objects: [{"id": "entry-1", "type": "web_url"}, {"id": "entry-2"}] (type optional)
  • JSON array of IDs: ["entry-1", "entry-2"]
  • Comma-separated id:type (or just id): entry-1:web_url,entry-2,entry-3:pdf
Accepted types: web_url (default), plain_text, json, zendesk, shelf, file.
Files must be under 10 MB and contain at least one row with a valid input value. Empty rows and rows with no input are skipped on import.

​
Create programmatically

You can also create and manage test sets outside the app:

API

Create, list, update, and delete test sets via the REST API.

CLI

Manage test sets from your terminal with the coval CLI.

MCP

Create and list test sets from an MCP client like Claude or Cursor.

SDKs

Manage test sets with the typed TypeScript and Python clients.