Skip to main content
There are three ways to build a test set: build it by hand, use the Coval Test Set Generator, or upload existing test cases from a file.

​
Build manually

Choose Use manual creation mode to start a blank test set and add test cases yourself.

​
Use the Coval Test Set Generator

The Coval Test Set Generator drafts test cases from a description of what you want to test. Open Test Sets in the sidebar and click New Test Set to reach it.
1

Enter your test scenario

Type the scenario you want to test into the input box.
2

(Optional) Add context

Attach files (text, JSON, or markdown), choose an agent to evaluate, or pick a suggested category to sharpen the results.
3

(Optional) Add attributes

Define attributes to generate for each test case — e.g. a ticket_number of format X-### yields X-001, or a destination described as “a possible airport code” yields SFO.
4

Submit and review

Submit with the arrow button or Enter, then review and edit the generated cases in the test set editor.
Be specific in your description, and attach agent prompts or documentation for more relevant tests. You can always edit, add, or remove cases after generation.

​
Upload a CSV or Excel file

Import test cases in bulk from a .csv or .xlsx file. The first row must be column headers (case-insensitive), and only input is required. Knowledge base entries accept any of these formats:
  • JSON array of objects: [{"id": "entry-1", "type": "web_url"}, {"id": "entry-2"}] (type optional)
  • JSON array of IDs: ["entry-1", "entry-2"]
  • Comma-separated id:type (or just id): entry-1:web_url,entry-2,entry-3:pdf
Accepted types: web_url (default), plain_text, json, zendesk, shelf, file.
Files must be under 10 MB and contain at least one row with a valid input value. Empty rows and rows with no input are skipped on import.

​
Create programmatically

You can also create and manage test sets outside the app:

API

Create, list, update, and delete test sets via the REST API.

CLI

Manage test sets from your terminal with the coval CLI.

MCP

Create and list test sets from an MCP client like Claude or Cursor.

SDKs

Manage test sets with the typed TypeScript and Python clients.