Tools Reference - Coval Documentation

Sofia

consult_sofia

Delegate a read-only evaluation question to Sofia. Sofia can use Coval product knowledge and inspect the authenticated organization’s runs, simulations, conversations, metrics, agents, personas, test sets, and dashboards. It cannot create, update, run, or delete resources.

Parameter	Type	Required	Description
`prompt`	string	Yes	Question or task for Sofia, up to 12,000 characters
`conversation`	object[]	No	Up to 12 prior messages needed for context
`session_id`	string	No	Stable, non-secret caller session identifier, up to 128 letters, numbers, `.`, `_`, `:`, or `-` characters

In ChatGPT, consult_sofia takes a single standalone prompt of up to 4,000 characters and no conversation context. Include the exact facts the question needs in the prompt itself. Other MCP clients use the full parameters below.

Each conversation item must contain exactly the message content needed for the current request:

Field	Type	Required	Limits
`role`	string	Yes	`user` or `assistant`
`content`	string	Yes	Non-empty text, up to 12,000 characters

Do not include API keys, access tokens, or other secrets in conversation context or session_id. For service reliability and evaluation, Coval records the prompt, the optional conversation messages supplied to this tool, and Sofia’s response. Messages elsewhere in the parent Claude, Codex, or other client session are not sent to Coval unless the client includes them in this tool call. Use direct tools for deterministic reads and writes. Use consult_sofia when the task benefits from evaluation expertise or synthesis across several Coval resources.

Run Management

list_runs

List evaluation runs with filtering and pagination.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token from previous response
`order_by`	string	No	Sort order (e.g., `-create_time` for newest first)
`filter`	string	No	Filter expression (e.g., `status="COMPLETED"`)

get_run

Get detailed information about a specific run.

Parameter	Type	Required	Description
`run_id`	string	Yes	The unique run ID

Returns run details including status, progress, and metrics (if completed).

create_run

Launch a new evaluation run.

Parameter	Type	Required	Description
`agent_id`	string	Yes	Agent ID from `list_agents`
`persona_id`	string	Yes	Persona ID from `list_personas`
`test_set_id`	string	Yes	Test set ID from `list_test_sets`
`metric_ids`	string[]	No	Specific metrics to evaluate
`options.iteration_count`	number	No	Iterations per test case (1-10, default: 1)
`options.concurrency`	number	No	Parallel simulations (1-5, default: 1)
`tags`	string[]	No	Up to 20 tags, 200 characters each
`metadata`	object	No	Custom metadata for tracking

Agent Management

list_agents

List all configured agents.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token
`order_by`	string	No	Sort order
`filter`	string	No	Filter by `model_type`, `display_name`, etc.

get_agent

Get detailed configuration for a specific agent.

Parameter	Type	Required	Description
`agent_id`	string	Yes	Agent ID from `list_agents`

create_agent

Create a new agent configuration.

Parameter	Type	Required	Description
`display_name`	string	Yes	Human-readable name (1-200 chars)
`model_type`	string	Yes	Agent type (see below)
`phone_number`	string	No	E.164 format for voice agents
`endpoint`	string	No	Webhook or WebSocket URL
`prompt`	string	No	System prompt/instructions
`metadata`	object	No	Custom metadata

Model Types:

MODEL_TYPE_VOICE - Inbound voice
MODEL_TYPE_OUTBOUND_VOICE - Outbound voice
MODEL_TYPE_CHAT - Chat/text
MODEL_TYPE_SMS - SMS messaging
MODEL_TYPE_WEBSOCKET - WebSocket

update_agent

Update an existing agent configuration.

Parameter	Type	Required	Description
`agent_id`	string	Yes	Agent to update
`display_name`	string	No	New name
`phone_number`	string	No	New phone number
`endpoint`	string	No	New endpoint URL
`prompt`	string	No	New system prompt
`metadata`	object	No	New metadata

Test Set Management

list_test_sets

List all test sets available for evaluation.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token
`order_by`	string	No	Sort order
`filter`	string	No	Filter expression

get_test_set

Get detailed information about a test set.

Parameter	Type	Required	Description
`test_set_id`	string	Yes	Test set ID from `list_test_sets`

create_test_set

Create a new test set.

Parameter	Type	Required	Description
`display_name`	string	Yes	Test set name (1-100 chars)
`slug`	string	No	URL-friendly ID (auto-generated if omitted)
`description`	string	No	Test set description
`test_set_type`	string	No	`DEFAULT`, `SCENARIO`, `TRANSCRIPT`, or `WORKFLOW`
`test_set_metadata`	object	No	Configuration metadata
`parameters`	object	No	Test parameterization

Test Case Management

list_test_cases

List test cases with optional filtering by test set.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token
`order_by`	string	No	Sort order
`filter`	string	No	Filter expression, for example `test_set_id="abc12345"`

get_test_case

Get detailed information about a test case.

Parameter	Type	Required	Description
`test_case_id`	string	Yes	Test case ID from `list_test_cases`

create_test_case

Create a new test case in a test set.

Parameter	Type	Required	Description
`test_set_id`	string	Yes	Eight-character test set ID
`input_str`	string	Yes	Scenario or input presented to the agent
`expected_behaviors`	string[]	No	Behaviors expected from the agent
`description`	string	No	Test case description
`input_type`	string	No	`SCENARIO` (default), `TRANSCRIPT`, `IVR`, `AUDIO`, `MANUAL`, or `SCRIPT`
`simulation_metadata_input`	object	No	Context passed to the simulation environment. For `SCRIPT` cases, put the ordered turns in `script_turns` (see below)
`metric_input`	object	No	Custom metric inputs
`user_notes`	string	No	Internal notes

Scripted test cases. With input_type set to SCRIPT, the simulated user follows an exact sequence of turns instead of improvising from a scenario. Put the turns, in order, in simulation_metadata_input.script_turns. Each entry is either a plain string the simulated user speaks verbatim, a keypad press {"type": "dtmf", "digits": "1"} (digits may contain 0-9, *, #, and phone punctuation for full numbers), or {"type": "skip"} to stay silent for one turn.

{
  "test_set_id": "abc12345",
  "input_type": "SCRIPT",
  "input_str": "Navigate the billing IVR to a live agent",
  "simulation_metadata_input": {
    "script_turns": [
      "Hi, I have a question about my bill.",
      { "type": "dtmf", "digits": "2" },
      "I'd like to speak to someone about a duplicate charge."
    ]
  }
}

Never write a keypad press as spoken text such as dtmf:1 or “press one”: the simulated user would read it aloud. Use a {"type": "dtmf", "digits": "..."} turn. For SCENARIO cases against a phone tree, instruct the simulated user in input_str to press keys instead.

update_test_case

Update an existing test case.

Parameter	Type	Required	Description
`test_case_id`	string	Yes	Test case to update
`input_str`	string	No	Updated scenario or input
`expected_behaviors`	string[]	No	Updated expected behaviors
`description`	string	No	New description
`input_type`	string	No	Updated input type. Switching away from `SCRIPT` clears the stored turns
`script_turns`	array	No	Replacement turn list for `SCRIPT` cases, using the same entry shapes as `create_test_case`. Merged into the stored simulation metadata, so sibling keys are preserved
`simulation_metadata_input`	object	No	Updated simulation context. Replaces the stored object wholesale, so include every key you want to keep
`metric_input`	object	No	Updated metric inputs
`user_notes`	string	No	Updated internal notes

To change the turns of a scripted case, prefer the top-level script_turns field. If you send both script_turns and simulation_metadata_input, the object is replaced first and the turns are then merged into the result, so script_turns always wins for that key.

Metrics

list_metrics

List available evaluation metrics.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token
`order_by`	string	No	Sort order
`filter`	string	No	Filter expression
`include_builtin`	boolean	No	Include built-in metrics

get_metric

Get detailed configuration for a specific metric.

Parameter	Type	Required	Description
`metric_id`	string	Yes	Metric ID from `list_metrics`

Reports

list_reports

List saved reports in the organization.

Parameter	Type	Required	Description
`page_size`	number	No	Reports per page (1-100, default: 50)
`page_token`	string	No	Pagination token from previous response

get_report

Get one saved report with a bounded page of its result rows.

Parameter	Type	Required	Description
`report_id`	string	Yes	Report ID from `list_reports`
`page_size`	number	No	Result rows per page (1-100, default: 20)
`page_token`	string	No	Pagination token from previous response
`metric_ids`	string[]	No	Limit row data to these metrics (up to 20)

Row data is trimmed to fit the response budget. When the response flags that output was trimmed, narrow it with metric_ids and a smaller page_size, then continue with next_page_token. Check scope.has_more before treating the report as complete.

create_report

Save a report over a set of runs. Reports created through the connector are private to your organization.

Parameter	Type	Required	Description
`name`	string	Yes	Report name (1-200 chars)
`run_ids`	string[]	Yes	Runs to include (1-200)
`compare_by`	string	No	`none`, `run`, `agent`, `mutation`, `persona`, `test_case`, or `metadata`
`metadata_key`	string	No	Metadata key to compare by. Required when `compare_by` is `metadata`, and not allowed otherwise
`view_mode`	string	No	`rows` or `grouped`

Scheduling

list_run_templates

List reusable run configurations that a schedule can execute.

Parameter	Type	Required	Description
`page_size`	number	No	Templates per page (1-100, default: 50)
`page_token`	string	No	Pagination token from previous response

list_scheduled_runs

List recurring evaluation schedules.

Parameter	Type	Required	Description
`page_size`	number	No	Schedules per page (1-100, default: 50)
`page_token`	string	No	Pagination token from previous response
`enabled`	boolean	No	Only enabled (`true`) or disabled (`false`) schedules
`template_id`	string	No	Only schedules that use this run template

get_scheduled_run

Get one schedule and a page of the runs it has triggered.

Parameter	Type	Required	Description
`scheduled_run_id`	string	Yes	Schedule ID from `list_scheduled_runs`
`history_size`	number	No	Triggered runs per page (1-100, default: 20)
`history_page_token`	string	No	Pagination token from `next_history_page_token`

History is served from the schedule’s most recent 500 runs. history_scope.upstream_capped is set when older runs exist beyond that window.

create_scheduled_run

Create a recurring evaluation schedule from a run template.

Parameter	Type	Required	Description
`display_name`	string	Yes	Schedule name (1-200 chars)
`run_template_id`	string	Yes	Run template from `list_run_templates`
`schedule_expression`	string	Yes	`rate(...)` or six-field `cron(...)` expression, for example `rate(1 day)` or `cron(0 9 ? * MON-FRI *)`. Five-field cron is not accepted
`schedule_timezone`	string	No	Timezone for cron schedules, such as `UTC` or `America/Los_Angeles`. Required when `schedule_expression` uses `cron(...)`
`enabled`	boolean	No	Activate the schedule immediately (default: `false`)

New schedules are created disabled. Enabling a schedule starts recurring evaluation work, which consumes simulation capacity on every trigger until you disable it.

update_scheduled_run

Update selected fields on an existing schedule. Provide at least one field besides the ID.

Parameter	Type	Required	Description
`scheduled_run_id`	string	Yes	Schedule to update
`display_name`	string	No	New name (1-200 chars)
`run_template_id`	string	No	Run template to execute
`schedule_expression`	string	No	New `rate(...)` or six-field `cron(...)` expression
`schedule_timezone`	string	No	Timezone for cron schedules. Required when changing to a `cron(...)` expression
`enabled`	boolean	No	Enable or disable the schedule

Personas

list_personas

List available simulated personas for testing.

Parameter	Type	Required	Description
`page_size`	number	No	Results per page (1-100, default: 50)
`page_token`	string	No	Pagination token
`order_by`	string	No	Sort order
`filter`	string	No	Filter expression

get_persona

Get detailed configuration for a specific persona.

Parameter	Type	Required	Description
`persona_id`	string	Yes	Persona ID from `list_personas`

Returns persona configuration including voice settings, language, and behavior.

​Sofia

​consult_sofia

​Run Management

​list_runs

​get_run

​create_run

​Agent Management

​list_agents

​get_agent

​create_agent

​update_agent

​Test Set Management

​list_test_sets

​get_test_set

​create_test_set

​Test Case Management

​list_test_cases

​get_test_case

​create_test_case

​update_test_case

​Metrics

​list_metrics

​get_metric

​Reports

​list_reports

​get_report

​create_report

​Scheduling

​list_run_templates

​list_scheduled_runs

​get_scheduled_run

​create_scheduled_run

​update_scheduled_run

​Personas

​list_personas

​get_persona

Sofia

consult_sofia

Run Management

list_runs

get_run

create_run

Agent Management

list_agents

get_agent

create_agent

update_agent

Test Set Management

list_test_sets

get_test_set

create_test_set

Test Case Management

list_test_cases

get_test_case

create_test_case

update_test_case

Metrics

list_metrics

get_metric

Reports

list_reports

get_report

create_report

Scheduling

list_run_templates

list_scheduled_runs

get_scheduled_run

create_scheduled_run

update_scheduled_run

Personas

list_personas

get_persona