Human Review

Using Claude Code? We have skills to support human review in your workflow.

create and update also accept --input-json (inline JSON, @file, or - for stdin); explicit flags override fields from the JSON. See Input JSON.

Review Projects

List Review Projects

coval review-projects list [OPTIONS]

Option	Type	Default	Description
`--filter`	string	—	Filter expression
`--page-size`	number	50	Results per page
`--order-by`	string	—	Sort order (e.g., `-create_time`)

Output columns: ID, NAME, TYPE, ASSIGNEES, SIMULATIONS, METRICS, CREATED

# List all review projects
coval review-projects list

# Sort by most recent
coval review-projects list --order-by "-create_time"

# JSON output
coval review-projects list --format json

Get Review Project

coval review-projects get <project_id>

Argument	Type	Required	Description
`project_id`	string	Yes	The review project ID

Returns full project details as JSON including assignees, linked simulations, and linked metrics.

coval review-projects get 01HXYZ1234567890ABCDEF

Create Review Project

coval review-projects create [OPTIONS]

The CLI does not enforce any flag on its own, because every field can also arrive through --input-json. The API requires a name, assignees, simulation IDs, and metric IDs.

Option	Type	Required	Description
`--name`	string	No	Project name
`--assignees`	string	No	Comma-separated reviewer emails
`--simulation-ids`	string	No	Comma-separated simulation output IDs
`--metric-ids`	string	No	Comma-separated metric IDs
`--description`	string	No	Optional project description
`--project-type`	string	No	`collaborative` or `individual`
`--notifications`	boolean	No	`true` or `false` — enable notifications for assignees
`--project-rules`	string	No	Comma-separated project rules (currently `require_disagreement_notes`)
`--blind-labeling-shown-metric-ids`	string	No	Comma-separated metric IDs whose machine score stays visible during blind labeling
`--enforced-collaboration[=true\|false]`	boolean	No	Require claims and explicit single-author completion (collaborative projects only); passing the flag without a value enables it

Creating a project auto-generates review annotations for every (simulation, metric, assignee) combination.

Finding your IDs: Run coval metrics list to get metric IDs and coval simulated-conversations list to get simulation IDs.

# Create a collaborative review project
coval review-projects create \
  --name "Q1 Voice Agent Review" \
  --assignees "alice@company.com,bob@company.com" \
  --simulation-ids "sim-output-001,sim-output-002" \
  --metric-ids "metric-accuracy,metric-latency" \
  --project-type collaborative \
  --enforced-collaboration

# Create with description, disagreement notes, and notifications disabled
coval review-projects create \
  --name "Internal Audit" \
  --assignees "reviewer@company.com" \
  --simulation-ids "sim-output-003" \
  --metric-ids "metric-accuracy" \
  --description "Spot-check accuracy labels" \
  --project-rules require_disagreement_notes \
  --notifications false

Update Review Project

coval review-projects update <project_id> [OPTIONS]

Argument	Type	Required	Description
`project_id`	string	Yes	The project ID to update

Option	Type	Description
`--name`	string	Updated project name
`--description`	string	Updated description
`--assignees`	string	Comma-separated updated reviewer emails
`--opted-out-assignees`	string	Comma-separated emails of assignees to opt out of notifications
`--simulation-ids`	string	Comma-separated simulation output IDs; replaces the linked set
`--add-simulation-ids`	string	Comma-separated simulation output IDs to add instead of replacing the linked set; conflicts with `--simulation-ids`
`--remove-simulation-ids`	string	Comma-separated simulation output IDs to remove instead of replacing the linked set; conflicts with `--simulation-ids`
`--metric-ids`	string	Comma-separated updated metric IDs
`--metric-addition-completion-action`	string	When linking a new metric, what to do with completed assignments: `KEEP_COMPLETED` or `REOPEN_COMPLETED`
`--notifications`	boolean	`true` or `false` — updated notification setting
`--project-rules`	string	Comma-separated project rules
`--blind-labeling-shown-metric-ids`	string	Comma-separated metric IDs whose machine score stays visible during blind labeling
`--enforced-collaboration[=true\|false]`	boolean	Require claims and explicit single-author completion (collaborative projects only)

# Add a new assignee
coval review-projects update 01HXYZ1234567890ABCDEF \
  --assignees "alice@company.com,bob@company.com,charlie@company.com"

# Add simulations without touching the ones already linked
coval review-projects update 01HXYZ1234567890ABCDEF \
  --add-simulation-ids "sim-output-004,sim-output-005"

# Link another metric and reopen completed assignments so it gets reviewed
coval review-projects update 01HXYZ1234567890ABCDEF \
  --metric-ids "metric-accuracy,metric-latency,metric-tone" \
  --metric-addition-completion-action REOPEN_COMPLETED

Delete Review Project

coval review-projects delete <project_id>

Argument	Type	Required	Description
`project_id`	string	Yes	The project ID to delete

coval review-projects delete 01HXYZ1234567890ABCDEF

Review Annotations

List Review Annotations

coval review-annotations list [OPTIONS]

Option	Type	Default	Description
`--filter`	string	—	Filter expression (e.g., `project_id="abc"`)
`--page-size`	number	50	Results per page
`--order-by`	string	—	Sort order (e.g., `-create_time`)

Supported filter fields: simulation_output_id, metric_id, assignee, status (ACTIVE/ARCHIVED), completion_status (PENDING/COMPLETED), project_id Output columns: ID, SIMULATION, METRIC, ASSIGNEE, STATUS, PRIORITY

# List all annotations
coval review-annotations list

# Filter by project
coval review-annotations list --filter 'project_id="01HXYZ1234567890ABCDEF"'

# Filter pending annotations for a specific assignee
coval review-annotations list \
  --filter 'completion_status="PENDING" AND assignee="alice@company.com"'

# JSON output
coval review-annotations list --format json

Get Review Annotation

coval review-annotations get <annotation_id>

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID

Returns full annotation details as JSON including ground-truth values, reviewer notes, and completion status.

coval review-annotations get abc123def456ghi789jklm

Create Review Annotation

coval review-annotations create [OPTIONS]

The CLI does not enforce any flag on its own, because every field can also arrive through --input-json. The API requires a simulation output ID, a metric ID, and an assignee.

Option	Type	Required	Description
`--simulation-output-id`	string	No	Simulation output ID to link
`--metric-id`	string	No	Metric ID to link
`--assignee`	string	No	Email of the reviewer to assign
`--ground-truth-float`	number	No	Ground-truth numeric value (auto-completes the annotation)
`--ground-truth-string`	string	No	Ground-truth string value (auto-completes the annotation)
`--ground-truth-subvalues`	JSON	No	Ground-truth subvalues by timestamp, as a JSON array (auto-completes the annotation)
`--notes`	string	No	Reviewer notes
`--priority`	string	No	`primary` or `standard`

# Create a basic annotation
coval review-annotations create \
  --simulation-output-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com

# Create with ground truth (auto-completes)
coval review-annotations create \
  --simulation-output-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com \
  --ground-truth-float 0.95 \
  --notes "Verified correct response"

Update Review Annotation

coval review-annotations update <annotation_id> [OPTIONS]

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID to update

Option	Type	Description
`--ground-truth-float`	number	Ground-truth numeric value (auto-completes the annotation)
`--ground-truth-string`	string	Ground-truth string value (auto-completes the annotation)
`--ground-truth-subvalues`	JSON	Ground-truth subvalues by timestamp, as a JSON array (auto-completes the annotation)
`--notes`	string	Reviewer notes
`--priority`	string	`primary` or `standard`
`--assignee`	string	Reassign to a different reviewer
`--completion-status`	string	`pending` or `completed`
`--status`	string	`active` or `archived`

# Submit a ground-truth value
coval review-annotations update abc123def456ghi789jklm \
  --ground-truth-float 0.85 \
  --notes "Agent responded accurately but with slight delay"

# Reassign an annotation
coval review-annotations update abc123def456ghi789jklm \
  --assignee new-reviewer@company.com

# Reopen a completed annotation
coval review-annotations update abc123def456ghi789jklm --completion-status pending

# Archive an annotation
coval review-annotations update abc123def456ghi789jklm --status archived

Delete Review Annotation

coval review-annotations delete <annotation_id>

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID to delete

coval review-annotations delete abc123def456ghi789jklm

Completion Statuses

Status	Description
`PENDING`	Annotation has not been reviewed yet
`COMPLETED`	Ground-truth value has been submitted

Annotation Priorities

The CLI accepts --priority primary or --priority standard; the API stores and filters on the values below.

Priority	Description
`PRIORITY_PRIMARY`	High-priority annotation — surfaces first in reviewer queues
`PRIORITY_STANDARD`	Default priority

Project Types

Type	Description
`collaborative`	All reviewers share a single queue with one annotation per simulation-metric pair
`individual`	Each reviewer gets their own private queue and annotations

Use collaborative projects when building ground-truth datasets. Use individual projects when measuring inter-annotator agreement.

Getting Started

Resources

Review Projects

List Review Projects

Get Review Project

Create Review Project

Update Review Project

Delete Review Project

Review Annotations

List Review Annotations

Get Review Annotation

Create Review Annotation

Update Review Annotation

Delete Review Annotation

Completion Statuses

Annotation Priorities

Project Types

​Review Projects

​List Review Projects

​Get Review Project

​Create Review Project

​Update Review Project

​Delete Review Project

​Review Annotations

​List Review Annotations

​Get Review Annotation

​Create Review Annotation

​Update Review Annotation

​Delete Review Annotation

​Completion Statuses

​Annotation Priorities

​Project Types

Review Projects

List Review Projects

Get Review Project

Create Review Project

Update Review Project

Delete Review Project

Review Annotations

List Review Annotations

Get Review Annotation

Create Review Annotation

Update Review Annotation

Delete Review Annotation

Completion Statuses

Annotation Priorities

Project Types