Human Review

Using Claude Code? We have skills to support human review in your workflow.

Review Projects

List Review Projects

coval review-projects list [OPTIONS]

Option	Type	Default	Description
`--page-size`	number	50	Results per page
`--order-by`	string	—	Sort order (e.g., `-create_time`)

Output columns: ID, NAME, TYPE, ASSIGNEES, SIMULATIONS, METRICS, CREATED

# List all review projects
coval review-projects list

# Sort by most recent
coval review-projects list --order-by "-create_time"

# JSON output
coval review-projects list --format json

Get Review Project

coval review-projects get <project_id>

Argument	Type	Required	Description
`project_id`	string	Yes	The review project ID

Returns full project details as JSON including assignees, linked simulations, and linked metrics.

coval review-projects get 01HXYZ1234567890ABCDEF

Create Review Project

coval review-projects create [OPTIONS]

Option	Type	Required	Description
`--name`	string	Yes	Display name for the project
`--assignees`	string	Yes	Comma-separated reviewer email addresses
`--simulation-ids`	string	Yes	Comma-separated simulation output IDs
`--metric-ids`	string	Yes	Comma-separated metric IDs
`--description`	string	No	Project description
`--type`	string	No	`collaborative` or `individual` (default: `individual`)
`--notifications`	boolean	No	Enable email notifications (default: `true`)

Creating a project auto-generates review annotations for every (simulation, metric, assignee) combination.

Finding your IDs: Run coval metrics list to get metric IDs and coval simulations list to get simulation IDs.

# Create a collaborative review project
coval review-projects create \
  --name "Q1 Voice Agent Review" \
  --assignees "alice@company.com,bob@company.com" \
  --simulation-ids "sim-output-001,sim-output-002" \
  --metric-ids "metric-accuracy,metric-latency" \
  --type collaborative

# Create with description and notifications disabled
coval review-projects create \
  --name "Internal Audit" \
  --assignees "reviewer@company.com" \
  --simulation-ids "sim-output-003" \
  --metric-ids "metric-accuracy" \
  --description "Spot-check accuracy labels" \
  --notifications false

Update Review Project

coval review-projects update <project_id> [OPTIONS]

Argument	Type	Required	Description
`project_id`	string	Yes	The project ID to update

Option	Type	Description
`--name`	string	Updated display name
`--assignees`	string	Updated comma-separated reviewer emails
`--simulation-ids`	string	Updated comma-separated simulation IDs
`--metric-ids`	string	Updated comma-separated metric IDs
`--description`	string	Updated description
`--notifications`	boolean	Updated notification setting

# Add a new assignee
coval review-projects update 01HXYZ1234567890ABCDEF \
  --assignees "alice@company.com,bob@company.com,charlie@company.com"

# Update project name
coval review-projects update 01HXYZ1234567890ABCDEF \
  --name "Q1 Voice Agent Review - Updated"

Delete Review Project

coval review-projects delete <project_id>

Argument	Type	Required	Description
`project_id`	string	Yes	The project ID to delete

coval review-projects delete 01HXYZ1234567890ABCDEF

Review Annotations

List Review Annotations

coval review-annotations list [OPTIONS]

Option	Type	Default	Description
`--filter`	string	—	Filter expression (e.g., `project_id="abc"`)
`--page-size`	number	50	Results per page
`--order-by`	string	—	Sort order (e.g., `-create_time`)

Supported filter fields: simulation_output_id, metric_id, assignee, status (ACTIVE/ARCHIVED), completion_status (PENDING/COMPLETED), project_id Output columns: ID, SIMULATION, METRIC, ASSIGNEE, STATUS, PRIORITY

# List all annotations
coval review-annotations list

# Filter by project
coval review-annotations list --filter 'project_id="01HXYZ1234567890ABCDEF"'

# Filter pending annotations for a specific assignee
coval review-annotations list \
  --filter 'completion_status="PENDING" AND assignee="alice@company.com"'

# JSON output
coval review-annotations list --format json

Get Review Annotation

coval review-annotations get <annotation_id>

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID

Returns full annotation details as JSON including ground-truth values, reviewer notes, and completion status.

coval review-annotations get abc123def456ghi789jklm

Create Review Annotation

coval review-annotations create [OPTIONS]

Option	Type	Required	Description
`--simulation-id`	string	Yes	Simulation output ID to link
`--metric-id`	string	Yes	Metric ID to link
`--assignee`	string	Yes	Reviewer email address
`--ground-truth-float`	number	No	Ground-truth numeric value (auto-completes)
`--ground-truth-string`	string	No	Ground-truth string value (auto-completes)
`--notes`	string	No	Reviewer notes
`--priority`	string	No	`primary` or `standard` (default: `standard`)

# Create a basic annotation
coval review-annotations create \
  --simulation-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com

# Create with ground truth (auto-completes)
coval review-annotations create \
  --simulation-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com \
  --ground-truth-float 0.95 \
  --notes "Verified correct response"

Update Review Annotation

coval review-annotations update <annotation_id> [OPTIONS]

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID to update

Option	Type	Description
`--ground-truth-float`	number	Ground-truth numeric value (auto-completes)
`--ground-truth-string`	string	Ground-truth string value (auto-completes)
`--notes`	string	Reviewer notes
`--assignee`	string	Reassign to a different reviewer
`--priority`	string	`primary` or `standard`

# Submit a ground-truth value
coval review-annotations update abc123def456ghi789jklm \
  --ground-truth-float 0.85 \
  --notes "Agent responded accurately but with slight delay"

# Reassign an annotation
coval review-annotations update abc123def456ghi789jklm \
  --assignee new-reviewer@company.com

Delete Review Annotation

coval review-annotations delete <annotation_id>

Argument	Type	Required	Description
`annotation_id`	string	Yes	The annotation ID to delete

coval review-annotations delete abc123def456ghi789jklm

Completion Statuses

Status	Description
`PENDING`	Annotation has not been reviewed yet
`COMPLETED`	Ground-truth value has been submitted

Annotation Priorities

Priority	Description
`PRIORITY_PRIMARY`	High-priority annotation — surfaces first in reviewer queues
`PRIORITY_STANDARD`	Default priority

Project Types

Type	Description
`collaborative`	All reviewers share a single queue with one annotation per simulation-metric pair
`individual`	Each reviewer gets their own private queue and annotations

Use collaborative projects when building ground-truth datasets. Use individual projects when measuring inter-annotator agreement.

Getting Started

Resources

Review Projects

List Review Projects

Get Review Project

Create Review Project

Update Review Project

Delete Review Project

Review Annotations

List Review Annotations

Get Review Annotation

Create Review Annotation

Update Review Annotation

Delete Review Annotation

Completion Statuses

Annotation Priorities

Project Types

​Review Projects

​List Review Projects

​Get Review Project

​Create Review Project

​Update Review Project

​Delete Review Project

​Review Annotations

​List Review Annotations

​Get Review Annotation

​Create Review Annotation

​Update Review Annotation

​Delete Review Annotation

​Completion Statuses

​Annotation Priorities

​Project Types

Review Projects

List Review Projects

Get Review Project

Create Review Project

Update Review Project

Delete Review Project

Review Annotations

List Review Annotations

Get Review Annotation

Create Review Annotation

Update Review Annotation

Delete Review Annotation

Completion Statuses

Annotation Priorities

Project Types