> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Human Review

> Manage human review projects and annotations with the Coval CLI

<Tip>
  Using Claude Code? We have [skills to support human review](https://github.com/coval-ai/coval-external-skills/tree/main/skills/human-review) in your workflow.
</Tip>

## Review Projects

### List Review Projects

```bash theme={null}
coval review-projects list [OPTIONS]
```

| Option        | Type   | Default | Description                       |
| ------------- | ------ | ------- | --------------------------------- |
| `--page-size` | number | 50      | Results per page                  |
| `--order-by`  | string | —       | Sort order (e.g., `-create_time`) |

**Output columns:** ID, NAME, TYPE, ASSIGNEES, SIMULATIONS, METRICS, CREATED

```bash theme={null}
# List all review projects
coval review-projects list

# Sort by most recent
coval review-projects list --order-by "-create_time"

# JSON output
coval review-projects list --format json
```

### Get Review Project

```bash theme={null}
coval review-projects get <project_id>
```

| Argument     | Type   | Required | Description           |
| ------------ | ------ | -------- | --------------------- |
| `project_id` | string | **Yes**  | The review project ID |

Returns full project details as JSON including assignees, linked simulations, and linked metrics.

```bash theme={null}
coval review-projects get 01HXYZ1234567890ABCDEF
```

### Create Review Project

```bash theme={null}
coval review-projects create [OPTIONS]
```

| Option             | Type    | Required | Description                                             |
| ------------------ | ------- | -------- | ------------------------------------------------------- |
| `--name`           | string  | **Yes**  | Display name for the project                            |
| `--assignees`      | string  | **Yes**  | Comma-separated reviewer email addresses                |
| `--simulation-ids` | string  | **Yes**  | Comma-separated simulation output IDs                   |
| `--metric-ids`     | string  | **Yes**  | Comma-separated metric IDs                              |
| `--description`    | string  | No       | Project description                                     |
| `--type`           | string  | No       | `collaborative` or `individual` (default: `individual`) |
| `--notifications`  | boolean | No       | Enable email notifications (default: `true`)            |

<Info>
  Creating a project auto-generates review annotations for every (simulation, metric, assignee) combination.
</Info>

<Info>
  **Finding your IDs:** Run `coval metrics list` to get metric IDs and `coval simulations list` to get simulation IDs.
</Info>

```bash theme={null}
# Create a collaborative review project
coval review-projects create \
  --name "Q1 Voice Agent Review" \
  --assignees "alice@company.com,bob@company.com" \
  --simulation-ids "sim-output-001,sim-output-002" \
  --metric-ids "metric-accuracy,metric-latency" \
  --type collaborative

# Create with description and notifications disabled
coval review-projects create \
  --name "Internal Audit" \
  --assignees "reviewer@company.com" \
  --simulation-ids "sim-output-003" \
  --metric-ids "metric-accuracy" \
  --description "Spot-check accuracy labels" \
  --notifications false
```

### Update Review Project

```bash theme={null}
coval review-projects update <project_id> [OPTIONS]
```

| Argument     | Type   | Required | Description              |
| ------------ | ------ | -------- | ------------------------ |
| `project_id` | string | **Yes**  | The project ID to update |

| Option             | Type    | Description                             |
| ------------------ | ------- | --------------------------------------- |
| `--name`           | string  | Updated display name                    |
| `--assignees`      | string  | Updated comma-separated reviewer emails |
| `--simulation-ids` | string  | Updated comma-separated simulation IDs  |
| `--metric-ids`     | string  | Updated comma-separated metric IDs      |
| `--description`    | string  | Updated description                     |
| `--notifications`  | boolean | Updated notification setting            |

```bash theme={null}
# Add a new assignee
coval review-projects update 01HXYZ1234567890ABCDEF \
  --assignees "alice@company.com,bob@company.com,charlie@company.com"

# Update project name
coval review-projects update 01HXYZ1234567890ABCDEF \
  --name "Q1 Voice Agent Review - Updated"
```

### Delete Review Project

```bash theme={null}
coval review-projects delete <project_id>
```

| Argument     | Type   | Required | Description              |
| ------------ | ------ | -------- | ------------------------ |
| `project_id` | string | **Yes**  | The project ID to delete |

```bash theme={null}
coval review-projects delete 01HXYZ1234567890ABCDEF
```

***

## Review Annotations

### List Review Annotations

```bash theme={null}
coval review-annotations list [OPTIONS]
```

| Option        | Type   | Default | Description                                  |
| ------------- | ------ | ------- | -------------------------------------------- |
| `--filter`    | string | —       | Filter expression (e.g., `project_id="abc"`) |
| `--page-size` | number | 50      | Results per page                             |
| `--order-by`  | string | —       | Sort order (e.g., `-create_time`)            |

**Supported filter fields:** `simulation_output_id`, `metric_id`, `assignee`, `status` (`ACTIVE`/`ARCHIVED`), `completion_status` (`PENDING`/`COMPLETED`), `project_id`

**Output columns:** ID, SIMULATION, METRIC, ASSIGNEE, STATUS, PRIORITY

```bash theme={null}
# List all annotations
coval review-annotations list

# Filter by project
coval review-annotations list --filter 'project_id="01HXYZ1234567890ABCDEF"'

# Filter pending annotations for a specific assignee
coval review-annotations list \
  --filter 'completion_status="PENDING" AND assignee="alice@company.com"'

# JSON output
coval review-annotations list --format json
```

### Get Review Annotation

```bash theme={null}
coval review-annotations get <annotation_id>
```

| Argument        | Type   | Required | Description       |
| --------------- | ------ | -------- | ----------------- |
| `annotation_id` | string | **Yes**  | The annotation ID |

Returns full annotation details as JSON including ground-truth values, reviewer notes, and completion status.

```bash theme={null}
coval review-annotations get abc123def456ghi789jklm
```

### Create Review Annotation

```bash theme={null}
coval review-annotations create [OPTIONS]
```

| Option                  | Type   | Required | Description                                   |
| ----------------------- | ------ | -------- | --------------------------------------------- |
| `--simulation-id`       | string | **Yes**  | Simulation output ID to link                  |
| `--metric-id`           | string | **Yes**  | Metric ID to link                             |
| `--assignee`            | string | **Yes**  | Reviewer email address                        |
| `--ground-truth-float`  | number | No       | Ground-truth numeric value (auto-completes)   |
| `--ground-truth-string` | string | No       | Ground-truth string value (auto-completes)    |
| `--notes`               | string | No       | Reviewer notes                                |
| `--priority`            | string | No       | `primary` or `standard` (default: `standard`) |

```bash theme={null}
# Create a basic annotation
coval review-annotations create \
  --simulation-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com

# Create with ground truth (auto-completes)
coval review-annotations create \
  --simulation-id sim-output-abc123 \
  --metric-id metric-accuracy-001 \
  --assignee reviewer@company.com \
  --ground-truth-float 0.95 \
  --notes "Verified correct response"
```

### Update Review Annotation

```bash theme={null}
coval review-annotations update <annotation_id> [OPTIONS]
```

| Argument        | Type   | Required | Description                 |
| --------------- | ------ | -------- | --------------------------- |
| `annotation_id` | string | **Yes**  | The annotation ID to update |

| Option                  | Type   | Description                                 |
| ----------------------- | ------ | ------------------------------------------- |
| `--ground-truth-float`  | number | Ground-truth numeric value (auto-completes) |
| `--ground-truth-string` | string | Ground-truth string value (auto-completes)  |
| `--notes`               | string | Reviewer notes                              |
| `--assignee`            | string | Reassign to a different reviewer            |
| `--priority`            | string | `primary` or `standard`                     |

```bash theme={null}
# Submit a ground-truth value
coval review-annotations update abc123def456ghi789jklm \
  --ground-truth-float 0.85 \
  --notes "Agent responded accurately but with slight delay"

# Reassign an annotation
coval review-annotations update abc123def456ghi789jklm \
  --assignee new-reviewer@company.com
```

### Delete Review Annotation

```bash theme={null}
coval review-annotations delete <annotation_id>
```

| Argument        | Type   | Required | Description                 |
| --------------- | ------ | -------- | --------------------------- |
| `annotation_id` | string | **Yes**  | The annotation ID to delete |

```bash theme={null}
coval review-annotations delete abc123def456ghi789jklm
```

***

## Completion Statuses

| Status      | Description                           |
| ----------- | ------------------------------------- |
| `PENDING`   | Annotation has not been reviewed yet  |
| `COMPLETED` | Ground-truth value has been submitted |

## Annotation Priorities

| Priority            | Description                                                  |
| ------------------- | ------------------------------------------------------------ |
| `PRIORITY_PRIMARY`  | High-priority annotation — surfaces first in reviewer queues |
| `PRIORITY_STANDARD` | Default priority                                             |

## Project Types

| Type            | Description                                                                       |
| --------------- | --------------------------------------------------------------------------------- |
| `collaborative` | All reviewers share a single queue with one annotation per simulation-metric pair |
| `individual`    | Each reviewer gets their own private queue and annotations                        |

<Tip>
  Use `collaborative` projects when building ground-truth datasets. Use `individual` projects when measuring inter-annotator agreement.
</Tip>