Human Review Projects

Human review is how you get ground truth from people: reviewers label real conversations so you can trust your metrics and find exactly where they’re wrong. A review project organizes that work — it bundles the metrics to validate, the reviewers, and the conversations to label, then tracks who has labeled what. A project supports two kinds of labeling, and you can use either or both:

• Metric review — reviewers give the ground-truth answer for a metric, so Coval can measure how often the metric agrees with humans. This is what powers improving your metrics.
• Triage — reviewers apply your own labels to sort and categorize conversations (failure type, theme, severity, follow-up). Triage labels are yours to define and aren’t tied to any metric.

​

What’s in a project

Choose Collaborative when you want one authoritative label per conversation and want to divide the work; choose Individual when you want to measure how consistently reviewers agree.

​

Triage categories

If you enable triage on a project, you configure:

• Input mode — whether reviewers pick from predefined categories only, enter custom categories only, or either.
• Selection mode — whether a conversation can take a single category or multiple.
• Categories — the predefined list reviewers choose from.

Reviewers apply these in the review interface; afterward you can sort and group conversations by label.

​

Create a project

​

Keep the queue full with auto-add

Add an auto-add rule to a project and conversations are pulled in automatically whenever their metric results match the conditions you set — for example, every call where a resolution metric returns No. You can require all conditions to be met, scope the rule, and apply a random sampling rate so you label a representative slice rather than everything. Assignees are notified as new work lands, so ground truth keeps building without anyone adding rows by hand. Any conversation can also be annotated ad hoc from its results page — no project required. Projects add assignment, triage labels, progress tracking, and agreement analytics on top.

​

Track progress

The Projects tab shows how labeling is going, with three ways to slice it:

• By conversation — a completed / total count and progress bar for each conversation, so you can see what’s finished, in progress, or untouched.
• By metric — completion per metric, plus an agreement rate: how often the metric matched human ground truth, color-coded so low-agreement metrics stand out. This is your cue to improve that metric.
• By assignee — how many annotations and conversations each reviewer has completed.

Totals account for the project mode — in Individual mode every assignee labels every row, so the target is conversations × reviewers; in Collaborative mode it’s one label per row. Filter the view by review status (Review Complete, In Progress, Pending) or a date range to focus on a batch. You can also track review progress from a dashboard: the Human review widget surfaces assignees and progress across projects alongside your other metrics.

​

Next