Skip to main content
A metric is only useful if you can trust its scores. Human review is how you earn that trust: reviewers label real conversations with ground truth, and Coval compares those labels against the metric’s output. The disagreements are the signal — they pinpoint the exact conversations the metric gets wrong, which is the strongest guide you have for improving it. This page assumes you’ve already collected labels. To set up a project and gather them, see Human Review Projects; for the reviewer’s side, see How to Review a Conversation.
Human review is supported for a subset of metric types — see the metric types reviewers can label.

​
Agreement insights

Once labels come in, Coval turns them into diagnostics:
  • Per-metric agreement rate — how often the metric matched human ground truth, with a drill-down of the exact conversations that disagreed.
  • Inter-annotator agreement — for Individual projects, how consistently your reviewers agree with each other. Low human agreement usually means the metric’s definition is ambiguous — fix the criteria before touching the prompt.
  • Agreement on the metric page — each reviewed metric shows its agreement stats where you edit it, so you can judge and improve it in one place.

​
The improvement loop

1

Label

Reviewers provide ground truth through a review project.
2

Measure

Check the metric’s agreement rate against the human labels.
3

Diagnose

Read the disagreeing conversations and reviewer notes to see what the metric misses.
4

Revise

Tighten the prompt for exactly those cases. Open the metric in the Metrics tab, draft a new version in the prompt box, and click Test Metric to run it against your labeled conversations.
5

Re-test

Confirm agreement improved on the ground-truth set before you ship the new version.
Metric Details
Repeat the loop until agreement plateaus. A metric validated this way becomes a trustworthy, automated stand-in for human judgment on every future run — which is the whole point of the continuous quality loop.