Skip to main content
At any real scale, an LLM judge scores your conversations. It reads every transcript and answers a question like did the agent resolve the issue? across thousands of calls, faster than any human team could. LLM judges are not always correct. Without human labeling, you cannot tell an agent failure apart from a judge error, and you can spend weeks fixing an agent that was behaving correctly the whole time. Calibration closes that gap. You label a small sample of calls yourself, measure how often the judge agrees, and fix it where it does not. Once the judge matches your judgment, the numbers on your dashboard mean what they say, and you can act on them.

​
Before you start

You need two things in place:
  • A live LLM judge metric already running on real calls, so there are verdicts to check.
  • Human review enabled, so you can record ground-truth labels. See Human Review Projects for project setup and labeling mechanics.

​
The calibration loop

Calibration is a loop you repeat until you trust the judge. Run it in the app, or hand Steps 2 to 4 to the review-llm-annotations-and-improve-prompt skill.
1

Label a sample of calls

Open a run. For each call in your sample, the transcript sits beside every metric that ran on it, each with the judge’s verdict. Agree or disagree: pick Yes, No, or N/A, and optionally add a short note saying why. Every label becomes ground truth.What labeling looks like depends on who is doing it:
  • Few-shot labeling. If you are the one in the weeds, browse conversations, filter to the ones you care about, and mark the calls that look wrong. Fast and manual, good for a first read.
  • Project labeling. For teams and production volumes, a Human Review project gives reviewers a shared queue, tracks progress, and can auto-add new calls, so a full sample gets covered.
2

View your labeling results

Open the metric on the Metrics page. The agreement rate is how often the judge’s verdicts matched your labels, across the calls you labeled. Read it in that context:
  • If you did few-shot labeling and marked only the calls that looked wrong, the rate starts low and climbs as you fix the metric. It is a starting point, not a trust score.
  • If you labeled a full sample in a project, the rate is a real measure of how often the judge agrees with you, and you are generally aiming high (around 90%).
There is no single good number. Read the rate against how you labeled, then look at where you and the judge split.Click into the annotations and read the disagreement rows. This is the highest-signal screen in the loop, because disagreements cluster around a pattern. Name the pattern before you touch the prompt.
This is where the skill starts. From here through Step 4, review-llm-annotations-and-improve-prompt can compute the agreement, find the disagreement pattern, and draft the fix for you.
3

Fix the metric prompt

With the pattern named, teach the judge the edge case. There are two ways to do it:
Make the smallest change that captures the pattern. Usually a sentence or two added to the judge’s prompt. No model change, no agent change. Change one thing at a time, so the re-verify tells you whether that change worked.
4

Re-verify on the same calls

Click Test Metric. Coval runs the updated judge against the same calls you already labeled and shows whether the disagreements flip. Old results stay as history, and the improved prompt scores every future run. If agreement is high enough, you are done. If not, read the remaining disagreements and repeat.

​
Let’s consider an example

A team running an outbound voice agent had a compliance check: did the agent disclose that it is an AI during the call? In regulated industries, skipping that disclosure is a legal problem. The metric kept failing calls and it was not clear why. Labeling the failures (Step 1), then viewing the results (Step 2), showed the judge and the reviewers splitting more than expected. Reading the disagreements, the pattern was clear: in every disputed call, a wrong party picked up or the caller hung up early, so the agent never confirmed who it was talking to and correctly never disclosed anything. The judge did not know that ending was legitimate, so it failed the agent for skipping a step the call never reached. The agent was not broken. The metric was. The fix (Step 3) was one paragraph added to the judge’s prompt: if the caller is a wrong party, or the call ends before identity verification, the answer is Yes. On re-verify (Step 4), the calls this judge had been failing came back as passes, matching the human labels.

​
Keep in mind

  • Calibrate your other metrics. Start with the ones tied to release decisions, then work down.
  • Re-calibrate after a change. A new agent version or a prompt edit can shift how calls read, so re-check agreement on a fresh sample.
  • Measure reviewer agreement. Use an individual (non-collaborative) Human Review project to collect independent labels and see how much your reviewers agree with each other, not just with the judge.

​
See also

Improving Metrics with Human Review

Measure how much to trust each metric, and tighten the ones that disagree.

Write Judge Prompts

The structure behind the prompt you are fixing.

Human Review via API

Label programmatically.