> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Improving your LLM Judge
> Calibrate an LLM judge against human labels until its scores mean what they say.
At any real scale, an LLM judge scores your conversations. It reads every transcript and answers a question like *did the agent resolve the issue?* across thousands of calls, faster than any human team could.
LLM judges are not always correct. Without human labeling, you cannot tell an agent failure apart from a judge error, and you can spend weeks fixing an agent that was behaving correctly the whole time.
Calibration closes that gap. You label a small sample of calls yourself, measure how often the judge agrees, and fix it where it does not. Once the judge matches your judgment, the numbers on your dashboard mean what they say, and you can act on them.
## Before you start
You need two things in place:
* **A live LLM judge metric** already running on real calls, so there are verdicts to check.
* **Human review enabled**, so you can record ground-truth labels. See [Human Review Projects](/concepts/metrics/human-review/human-review) for project setup and labeling mechanics.
## The calibration loop
Calibration is a loop you repeat until you trust the judge. Run it in the app, or hand Steps 2 to 4 to the [`review-llm-annotations-and-improve-prompt`](https://github.com/coval-ai/coval-external-skills/tree/main/skills/human-review/review-llm-annotations-and-improve-prompt) skill.
Open a run. For each call in your sample, the transcript sits beside every metric that ran on it, each with the judge's verdict. Agree or disagree: pick **Yes**, **No**, or **N/A**, and optionally add a short note saying why. Every label becomes ground truth.
What labeling looks like depends on who is doing it:
* **Few-shot labeling.** If you are the one in the weeds, browse conversations, filter to the ones you care about, and mark the calls that look wrong. Fast and manual, good for a first read.
* **Project labeling.** For teams and production volumes, a [Human Review project](/concepts/metrics/human-review/how-to-review) gives reviewers a shared queue, tracks progress, and can auto-add new calls, so a full sample gets covered.
Open the metric on the **Metrics** page. The **agreement rate** is how often the judge's verdicts matched your labels, across the calls you labeled. Read it in that context:
* If you did **few-shot labeling** and marked only the calls that looked wrong, the rate starts low and climbs as you fix the metric. It is a starting point, not a trust score.
* If you labeled a **full sample in a project**, the rate is a real measure of how often the judge agrees with you, and you are generally aiming high (around 90%).
There is no single good number. Read the rate against how you labeled, then look at where you and the judge split.
Click into the annotations and read the disagreement rows. This is the highest-signal screen in the loop, because disagreements cluster around a pattern. **Name the pattern before you touch the prompt.**
This is where the skill starts. From here through Step 4, [`review-llm-annotations-and-improve-prompt`](https://github.com/coval-ai/coval-external-skills/tree/main/skills/human-review/review-llm-annotations-and-improve-prompt) can compute the agreement, find the disagreement pattern, and draft the fix for you.
With the pattern named, teach the judge the edge case. There are two ways to do it:
Make the smallest change that captures the pattern. Usually a sentence or two added to the judge's prompt. No model change, no agent change. Change one thing at a time, so the re-verify tells you whether that change worked.
The [`review-llm-annotations-and-improve-prompt`](https://github.com/coval-ai/coval-external-skills/tree/main/skills/human-review/review-llm-annotations-and-improve-prompt) skill does the reading and drafting for you. It pulls your labels and the judge's own reasoning, calculates agreement, finds the pattern in the disagreements, and proposes an improved metric prompt. You stay in the loop: it never changes a metric without your approval.
Paste this into your coding agent:
```text theme={null}
Use the Coval review-llm-annotations-and-improve-prompt skill.
Calibrate a text LLM judge metric against the human labels my team
recorded:
Pull the human ground-truth labels and the judge's verdicts and
reasoning. Calculate the agreement rate, find the pattern in the calls
where the judge and the reviewers disagreed, and propose an improved
metric prompt that fixes that pattern. Show me the proposed prompt and
the before-and-after agreement before changing anything. Do not update
the metric without my approval.
```
You can drive the same workflow yourself from the [Coval CLI](/cli/overview) (the skill runs on `coval metrics get` and `coval metrics update` under the hood), or from the [MCP server](/mcp/overview) if you work in Claude Desktop, Cursor, or another MCP client.
Click **Test Metric**. Coval runs the updated judge against the same calls you already labeled and shows whether the disagreements flip. Old results stay as history, and the improved prompt scores every future run. If agreement is high enough, you are done. If not, read the remaining disagreements and repeat.
## Let's consider an example
A team running an outbound voice agent had a compliance check: *did the agent disclose that it is an AI during the call?* In regulated industries, skipping that disclosure is a legal problem. The metric kept failing calls and it was not clear why.
Labeling the failures (Step 1), then viewing the results (Step 2), showed the judge and the reviewers splitting more than expected. Reading the disagreements, the pattern was clear: in every disputed call, a wrong party picked up or the caller hung up early, so the agent never confirmed who it was talking to and correctly never disclosed anything. The judge did not know that ending was legitimate, so it failed the agent for skipping a step the call never reached.
**The agent was not broken. The metric was.**
The fix (Step 3) was one paragraph added to the judge's prompt: *if the caller is a wrong party, or the call ends before identity verification, the answer is Yes.* On re-verify (Step 4), the calls this judge had been failing came back as passes, matching the human labels.
## Keep in mind
* **Calibrate your other metrics.** Start with the ones tied to release decisions, then work down.
* **Re-calibrate after a change.** A new [agent version](/concepts/versioning/overview) or a prompt edit can shift how calls read, so re-check agreement on a fresh sample.
* **Measure reviewer agreement.** Use an individual (non-collaborative) Human Review project to collect independent labels and see how much your reviewers agree with each other, not just with the judge.
## See also
Measure how much to trust each metric, and tighten the ones that disagree.
The structure behind the prompt you are fixing.
Label programmatically.