Skip to main content
These techniques apply to any LLM judge metric — Binary, Categorical, Numerical, Composite Evaluation, and Audio/Multimodal. The goal is a prompt that produces reliable, deterministic results across repeated evaluations. For the values you can inject into a prompt and the scope it runs against, see Configure metrics.

​
Core principles

Specificity over generality — Define exact evaluation criteria rather than subjective assessments. Use concrete, measurable behaviors instead of abstract concepts, and provide clear boundary conditions for edge cases. Role consistency — Always refer to the AI agent as “the assistant” and to human participants as “the user” or “the customer.” Keep terminology consistent throughout the prompt. Deterministic design — Structure prompts to minimize LLM variance. Provide explicit decision trees where possible and define what constitutes partial vs. complete success.

​
Best practices

  • Be objective. “Did the assistant acknowledge the user’s concern within their first two responses?” is verifiable; “Did the assistant provide good customer service?” is not.
  • Single focus. One metric should measure one thing. Split resolution and professionalism into separate metrics rather than asking “Did the assistant resolve the issue and stay professional?”
  • Clear logic. Use explicit AND/OR and ANY/ALL operators. When using OR conditions, make it explicit that the result applies if any condition is met (e.g., “Return YES if ANY of the following apply”). Make sure your evaluation logic matches the question — a question phrased with “or” but evaluated with “and” produces wrong results.
Use Coval’s Optimize Metric and Test Metric buttons to refine prompt clarity and confidence. Iterate against real test transcripts and aim for >90% consistency across similar evaluations.
The strongest signal for improving an LLM judge prompt is human review. Have reviewers label real transcripts with ground truth, then compare those labels against the metric’s output and tighten the prompt wherever they disagree — the disagreements pinpoint exactly which cases the prompt gets wrong. See Human Reviews.

​
Advanced techniques

For complex evaluations, structure the prompt to guide the model’s reasoning: Chain of thought — Ask the model to reason through intermediate steps before deciding:
Few-shot examples — Anchor edge cases with concrete examples:
Hierarchical decision making — Gate the evaluation through ordered steps so ambiguous cases fall to a conservative default.
Example prompt — Issue resolution detection, using ANY-of logic with explicit YES/NO criteria:

​
Common issues

Keep prompts under ~2,000 characters where possible — shorter, focused prompts are faster and more consistent.

​
Next

Configure metrics

Inject template variables, scope the transcript, and add trace context.

Improve with human review

Use ground-truth labels to find exactly where a prompt is wrong.