| Metric | Tells you | Unit | Scope |
|---|---|---|---|
| Agent Fails to Respond | Agent went silent when it owed a response | YES / NO | — |
| Agent Needs Reprompting | User had to re-prompt a silent agent | count | Audio |
| API State | External system state matches expectations | MATCH / DIFF / NOT_FOUND | — |
| End Reason | How the conversation ended | end reason | — |
| Match Expected Output | Captured observation matches an expected value | MATCH / DIFF / NOT_FOUND | — |
| Metadata Field | A custom metadata value, surfaced as a metric | follows field | — |
| Music Detection | Hold or background music segments | count | — |
| Transcript Regex Match | Pattern present or absent in the transcript | YES / NO | — |
| Words per Message (Threshold) | All agent messages meet a word-count rule | YES / NO / UNKNOWN | — |
Agent Fails to Respond
Detects whether the agent went silent after the user’s last turn when it owed a response, distinguishing a missed response from a call that simply ended. What it measures — Whether the agent failed to respond following the user’s last turn — the user asked a question, made a request, or supplied information that needed acknowledgement — rather than the call simply ending naturally. When to use — Catching critical responsiveness failures, such as the agent dropping out right after the user asks a final question or supplies requested information. Any occurrence is worth investigating. How it works — A trailing-silence check on the call’s speaking segments flags candidates: if the recording ends with user speech followed by silence beyond the threshold (measured against the audio duration), the simulation is flagged. An LLM judge then reviews the transcript to confirm the agent actually owed a response, rather than the call having concluded naturally (e.g., the user said “thanks, goodbye”). How to interpret — YES means the agent failed to respond — a critical responsiveness failure worth immediate investigation; the flagged silence is marked on the timeline with the judge’s explanation. NO means it responded as expected or the conversation had already concluded. Configuration — None for the base metric. A configurable variant exposes the trailing-silence threshold (seconds) before a candidate is flagged. Requires Audio · Unit YES / NOAgent Needs Reprompting
Counts how often the user had to speak again because the agent stayed silent too long. Unlike Agent Fails to Respond, the agent does eventually answer — but only after the user repeats themselves. What it measures — The number of times the user re-prompted because the agent failed to pick up its turn within the silence gap. When to use — Evaluating conversational continuity and naturalness — for example, spotting intermittent failures in the agent’s response pipeline where users have to say “hello?” or repeat a question before getting an answer. How it works — Finds silence gaps between consecutive user turns and counts each gap where the agent eventually responded only after the user spoke again. Each silence gap plus the eventual response counts as one event, marked on the timeline. How to interpret — Lower is better. Higher values indicate the agent is failing to pick up its turn reliably; recurring events across calls suggest the agent’s turn-taking or response mechanism is failing intermittently. Configuration — None for the base metric. A configurable variant exposes the minimum silence gap (seconds) that triggers a reprompt event. Requires Audio · Unit count of reprompt events · Scope AudioAPI State
Validates real-world system outcomes by checking an external API response against an expected value after the conversation. What it measures — Whether your backend ended up in the expected state after the conversation — for example, the appointment was actually booked, the account record was updated, or the order was created. When to use — Verifying end-to-end outcomes the transcript alone can’t prove: booking or order confirmation, CRM record updates, tool-call results in real services, or validating mocked API responses in simulations. How it works — After the simulation, Coval calls the configured endpoint, inspects the response body at the configured response path, and compares the extracted value to the expected value. The expected value can be a full JSON object, a primitive, or a template variable such as{{expected_output.balance}}.
How to interpret — MATCH means the API response matched the expected value — your system reached the intended state; DIFF means it did not, pointing to a failed or incorrect action by the agent. NOT_FOUND means the configured response path was missing from the response — usually a misconfigured path or an unexpected response shape.
Configuration — API endpoint, HTTP method (GET or POST) with optional headers and request body, expected value, an optional response path for extraction, and a request timeout (default 30 s).
Requires Transcript + a reachable API endpoint · Unit MATCH / DIFF / NOT_FOUND
End Reason
Reports how the simulated conversation ended — natural completion, hangup, timeout, or error. What it measures — How the conversation terminated. Values includeCOMPLETED (natural conclusion), MAX_TURNS, MAX_DURATION, USER_HANGUP and AGENT_HANGUP (voice calls), IDLE_TIMEOUT (chat/SMS), ERROR, and UNKNOWN.
When to use — Spotting patterns in call completion — for example, confirming calls finish with COMPLETED rather than hitting turn or duration limits, or catching a spike in ERROR or USER_HANGUP terminations after a change to your agent.
How it works — Reads the end reason recorded for the simulation and reports it directly.
How to interpret — A healthy agent should mostly end with COMPLETED; frequent MAX_TURNS or MAX_DURATION suggests conversations are stalling or looping, and USER_HANGUP can signal user frustration. Use the configurable variant to turn an acceptable set of end reasons into a pass/fail check.
Configuration — None for the base metric. A configurable variant lets you define which end reasons count as acceptable, turning it into a YES/NO pass check (YES if the actual end reason matches the acceptable set, NO otherwise).
Requires Transcript · Unit end reason value (or YES / NO in the configurable variant)
Match Expected Output
Compares a named observation (a value Coval captured during the simulation) against an expected value, without calling an external service. What it measures — Reads a named observation (a value Coval captured during the simulation, such as a pre- or post-simulation API payload) and checks whether its value matches a configured expected value. When to use — Verifying the agent produced the correct structured output or tool-call result without touching real services — for example, checking that a mocked booking API’s post-simulation payload contains the right confirmation status. How it works — Reads the selected observation, extracts a value at the configured observation path, and compares it to the expected value. As with API State, the expected value can be a full JSON object, a primitive, or a template variable. How to interpret — MATCH means the captured value matched the expected value; DIFF means the agent produced a different output than expected. NOT_FOUND means the configured observation path was missing — check the observation name and match path. Configuration — Observation name (which observation to read), expected value, and an optional match path to extract a nested value. Requires Transcript + the named simulation observation · Unit MATCH / DIFF / NOT_FOUNDMetadata Field
Surfaces the value of a named field from the conversation’s custom metadata as a metric, so business data you already track becomes visible alongside Coval’s scores. What it measures — Reports the value of a configured field from the conversation’s custom metadata, supporting string, numeric, and boolean fields. When to use — Tracking custom business KPIs alongside Coval metrics — for example, CSAT scores, call type, or boolean flags like “escalation occurred” or “customer authenticated” that your application sends with each conversation. How it works — Reads the named metadata field and outputs its value directly, with no model inference; boolean fields are emitted as 0.0/1.0 for aggregation. How to interpret — The result is the raw metadata value, making any metadata attribute (e.g., call type, escalation flag, CSAT score) visible and aggregatable as a metric. Values aggregate automatically across conversations, so numeric and boolean fields can be charted and compared like any other metric. Configuration — Metadata field key and an explicitly configured field type (string, float, or boolean). Requires metadata to be sent with the conversation. Requires Transcript + custom metadata sent with the conversation · Unit follows the configured field type (string / float / boolean)Music Detection
Counts audio segments classified as music during the call, with timestamps marking each segment. What it measures — The number of audio segments classified as music, with timeline data marking when each music segment occurs. When to use — Detecting hold or queue music during voice calls, identifying unwanted background music in recordings, or debugging audio quality issues related to music interference. How it works — Analyzes the audio to identify non-speech segments and classify them as music, returning a count and per-segment start/end offsets. How to interpret — Higher values indicate more hold music or background music events — useful for spotting when the user is placed on hold. Timeline subvalues show the exact timestamps of each segment so you can jump straight to them in the recording. Configuration — None (built-in). Requires Audio · Unit count of music segmentsTranscript Regex Match
Detects whether a regular expression pattern is present in (or absent from) the conversation transcript — fast, exact, and requiring no prompt. What it measures — Whether a configured regex pattern appears in the transcript, optionally scoped to a speaker role and to the first or last message. When to use — Exact-phrase checks where an LLM judge would be overkill: greeting detection, required disclosures in the first message (“this call may be recorded”), prohibited language compliance (“guarantee”, “promise”), closing statements, or format validation like phone numbers. How it works — Filters transcript messages by the configured role and position, then matches the pattern against them; absent-match mode inverts the result so the metric passes when the pattern is not found. Pure pattern matching — no LLM involved. How to interpret — YES means the match condition is satisfied, NO otherwise. For compliance rules, use absent-match so YES means the prohibited phrase never appeared; use word boundaries (\b) in patterns to avoid partial-word matches.
Configuration — Regex pattern, speaker-role filter, first/last-message position, absent-match (compliance) mode, and case-insensitive matching.
Requires Transcript · Unit YES / NO