Skip to main content
You can send traces from your agent to Coval using the OpenTelemetry SDK. This lets you capture detailed span data — such as tool calls, LLM invocations, and other operations — and export it directly to Coval for analysis alongside your simulation or conversation results. Tracing works for both simulations (where Coval calls your agent) and conversations (where you submit post-hoc call data). The setup differs only in how you identify the call — everything else (instrumentation, span naming, viewing) is the same.
This page covers manual instrumentation with the OpenTelemetry SDK. For other setup paths — the one-command Wizard, AI-assisted Tracing Skills, or importing from Langfuse, Arize Phoenix, or LangSmith — start from the Traces overview.

Prerequisites

  • A Coval account with an API key (manage your keys)
  • A simulation output ID (for simulations) or a conversation ID (for conversations)
  • Python 3.8+ with the OpenTelemetry SDK installed
Install the required packages:

Configuration

Configure the OpenTelemetry tracer provider to export spans to Coval’s trace ingestion endpoint:
The timeout parameter must be set to 30 seconds to ensure spans are exported reliably. We are working on reducing this requirement in a future update.

Getting the Simulation Output ID

The X-Simulation-Id header must be set to the simulation output ID for the specific call you’re tracing. The simulation output ID is a per-call identifier — different from the run ID. Here’s how to obtain it at runtime.

Inbound voice agents

When Coval places an inbound call, it passes the simulation output ID as a SIP header: X-Coval-Simulation-Id. Read this header when the call arrives and use it to configure your OTLP exporter.
See the Inbound Voice Agent guide for provider-specific instructions on reading SIP headers (Twilio SIP trunking, Telnyx, etc.).
Twilio Programmable Voice (PSTN) — Standard Twilio phone numbers route over the public telephone network, which strips SIP headers. Use the pre_call_webhook_url agent config instead: Coval will POST the simulation ID to your agent before dialing. See the Twilio ConversationRelay guide.

LiveKit agents

For a LiveKit agent, Coval calls your token-generation endpoint before joining the room. Add the simulation output ID to that request:
1

Configure the token request header

In the LiveKit agent configuration, expand Advanced Configuration and set Custom Headers (JSON) to:
Coval resolves {{simulation_id}} to the simulation output ID for each call.
2

Carry the ID into the agent job

Read X-Coval-Simulation-Id in the token endpoint and include its value in the dispatch metadata or other per-room context your LiveKit agent reads when it starts.
3

Configure the exporter

Read the ID from the LiveKit job context and use it as the X-Simulation-Id header on the OTLP exporter:

Outbound voice agents

Coval’s outbound trigger POST can include the simulation output ID in the request payload. Add simulation_output_id to your trigger_call_payload configuration in your template, then read it when your webhook receives the trigger and use it to configure the exporter.

WebSocket agents

For direct and HTTP-first WebSocket voice agents and Chat WebSocket agents, pass the simulation output ID in the WebSocket upgrade headers.
1

Add the correlation header

In the agent configuration, set Custom Headers to:
Coval resolves {{simulation_id}} separately for each simulation before opening the connection.
2

Read it and configure the exporter

Read X-Coval-Simulation-Id from the WebSocket upgrade request, then configure the OTLP exporter with it as X-Simulation-Id:
If your server cannot inspect WebSocket upgrade headers, put {"coval_simulation_id": "{{simulation_id}}"} in Initialization JSON and read the ID from the first frame instead.
{{simulation_output_id}} is the canonical placeholder; {{simulation_id}} is accepted as a legacy alias. Voice WebSocket agents can also put it in HTTP-first request fields or TwiML voice-webhook fields. Chat WebSocket agents support it in Custom Headers and Initialization JSON.
TwiML webhook mode generates its own Twilio connected and start events from the returned TwiML instead of sending initialization_json. Put {{simulation_output_id}} in voice_form_fields; the webhook can use it immediately and return it as a <Stream><Parameter> if the WebSocket server also needs it.
You can also find simulation output IDs in the Coval dashboard under any run’s results, or via the Coval API.

Tracing for Conversations

For conversations (post-hoc call evaluation), there is no Coval-initiated call, so there is no simulation output ID available at call time. Instead, you use a conversation ID to associate traces with a conversation. The conversation ID is only available after the call ends and you submit the transcript to Coval — which means you can’t configure the OTLP exporter up front. The solution is to buffer spans in memory during the call, then flush them once you have the ID.
1

Buffer spans during the call

Use InMemorySpanExporter (included in opentelemetry-sdk) to hold spans locally during the call instead of exporting them in real time.
2

Submit the conversation after the call ends

Post the transcript (and optionally audio) to POST /v1/conversations:submit. The response contains the conversation_id you need for trace export.
See POST /v1/conversations:submit for the full request schema including optional audio, metadata, and metrics fields.
If your recording URL isn’t available at call end (common with Twilio Programmable Voice in multi-replica deployments), submit the transcript now to get a conversation_id for trace correlation, then attach the audio later with PATCH /v1/conversations/{conversation_id}. Text-only metrics fire after submit, audio metrics fire after PATCH.
3

Export the buffered spans

Create an OTLP exporter with X-Conversation-Id and flush the buffered spans to Coval.
Traces can be sent immediately after submitting a conversation — no delay is needed.

Full conversation tracing example

Uploading Traces via the Dashboard

You can also upload traces directly from the Coval dashboard without using the SDK. In the Conversations page, click Upload to Conversations and:
  1. Add your audio file or transcript as usual
  2. In the Traces (Optional) section, select your OTLP traces JSON file (must contain a resourceSpans array)
  3. Click Upload — the conversation and traces are submitted together
This is useful for testing, debugging, or uploading historical traces that were captured separately.

Payload Limits & Batching

A single export request to /v1/traces has a size limit. Large buffered exports — most commonly the end-of-call flush in the conversation flow above — can exceed it and fail with 413 Request Entity Too Large. Keep each export request under roughly 3–4 MB. Treat this as a practical target, not a fixed contract: stay comfortably below it rather than tuning to an exact boundary.

Splitting spans across requests

You can split one call’s spans across multiple export requests. Every request carrying the same X-Conversation-Id (or X-Simulation-Id) is merged server-side into a single trace, reconstructed from each span’s parent/child relationships. There is no ordering requirement between requests. The simplest way to stay under the limit is BatchSpanProcessor with a bounded batch size, which chunks exports for you:
Lower max_export_batch_size if your spans carry large attributes such as full transcripts or prompts.
Retry only the failed batch. Spans are stored append-only with no de-duplication. If an export fails, resend only that batch — re-sending batches that already succeeded will duplicate spans in the trace view and double-count trace-based metrics.
Spans can arrive before POST /v1/conversations:submit has finished registering the conversation. They are still attributed correctly and reconcile automatically — no special handling needed on your side.

Instrumenting Your Agent

Once the tracer is configured, wrap operations in spans to capture trace data:
You can nest spans to capture the full call hierarchy of your agent — for example, a parent span for the overall request and child spans for individual tool calls or LLM invocations.
Shutdown — Call provider.shutdown() when your agent exits. With SimpleSpanProcessor, spans are exported synchronously as each span ends (not buffered), so they are already in Coval before shutdown is called. Shutdown is still good practice for clean resource teardown.

Span Naming Conventions

Coval’s trace viewer applies semantic colors and labels to well-known span names. Using these names gives a richer experience in the UI and enables built-in trace metrics. Any span name works — spans with names not listed above will still appear in the UI with auto-assigned colors. Use service.name in your Resource to group spans by service.
For complete working implementations, see the voice agent examples on GitHub — Vapi, Pipecat, and LiveKit agents that emit the full span schema.

Instrumenting STT Spans

To use the STT Word Error Rate metric (or its Audio Upload variant), your agent must emit stt spans with a transcript attribute containing the transcribed text. This is what allows Coval to compare your agent’s STT output against a reference transcript. Coval also accepts the older stt.transcription alias for compatibility, but transcript is the canonical attribute for new integrations. We also recommend attaching stt.confidence when your STT provider exposes a per-utterance confidence score. Here is an example using the Pipecat framework:
Instantiate the subclass in your pipeline. With PipelineTask(..., enable_tracing=True), Pipecat still emits the standard stt span, and the subclass adds stt.confidence onto that same span:
For non-Pipecat agents, emit equivalent spans wherever your STT returns final transcriptions:
The span must be named "stt" and include the transcript attribute with the transcribed text. stt.confidence is optional, but when present it should be a 0.0-1.0 score for the final utterance.

Instrumenting LLM Spans

Include llm.finish_reason on llm spans so you can tell why the model stopped generating. This is especially useful when debugging responses that were silently cut off because llm.finish_reason=length. Here is a Pipecat example that enriches the built-in traced llm span:
For non-Pipecat agents, set the attribute directly on your llm span after the provider response finishes:
Common values include stop, length, tool_calls, and content_filter.

Provider Fallback Spans

Many voice agents use a provider fallback chain for STT — for example, Deepgram → Google → Azure. Without per-provider spans, a single stt span only shows the final result; there is no visibility into which provider served the call, how long each attempt took, or why a fallback triggered. The convention is to create one stt.provider.<name> child span per provider attempt, nested inside the parent stt span:
Or for a fallback:

Span attributes

Code example

Full Example

Using Span Attributes in Custom Metrics

Any numeric span attribute your agent emits can be measured using a Custom Trace Metric (METRIC_CUSTOM_TRACE). This lets you track latency, token counts, or any other numeric value from your traces without writing custom evaluation code. To create a custom trace metric, specify:
  • Span Name — the span_name of the spans to aggregate (e.g. llm, tts, or any custom span you create)
  • Metric Attribute — the span attribute key containing the numeric value (e.g. metrics.ttfb, token_count)
  • Aggregation Method — how to aggregate across turns: average, median, p90, p95, p99, max, min, sum, count, error_rate, or success_rate
See Create Metric for the full API reference.