> ## Documentation Index > Fetch the complete documentation index at: https://docs.coval.ai/llms.txt > Use this file to discover all available pages before exploring further. # OpenTelemetry Traces > Send traces from your agent to Coval using the OpenTelemetry SDK. You can send traces from your agent to Coval using the [OpenTelemetry](https://opentelemetry.io/) SDK. This lets you capture detailed span data — such as tool calls, LLM invocations, and other operations — and export it directly to Coval for analysis alongside your simulation or conversation results. Tracing works for both **simulations** (where Coval calls your agent) and **conversations** (where you submit post-hoc call data). The setup differs only in how you identify the call — everything else (instrumentation, span naming, viewing) is the same. This page covers manual instrumentation with the OpenTelemetry SDK. For other setup paths — the one-command [Wizard](/concepts/simulations/traces/wizard), AI-assisted [Tracing Skills](/concepts/simulations/traces/tracing-skills), or [importing](/concepts/simulations/traces/imports) from Langfuse, Arize Phoenix, or LangSmith — start from the [Traces overview](/concepts/simulations/traces/overview). ## Prerequisites * A Coval account with an API key ([manage your keys](https://app.coval.dev/settings)) * A simulation output ID (for simulations) or a conversation ID (for conversations) * Python 3.8+ with the OpenTelemetry SDK installed Install the required packages: ```bash theme={null} pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` ## Configuration Configure the OpenTelemetry tracer provider to export spans to Coval's trace ingestion endpoint: ```python theme={null} from opentelemetry.sdk import trace as trace_sdk from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.resources import SERVICE_NAME, Resource # Configure tracer resource = Resource.create({SERVICE_NAME: "my-agent"}) provider = trace_sdk.TracerProvider(resource=resource) exporter = OTLPSpanExporter( endpoint="https://api.coval.dev/v1/traces", headers={ "X-API-Key": "", "X-Simulation-Id": "", }, timeout=30, ) provider.add_span_processor(SimpleSpanProcessor(exporter)) tracer = provider.get_tracer("my-agent") ``` | Parameter | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------- | | `endpoint` | Coval's OTLP trace ingestion URL: `https://api.coval.dev/v1/traces` | | `X-API-Key` | Your Coval API key | | `X-Simulation-Id` | The **simulation output ID** for the individual call being traced. This is per-simulation-call, not the run ID. | | `timeout` | Export timeout in seconds. Must be set to `30` (see note below) | | `SERVICE_NAME` | A name identifying your agent service | The `timeout` parameter must be set to **30 seconds** to ensure spans are exported reliably. We are working on reducing this requirement in a future update. ## Getting the Simulation Output ID The `X-Simulation-Id` header must be set to the **simulation output ID** for the specific call you're tracing. The simulation output ID is a per-call identifier — different from the run ID. Here's how to obtain it at runtime. ### Inbound voice agents When Coval places an inbound call, it passes the simulation output ID as a SIP header: `X-Coval-Simulation-Id`. Read this header when the call arrives and use it to configure your OTLP exporter. ```python theme={null} # Example: reading the simulation output ID from a SIP header # In your call.initiated webhook handler (Telnyx example): simulation_id = next( h["value"] for h in event["payload"]["sip_headers"] if h["name"] == "X-Coval-Simulation-Id" ) exporter = OTLPSpanExporter( endpoint="https://api.coval.dev/v1/traces", headers={ "X-API-Key": "", "X-Simulation-Id": simulation_id, }, timeout=30, ) ``` See the [Inbound Voice Agent guide](/concepts/agents/connections/inbound-voice) for provider-specific instructions on reading SIP headers (Twilio SIP trunking, Telnyx, etc.). **Twilio Programmable Voice (PSTN)** — Standard Twilio phone numbers route over the public telephone network, which strips SIP headers. Use the `pre_call_webhook_url` agent config instead: Coval will POST the simulation ID to your agent before dialing. See the [Twilio ConversationRelay guide](/guides/simulations/twilio-conversationrelay). ### Outbound voice agents Coval's outbound trigger POST can include the simulation output ID in the request payload. Add `simulation_output_id` to your `trigger_call_payload` configuration in your template, then read it when your webhook receives the trigger and use it to configure the exporter. You can also find simulation output IDs in the Coval dashboard under any run's results, or via the Coval API. ## Tracing for Conversations For [conversations](/concepts/conversations/overview) (post-hoc call evaluation), there is no Coval-initiated call, so there is no simulation output ID available at call time. Instead, you use a **conversation ID** to associate traces with a conversation. The conversation ID is only available *after* the call ends and you submit the transcript to Coval — which means you can't configure the OTLP exporter up front. The solution is to buffer spans in memory during the call, then flush them once you have the ID. Use `InMemorySpanExporter` (included in `opentelemetry-sdk`) to hold spans locally during the call instead of exporting them in real time. ```python theme={null} from opentelemetry.sdk import trace as trace_sdk from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter from opentelemetry.sdk.resources import SERVICE_NAME, Resource resource = Resource.create({SERVICE_NAME: "my-agent"}) in_memory_exporter = InMemorySpanExporter() provider = trace_sdk.TracerProvider(resource=resource) provider.add_span_processor(SimpleSpanProcessor(in_memory_exporter)) tracer = provider.get_tracer("my-agent") # Instrument your agent as normal — spans accumulate in memory with tracer.start_as_current_span("llm") as span: span.set_attribute("metrics.ttfb", 0.42) response = call_llm() ``` Post the transcript (and optionally audio) to `POST /v1/conversations:submit`. The response contains the `conversation_id` you need for trace export. ```python theme={null} import requests response = requests.post( "https://api.coval.dev/v1/conversations:submit", headers={ "x-api-key": "", "Content-Type": "application/json", }, json={ "transcript": [ {"role": "user", "content": "Hello", "start_time": 0.0, "end_time": 1.2}, {"role": "assistant", "content": "Hi! How can I help?", "start_time": 1.5, "end_time": 3.0}, ], }, ) conversation_id = response.json()["conversation"]["conversation_id"] ``` See [`POST /v1/conversations:submit`](/api-reference/v1/conversations/conversations/submit-conversation-for-evaluation) for the full request schema including optional audio, metadata, and metrics fields. If your recording URL isn't available at call end (common with Twilio Programmable Voice in multi-replica deployments), submit the transcript now to get a `conversation_id` for trace correlation, then attach the audio later with `PATCH /v1/conversations/{conversation_id}`. Text-only metrics fire after submit, audio metrics fire after PATCH. Create an OTLP exporter with `X-Conversation-Id` and flush the buffered spans to Coval. ```python theme={null} from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter otlp_exporter = OTLPSpanExporter( endpoint="https://api.coval.dev/v1/traces", headers={ "X-API-Key": "", "X-Conversation-Id": conversation_id, }, timeout=30, ) finished_spans = in_memory_exporter.get_finished_spans() if finished_spans: otlp_exporter.export(list(finished_spans)) ``` | Parameter | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------- | | `X-Conversation-Id` | The `conversation_id` returned by `POST /v1/conversations:submit`. Use this **instead of** `X-Simulation-Id`. | Traces can be sent immediately after submitting a conversation — no delay is needed. ### Full conversation tracing example ```python theme={null} import requests from opentelemetry.sdk import trace as trace_sdk from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter from opentelemetry.sdk.resources import SERVICE_NAME, Resource COVAL_API_KEY = "" # --- Call setup: buffer spans in memory --- resource = Resource.create({SERVICE_NAME: "my-agent"}) in_memory_exporter = InMemorySpanExporter() provider = trace_sdk.TracerProvider(resource=resource) provider.add_span_processor(SimpleSpanProcessor(in_memory_exporter)) tracer = provider.get_tracer("my-agent") # --- During the call: instrument as normal --- with tracer.start_as_current_span("llm") as span: span.set_attribute("metrics.ttfb", 0.42) response = call_llm() with tracer.start_as_current_span("tts") as span: span.set_attribute("metrics.ttfb", 0.18) audio = synthesize_speech(response) # --- After the call ends: submit transcript, then export spans --- submit_response = requests.post( "https://api.coval.dev/v1/conversations:submit", headers={"x-api-key": COVAL_API_KEY, "Content-Type": "application/json"}, json={"transcript": transcript}, ) conversation_id = submit_response.json()["conversation"]["conversation_id"] otlp_exporter = OTLPSpanExporter( endpoint="https://api.coval.dev/v1/traces", headers={"X-API-Key": COVAL_API_KEY, "X-Conversation-Id": conversation_id}, timeout=30, ) finished_spans = in_memory_exporter.get_finished_spans() if finished_spans: otlp_exporter.export(list(finished_spans)) ``` ### Uploading Traces via the Dashboard You can also upload traces directly from the Coval dashboard without using the SDK. In the **Conversations** page, click **Upload to Conversations** and: 1. Add your audio file or transcript as usual 2. In the **Traces (Optional)** section, select your OTLP traces JSON file (must contain a `resourceSpans` array) 3. Click **Upload** — the conversation and traces are submitted together This is useful for testing, debugging, or uploading historical traces that were captured separately. ## Payload Limits & Batching A single export request to `/v1/traces` has a size limit. Large buffered exports — most commonly the end-of-call flush in the conversation flow above — can exceed it and fail with `413 Request Entity Too Large`. Keep each export request under roughly **3–4 MB**. Treat this as a practical target, not a fixed contract: stay comfortably below it rather than tuning to an exact boundary. ### Splitting spans across requests You can split one call's spans across multiple export requests. Every request carrying the same `X-Conversation-Id` (or `X-Simulation-Id`) is merged server-side into a single trace, reconstructed from each span's parent/child relationships. There is no ordering requirement between requests. The simplest way to stay under the limit is `BatchSpanProcessor` with a bounded batch size, which chunks exports for you: ```python theme={null} from opentelemetry.sdk.trace.export import BatchSpanProcessor provider.add_span_processor( BatchSpanProcessor(otlp_exporter, max_export_batch_size=512) ) ``` Lower `max_export_batch_size` if your spans carry large attributes such as full transcripts or prompts. **Retry only the failed batch.** Spans are stored append-only with no de-duplication. If an export fails, resend only that batch — re-sending batches that already succeeded will duplicate spans in the trace view and double-count trace-based metrics. Spans can arrive before `POST /v1/conversations:submit` has finished registering the conversation. They are still attributed correctly and reconcile automatically — no special handling needed on your side. ## Instrumenting Your Agent Once the tracer is configured, wrap operations in spans to capture trace data: ```python theme={null} # Use tracer in agent code with tracer.start_as_current_span("llm_tool_call") as span: span.set_attribute("function.name", "search_database") span.set_attribute("tool_call_id", "call_123") result = call_tool() ``` You can nest spans to capture the full call hierarchy of your agent — for example, a parent span for the overall request and child spans for individual tool calls or LLM invocations. **Shutdown** — Call `provider.shutdown()` when your agent exits. With `SimpleSpanProcessor`, spans are exported synchronously as each span ends (not buffered), so they are already in Coval before shutdown is called. Shutdown is still good practice for clean resource teardown. ```python theme={null} # Call on agent exit for clean resource teardown. provider.shutdown() ``` ## Span Naming Conventions Coval's trace viewer applies semantic colors and labels to well-known span names. Using these names gives a richer experience in the UI and enables built-in trace metrics. | Span Name | Use For | Required Attributes | Optional / Recommended Attributes | Accepted Compatibility Aliases | | --------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `llm` | LLM invocations | — | `metrics.ttfb` (seconds), `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `llm.finish_reason` (`stop`, `tool_calls`, `length`, `content_filter`) | — | | `tts` | Text-to-Speech | — | `metrics.ttfb` (seconds) | — | | `stt` | Speech-to-Text | `transcript` when using [STT Word Error Rate](/concepts/metrics/types/trace#stt-word-error-rate) or the [Audio Upload variant](/concepts/metrics/types/trace#stt-word-error-rate-audio-upload) | `metrics.ttfb` (seconds), `stt.confidence` (ASR confidence 0.0-1.0) | `stt.transcription` is accepted by STT WER for older integrations, but new integrations should emit `transcript` | | `stt.provider.` | Per-provider STT attempt (child of `stt`) | — | `stt.providerName`, `stt.confidence`, `metrics.ttfb` | — | | `vad` | Voice Activity Detection | — | — | — | | `llm_tool_call` | Individual tool/function calls | — | `function.name`, `tool_call_id`, `function.arguments` | Span name `tool_call`; attributes `tool.name`, `tool.call_id`, `tool.arguments` | | `turn` | A single conversation turn | — | — | — | | `conversation` | Full conversation | — | — | — | | `pipeline` | Processing pipeline | — | — | — | | `transport` | Audio/network transport | — | — | — | Any span name works — spans with names not listed above will still appear in the UI with auto-assigned colors. Use `service.name` in your `Resource` to group spans by service. For complete working implementations, see the [voice agent examples](https://github.com/coval-ai/coval-examples/tree/main/voice-agents) on GitHub — Vapi, Pipecat, and LiveKit agents that emit the full span schema. ## Instrumenting STT Spans To use the [STT Word Error Rate](/concepts/metrics/types/trace#stt-word-error-rate) metric (or its [Audio Upload](/concepts/metrics/types/trace#stt-word-error-rate-audio-upload) variant), your agent must emit `stt` spans with a `transcript` attribute containing the transcribed text. This is what allows Coval to compare your agent's STT output against a reference transcript. Coval also accepts the older `stt.transcription` alias for compatibility, but `transcript` is the canonical attribute for new integrations. We also recommend attaching `stt.confidence` when your STT provider exposes a per-utterance confidence score. Here is an example using the [Pipecat](https://github.com/pipecat-ai/pipecat) framework: ```python theme={null} from opentelemetry import trace as otel_trace from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.utils.tracing.service_decorators import traced_stt def _read_path(value, *path): current = value for segment in path: if current is None: return None if isinstance(segment, int): if isinstance(current, (list, tuple)) and 0 <= segment < len(current): current = current[segment] else: return None continue if isinstance(current, dict): current = current.get(segment) else: current = getattr(current, segment, None) return current def extract_stt_confidence(result): confidence = _read_path(result, "channel", "alternatives", 0, "confidence") if confidence is None: return None normalized = float(confidence) if 0.0 <= normalized <= 1.0: return round(normalized, 4) return None class CovalDeepgramSTTService(DeepgramSTTService): """Adds stt.confidence to Pipecat's built-in traced `stt` spans.""" def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._current_stt_confidence = None async def _on_message(self, message): is_final = bool(getattr(message, "is_final", False)) self._current_stt_confidence = extract_stt_confidence(message) if is_final else None try: await super()._on_message(message) finally: if is_final: self._current_stt_confidence = None @traced_stt async def _handle_transcription(self, transcript, is_final, language=None): if is_final and self._current_stt_confidence is not None: span = otel_trace.get_current_span() if span.is_recording(): span.set_attribute("stt.confidence", self._current_stt_confidence) ``` Instantiate the subclass in your pipeline. With `PipelineTask(..., enable_tracing=True)`, Pipecat still emits the standard `stt` span, and the subclass adds `stt.confidence` onto that same span: ```python theme={null} stt = CovalDeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), ]) ``` For non-Pipecat agents, emit equivalent spans wherever your STT returns final transcriptions: ```python theme={null} from opentelemetry import trace as otel_trace tracer = otel_trace.get_tracer("my-stt-instrumentation") with tracer.start_as_current_span("stt") as span: span.set_attribute("transcript", transcription_text) if confidence_score is not None: span.set_attribute("stt.confidence", confidence_score) ``` The span must be named `"stt"` and include the `transcript` attribute with the transcribed text. `stt.confidence` is optional, but when present it should be a 0.0-1.0 score for the final utterance. ## Instrumenting LLM Spans Include `llm.finish_reason` on `llm` spans so you can tell why the model stopped generating. This is especially useful when debugging responses that were silently cut off because `llm.finish_reason=length`. Here is a Pipecat example that enriches the built-in traced `llm` span: ```python theme={null} from opentelemetry import trace as otel_trace from pipecat.services.openai.llm import OpenAILLMService def _read_path(value, *path): current = value for segment in path: if current is None: return None if isinstance(segment, int): if isinstance(current, (list, tuple)) and 0 <= segment < len(current): current = current[segment] else: return None continue if isinstance(current, dict): current = current.get(segment) else: current = getattr(current, segment, None) return current class _FinishReasonTrackingStream: def __init__(self, stream): self._stream = stream self._iter = stream.__aiter__() def __aiter__(self): return self async def __anext__(self): chunk = await self._iter.__anext__() finish_reason = _read_path(chunk, "choices", 0, "finish_reason") if finish_reason is not None: span = otel_trace.get_current_span() if span.is_recording(): span.set_attribute("llm.finish_reason", str(finish_reason)) return chunk async def aclose(self): if hasattr(self._iter, "aclose"): await self._iter.aclose() elif hasattr(self._stream, "aclose"): await self._stream.aclose() async def close(self): if hasattr(self._stream, "close"): await self._stream.close() else: await self.aclose() class CovalOpenAILLMService(OpenAILLMService): """Adds llm.finish_reason to Pipecat's built-in traced `llm` spans.""" async def get_chat_completions(self, params_from_context): stream = await super().get_chat_completions(params_from_context) return _FinishReasonTrackingStream(stream) ``` For non-Pipecat agents, set the attribute directly on your `llm` span after the provider response finishes: ```python theme={null} with tracer.start_as_current_span("llm") as span: response = client.responses.create(...) if response.finish_reason: span.set_attribute("llm.finish_reason", response.finish_reason) ``` Common values include `stop`, `length`, `tool_calls`, and `content_filter`. ## Provider Fallback Spans Many voice agents use a provider fallback chain for STT — for example, Deepgram → Google → Azure. Without per-provider spans, a single `stt` span only shows the final result; there is no visibility into which provider served the call, how long each attempt took, or why a fallback triggered. The convention is to create one `stt.provider.` child span per provider attempt, nested inside the parent `stt` span: ``` stt ← parent span: final result └── stt.provider.deepgram ← attempt 1 (succeeded) ``` Or for a fallback: ``` stt ← parent span: final result ├── stt.provider.deepgram ← attempt 1 (failed, span status = ERROR) └── stt.provider.google ← attempt 2 (succeeded) ``` ### Span attributes | Attribute | Type | Description | | ------------------ | ------ | ------------------------------------------------------- | | `stt.providerName` | string | Provider name, e.g. `"deepgram"`, `"google"`, `"azure"` | | `stt.confidence` | float | ASR confidence score from this provider (0.0–1.0) | | `metrics.ttfb` | float | Time to first byte for this provider attempt (seconds) | ### Code example ```python theme={null} import time from opentelemetry import trace as otel_trace tracer = otel_trace.get_tracer("my-stt-instrumentation") def transcribe_with_fallback(audio): providers = [("deepgram", deepgram_stt), ("google", google_stt)] final_transcript = None with tracer.start_as_current_span("stt") as stt_span: for provider_name, stt_fn in providers: attempt_start = time.time() with tracer.start_as_current_span(f"stt.provider.{provider_name}") as provider_span: provider_span.set_attribute("stt.providerName", provider_name) try: result = stt_fn(audio) ttfb = time.time() - attempt_start provider_span.set_attribute("metrics.ttfb", round(ttfb, 4)) confidence = getattr(result, "confidence", None) if confidence is not None: provider_span.set_attribute("stt.confidence", confidence) final_transcript = result.transcript break # success — stop trying fallbacks except Exception as e: provider_span.set_status(otel_trace.StatusCode.ERROR, str(e)) if final_transcript: stt_span.set_attribute("transcript", final_transcript) return final_transcript ``` ## Full Example ```python theme={null} from opentelemetry.sdk import trace as trace_sdk from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.resources import SERVICE_NAME, Resource # Configure tracer resource = Resource.create({SERVICE_NAME: "my-agent"}) provider = trace_sdk.TracerProvider(resource=resource) exporter = OTLPSpanExporter( endpoint="https://api.coval.dev/v1/traces", headers={ "X-API-Key": "", "X-Simulation-Id": "", }, timeout=30, ) provider.add_span_processor(SimpleSpanProcessor(exporter)) tracer = provider.get_tracer("my-agent") # Use tracer in agent code with tracer.start_as_current_span("llm_tool_call") as span: span.set_attribute("function.name", "search_database") span.set_attribute("tool_call_id", "call_123") result = call_tool() # Call on agent exit for clean resource teardown. provider.shutdown() ``` ## Using Span Attributes in Custom Metrics Any numeric span attribute your agent emits can be measured using a **Custom Trace Metric** (`METRIC_CUSTOM_TRACE`). This lets you track latency, token counts, or any other numeric value from your traces without writing custom evaluation code. To create a custom trace metric, specify: * **Span Name** — the `span_name` of the spans to aggregate (e.g. `llm`, `tts`, or any custom span you create) * **Metric Attribute** — the span attribute key containing the numeric value (e.g. `metrics.ttfb`, `token_count`) * **Aggregation Method** — how to aggregate across turns: `average`, `median`, `p90`, `p95`, `p99`, `max`, `min`, `sum`, `count`, `error_rate`, or `success_rate` See [Create Metric](/api-reference/v1/metrics/metrics/create-metric) for the full API reference.