Changelog

Sofia now runs scheduled investigations over your workspace and reports its findings in a Recent Investigations section on Home, with evidence for each one. Acknowledge, resolve, or dismiss findings, flag the ones Sofia got wrong, and review the full history on the Investigations page.

The Coval connector now exposes tools to list, read, and create reports, and to list run templates and create, update, and inspect recurring evaluation schedules. New schedules stay disabled until you enable them. Test case tools also accept an input type and scripted turns, including keypad presses.

In the app, Sofia can now create and update alerts, configure a persona including its voice settings, build a complete dashboard in one action, update an existing run, and stage a batch of test case changes in one reply. Every change still waits for your confirmation.

Create a new voice by describing it in a prompt, then use it for personas in your simulations.

Save, schedule, and share Sofia analyses as notebooks, so a recurring investigation and its findings stay in one place. In beta.

Every organization now comes with a ready-made sample agent, so you can run a baseline evaluation without building an agent first.

Load testing now runs up to 1,000 concurrent simulated calls, so you can see how an agent holds up under production-scale traffic.

Dashboard metrics can now aggregate by median (P50), alongside the existing aggregations.

Composite evaluation scores now display as percentages in runs tables, result badges, and Human Review labels.

The SQL metric editor now shows results from recent simulations so you can check a query before saving, and surfaces clearer validation errors.

Conversation filters now suggest metadata keys and values as you type.

Click any row in a report to open the underlying simulation.

While editing a dashboard, you can show or hide a chart’s series by clicking its legend.

Simulation and conversation lists load faster and no longer pull full transcripts just to render the list.

Sofia can now help you create, edit, and troubleshoot alerts.

Human Review insights give managers a fuller picture, with reviewer agreement, per-labeler leaderboards and activity over time, suspect-label detection, and outlier views.

Split your organization into separate workspaces so agents, test sets, personas, metrics, and tags can be scoped to a team. In beta.

The public v1 conversations listing no longer times out on large default result sets.

Sofia’s scheduled investigations now surface findings on your Home page, where you can acknowledge, dismiss, or flag them.

OpenAI Realtime agents can now use web search during simulations, available in the app and through the API.

You can now set run metadata directly from the launch page when you start a run, not just through the API.

Template variables now resolve in more places, including run metadata and test case fields, so you can drive more per-scenario values into a simulation.

You can now set the order in which metrics appear to reviewers in Human Review.

Sofia is now available through the Coval connector in Claude, so you can reach your Coval data and runs from the assistant you already work in.

Your home page now opens with a Sofia-generated brief of recent activity and findings across your runs.

New voices add Japanese, Mandarin, Cantonese, and Bengali for simulations, and voice previews now show each voice’s language and accent.

The API and CLI can now create LiveKit and Pipecat agents directly, in SDK 0.6.0.

You can now merge reports together through the API and CLI, matching what the app already offers.

Outlier metric scores are now highlighted in runs and reports, so unusual results stand out at a glance.

Reviewers can claim a conversation when they start, which locks it and its labels so two people do not review the same one.

Install Coval from the ChatGPT plugin listing or Claude connector directory to inspect evaluation data, manage supported resources, launch supported runs, and consult Sofia from the assistant where you already work.

Search traces across your whole organization from the app, and through the API and CLI, with configurable monitoring reports.

Human Review insights add reviewer agreement, a reviewer leaderboard, suspect-label detection, and outlier views, so you can see how reviewers compare and spot labels worth a second look.

Call playback now shows speech-activity markers, so you can see who was speaking and when while reviewing a conversation.

You can now rerun metrics across many simulations at once through the v1 API, with a new rerun-metrics endpoint, so you do not have to trigger them one at a time.

Human Review projects now support a required manager, membership history, and a membership audit log, so you can see who has access to a project and when it changed.

Reviewers can now mark an assignment review complete directly, making it clearer when a set of labels is finished.

Sofia, the in-app agent, now streams its responses as they generate, with speed controls, so answers start appearing right away.

Test run names now display correctly in the runs list.

Reviewers can now label conversations blind, without seeing existing or AI-generated labels, so human judgments stay independent. Blind mode can be set per metric.

Human Review project insights add a per-labeler assignee tab and a per-labeler-per-day activity plot, so you can see who labeled what and how labeling is progressing.

Metric configurations are now validated as you set them up, so misconfigured metrics are caught early instead of failing at run time.

Reports now remember your simulation-level scope, so a report keeps the exact set of simulations you selected.

Sofia, the in-app agent, can now take Human Review actions and summarize behavioral outcomes from your runs.

Deleted test sets, personas, and agents can now be restored from a Recently Deleted view, so an accidental delete is no longer permanent.

Reports built from a Human Review project can now filter to just the human-labeled simulations, so the report reflects only the conversations your team reviewed.

Default metric rules can now match more than one metadata condition at once, combined with AND, so you can target metrics more precisely.

Creating a new agent now opens a type picker for voice, text, and advanced setups instead of defaulting straight to voice.

Sofia, the in-app agent, can now generate charts and visualizations in its answers when you ask about your results.

You can now add tags to a metric directly from the metric page, without going back to the list.

Simulation results now include the caller’s source and destination phone numbers, in the app and through the API.

Scatter plots in Multi-Run Analysis add an exclude-outliers toggle that dims Tukey IQR outliers and drops them from the trend line, plus a legend toggle and a point-count caption for shared reports.

A test set’s last-edited time now updates correctly after edits made through the API.

Deleted simulations and runs no longer appear in report rows.

Sofia is an in-app agent that helps you set up tests, create metrics, and dig into results through chat, without leaving Coval.

Chat agents now resolve test case and agent template variables in custom headers, request payloads, and initialization payloads, so you can drive per-scenario values into your endpoint.

Metrics that do not apply to a call now skip with a clear reason instead of a generic failure, and LLM judge metrics retry automatically on truncated or off-list responses, so you see fewer spurious failures.

Metrics now keep a version history, so you can see how a metric changed over time and review prior versions.

A library of ready-made LLM judge metric templates is now available when you create a metric, and the metric gallery is reorganized into clearer categories.

Runs now include an AI-generated summary of what happened, available in beta. Mark a summary helpful or not to help it improve.

Human Review now has project-scoped pages, so you can open a project’s overview and assignments on their own pages and share a direct link to them.

Coval now validates Pipecat, LiveKit, and WebSocket agent connections when you set them up, so configuration problems surface before you run.

You can now upload your own background sounds for simulations, so agents can be tested against the exact ambient conditions they will face in production.

Telephony call recordings in MP3 format now upload reliably, including low-sample-rate recordings that were previously rejected.

A new metric that flags whether an agent sounds monotone or expressive across a call.

A new metric that measures perceived loudness across a call, so you can catch audio that is too quiet or too loud.

Create and delete dashboards programmatically through the API and CLI.

Chat agents now support SSE streaming and a configurable response format.

Dynamic metrics can now reference per-conversation metadata directly in the prompt template.

When your organization is running evaluations beyond your concurrency limits, we will store your data but flag the evaluations as an error. This lets you rerun them later, once you are operating within your concurrency limits.

Metric descriptions auto-populate and the name auto-fills when you create a metric.

Categorical audio metrics now flag a clear, descriptive error when no categories are configured, so misconfigurations surface right away.

A built-in metric that checks whether a call follows the intended IVR navigation path, with no custom scoring logic required.

Define and simulate branching IVR call flows visually, directly in Coval.

Choose how per-turn scores roll up across a multi-turn trace, for example worst-case or first-occurrence.

Apply custom tags to metrics, run templates, and test sets to organize and filter your library.

New voices are available in the persona picker, ready to use with no setup.

Custom sign-in now works across all configured authentication methods.

Password-protected shared runs now open reliably across different saved password formats.

Large reports that used to struggle to load now open reliably even at scale, thanks to a change in how we load them.

The metric picker now uses grouped, nested categories, so the right metric is faster to find and apply.

There is no longer a 1000-character limit on metadata for uploaded conversations.

Auto-generated metrics are now reliably tagged LLM Judge, so AI-scored and rule-based metrics are easy to tell apart.

Speech anomaly, volume variance, and audio sentiment metrics now scope to the channel and time range you configure.

Metrics now do a better job of handling a wider variety of metadata inputs.