Agent Skills - Coval Documentation

Agent Skills are modular knowledge packages that teach your AI coding agent how to evaluate effectively. They follow the open Agent Skills standard and work with Claude Code, Cursor, Windsurf, Codex, and 40+ other agents.

Install

npx skills add coval-ai/coval-external-skills

This installs all Coval skills into your agent’s skills directory. Skills are loaded on demand — only the name and description are in memory until activated. To install only the recommended starter collection, pass the skill names to the installer’s --skill selector:

npx skills add coval-ai/coval-external-skills --skill coval-eval-start onboard coval-discover-failures build-test-suite configure-metrics quick-eval coval-calibrate-metric coval-compare-runs coval-eval-audit

Choose the target agent and the project or global installation scope in the installer. Skill names are instructions to your coding agent, not terminal commands.

Skills vs Connector vs CLI

	Skills	Coval Connector	CLI
What it provides	Knowledge (how to evaluate well)	Tools (execute operations)	Operations (run from terminal)
Install	`npx skills add coval-ai/coval-external-skills`	ChatGPT, Claude, Codex, or another MCP client	`brew install coval-ai/tap/coval`
Use when	Agent needs to design evaluations	Agent needs to run evaluations natively	Scripting, CI/CD, any terminal
Works with	Any agent supporting skills	MCP-compatible clients	Any shell environment

We recommend Skills + CLI for the most complete experience. Skills teach your agent what to create, and the CLI executes it with structured JSON output.

Available Skills

Recommended starter collection

Start with a product question: what must your agent get right, and what evidence would show it? The starter collection guides a small first evaluation, failure discovery, metric calibration, and regression checks. Install it, then ask your coding agent:

Use coval-eval-start to help me evaluate my agent with Coval. Reuse what I already have, propose a small execution budget, and distinguish working setup from evidence that my agent performs well.

Skill	When it helps	What you get
coval-eval-start	You want help choosing the next step	A route based on existing evidence and your product decision
onboard	No usable evaluation yet	A small first run with inspected results
coval-discover-failures	You have conversations or recordings	Failure hypotheses grounded in transcript, audio, and trace evidence
build-test-suite	You know the requirement or coverage gap	Realistic cases with separate expectations and provenance
configure-metrics	You know what needs measuring	A suitable metric and a tested, provisional criterion
quick-eval	You want to run selected cases	Explicit execution bounds and a complete result audit
coval-calibrate-metric	You need to trust a judge’s scores	Human-label error rates, uncertainty, and held-out validation
coval-compare-runs	You changed the agent	Matched before/after evidence and visible regressions
coval-eval-audit	You already have an eval setup	Prioritized findings about what its results can support

A first voice run starts small: one case, one persona, one iteration, concurrency one. The skills present the concrete plan before spending and count reruns and base + mutation variants against an agreed session budget. These are workflow controls, not server-side spending limits. Human calibration requires real human labels; without them, the output is a review-ready plan, not fabricated validation.

Full catalog

Every skill in the repository, grouped by category. You can install any skill individually; router handoffs (for example from coval-eval-start) require the destination skill to be installed too.

Evaluation

Skill	Description
coval-eval-start	Choose the next useful step for evaluating a voice or chat agent with Coval
coval-discover-failures	Discover failure modes in Coval runs, uploaded conversations, or local conversation exports using scoped sampling and transcript, audio, and trace evidence
coval-calibrate-metric	Validate one Coval judge against independent human labels, with grouped development/test separation, class-specific errors, and uncertainty
coval-compare-runs	Compare before/after Coval runs using matched cases, stable metric versions, and inspected conversation evidence
coval-eval-audit	Audit an existing Coval evaluation setup for coverage, misleading scores, missing evidence, judge validation, and execution risk

Onboarding

Skill	Description
onboard	Connect your agent to Coval and complete a small first evaluation with inspected voice or chat results

Runs

Skill	Description
launch-run	Launch a specific Coval evaluation with explicit cases and execution bounds
watch-run	Monitor a Coval run’s progress with live updates
quick-eval	Execute a bounded Coval evaluation on explicit test cases, then audit conversation, recording, and metric evidence
run-accent-testing	Create one persona per accent, launch one run per accent against the same voice agent, test set, and metrics, and build the saved multi-run report grouped by persona (cookbook)
run-adversarial-testing	Build an adversarial test set with legitimate controls, an “Adversarial User” persona, and a Composite Evaluation metric, launch a multi-iteration run, and build a saved report grouped by test case (cookbook)
run-audio-quality-testing	Launch one run per audio-robustness scenario against the same voice agent, test set, and metrics, and produce the multi-run report grouped by persona (cookbook)

Simulations

Skill	Description
get-results	Retrieve and analyze simulation results from a Coval run
download-audio	Download audio recordings from Coval voice simulations

Reports

Skill	Description
analyze-accent-report	Analyze a Coval accent testing report from runs across different speaker accents and turn it into evidence-backed next steps
analyze-adversarial-report	Analyze a Coval adversarial / red-team testing report and turn it into an agent-hardening plan
analyze-audio-quality-report	Analyze a Coval audio-quality testing report from runs across voice, speaking-style, volume, interruption, and background-noise scenarios

Agents

Skill	Description
setup-agent	Set up and connect a new Coval agent for evaluation: agent type selection, endpoint configuration, system prompt, and default resource attachment

Personas

Skill	Description
design-persona	Design and create a simulation persona: use case selection, voice and language configuration, behavior prompt crafting, and interruption calibration
personas-from-artifacts	Derive a set of simulation personas from product artifacts (backend payloads, UI screenshots, journey docs, sample user messages) instead of designing one persona by hand

Test Cases

Skill	Description
build-test-suite	Build a compact Coval test set from product requirements, observed conversation failures, or a coverage gap
distill-test-set	Turn a large dataset (an oversized test set, an export of past conversations, or a CSV/JSON of cases) into a small, high-signal Coval test set
huggingface-import	Import datasets from HuggingFace and convert them to Coval test sets

Metrics

Skill	Description
configure-metrics	Select and configure a Coval metric for a concrete product criterion, choosing transcript, audio, trace, or deterministic evidence and testing the result

Dashboards

Skill	Description
build-dashboard	Build or improve a Coval dashboard with metric visualizations backed by real data

Traces

Skill	Description
setup-tracing	Configure an AI agent to send OpenTelemetry traces to Coval
optimize-trace-observability	Improve Coval trace quality after basic ingestion works
configure-trace-metrics	Recommend, create, preview, and attach Coval trace-based metrics from OpenTelemetry spans
debug-traces	Troubleshoot Coval OpenTelemetry trace ingestion, missing trace UI, sparse traces, bad simulation or conversation correlation, auth/org errors, oversized payloads, and duplicate spans

See Tracing Skills for copy-paste prompts and validation guidance.

Human Review

Skill	Description
review-llm-annotations-and-improve-prompt	Analyze development-set disagreements between human annotations and one Coval text LLM judge, then propose a focused prompt revision

Migrations

Skill	Description
migrate-bluejay	Migrate configuration from the Bluejay voice AI testing platform to Coval

Resources

Skill	Description
coval-resources	Comprehensive overview of all Coval platform resources, their hierarchy, relationships, API endpoints, and ID formats

Sofia

Skill	Description
consult-sofia	Delegate a read-only Coval evaluation, simulation, product-how-to, or troubleshooting subtask to Sofia through the Coval Connector’s `consult_sofia` tool

How Skills Work

Skills use progressive disclosure to stay lightweight:

At startup (~100 tokens per skill): Only the name and description are loaded
When activated (under 5000 tokens): The full skill instructions load when your agent detects a relevant task
On demand: Reference files (templates, examples) load only when needed

This means having all Coval skills installed adds minimal overhead to your agent’s context.

Skill Structure

Each skill follows the Agent Skills spec:

skill-name/
├── SKILL.md          # Instructions (required)
├── references/       # Templates, detailed docs (optional)
├── scripts/          # Executable code (optional)
└── assets/           # Static resources (optional)

Source Code

All skills are open source: github.com/coval-ai/coval-external-skills

​Install

​Skills vs Connector vs CLI

​Available Skills

​Recommended starter collection

​Full catalog

​Evaluation

​Onboarding

​Runs

​Simulations

​Reports

​Agents

​Personas

​Test Cases

​Metrics

​Dashboards

​Traces

​Human Review

​Migrations

​Resources

​Sofia

​How Skills Work

​Skill Structure

​Source Code

Install

Skills vs Connector vs CLI

Available Skills

Recommended starter collection

Full catalog

Evaluation

Onboarding

Runs

Simulations

Reports

Agents

Personas

Test Cases

Metrics

Dashboards

Traces

Human Review

Migrations

Resources

Sofia

How Skills Work

Skill Structure

Source Code