> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GitHub Actions

> Launch Coval evaluation runs from GitHub Actions

## Prerequisites

<Steps>
  <Step title="Get Your API Key">
    Navigate to your [Coval dashboard](https://app.coval.dev) and generate an API key from Settings.
  </Step>

  <Step title="Add GitHub Secret">
    1. Go to your repository **Settings > Secrets and variables > Actions**
    2. Click **New repository secret**
    3. Name: `COVAL_API_KEY`
    4. Value: Your Coval API key
  </Step>

  <Step title="Gather Required IDs">
    You'll need the following identifiers:

    * **Agent ID** (22 chars): Found in Agents page → Select agent → Copy ID
    * **Persona ID** (22 chars): Found in Personas page → Select persona → Copy ID
    * **Test Set ID** (8 chars): Found in Test Sets page → Select test set → Copy ID
    * **Metric IDs** (22 chars each, optional): Found in Metrics page → Click metric → Copy ID
  </Step>
</Steps>

## Quick Start

### Automatic PR Checks

Create `.github/workflows/coval-eval.yml`:

```yaml theme={null}
name: Coval Evaluation

on:
  pull_request:
    branches: [main]

jobs:
  evaluate-agent:
    runs-on: ubuntu-latest
    steps:
      - name: Run Coval Evaluation
        uses: coval-ai/coval-github-action@v1
        env:
          COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
        with:
          agent_id: "gk3jK9mPq2xRt5vW8yZaBc"
          persona_id: "hL4kL0nQr3ySt6vX9zAcDd"
          test_set_id: "aB1cD2eF"
```

### Manual Workflow Dispatch

Create `.github/workflows/manual-eval.yml`:

```yaml theme={null}
name: Manual Evaluation

on:
  workflow_dispatch:
    inputs:
      agent_id:
        description: "Agent ID (22 characters)"
        required: true
        type: string
      persona_id:
        description: "Persona ID (22 characters)"
        required: true
        type: string
      test_set_id:
        description: "Test Set ID (8 characters)"
        required: true
        type: string

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Run Evaluation
        uses: coval-ai/coval-github-action@v1
        env:
          COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
        with:
          agent_id: ${{ inputs.agent_id }}
          persona_id: ${{ inputs.persona_id }}
          test_set_id: ${{ inputs.test_set_id }}
```

To trigger:

1. Navigate to **Actions** tab
2. Select **Manual Evaluation**
3. Click **Run workflow**
4. Enter your IDs and click **Run workflow**

## Advanced Configuration

### Custom Metrics and Options

```yaml theme={null}
- name: Advanced Evaluation
  uses: coval-ai/coval-github-action@v1
  env:
    COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
  with:
    agent_id: "gk3jK9mPq2xRt5vW8yZaBc"
    persona_id: "hL4kL0nQr3ySt6vX9zAcDd"
    test_set_id: "aB1cD2eF"
    # Specific metrics to evaluate
    metric_ids: '["iM5lM1oRs4zTu7wY0aBdEe", "jN6mN2pSt5aUv8xZ1bCeFf"]'
    # Run each test case 3 times
    iteration_count: 3
    # Run 2 simulations concurrently
    concurrency: 2
    # Custom metadata for tracking
    metadata: '{"campaign": "q4_2025", "env": "staging"}'
```

### Using Outputs

```yaml theme={null}
- name: Run Evaluation
  id: coval
  uses: coval-ai/coval-github-action@v1
  env:
    COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
  with:
    agent_id: "gk3jK9mPq2xRt5vW8yZaBc"
    persona_id: "hL4kL0nQr3ySt6vX9zAcDd"
    test_set_id: "aB1cD2eF"

- name: Post Results
  run: |
    echo "Run ID: ${{ steps.coval.outputs.run_id }}"
    echo "Status: ${{ steps.coval.outputs.status }}"
    echo "View: ${{ steps.coval.outputs.run_url }}"

- name: Comment on PR
  if: github.event_name == 'pull_request'
  uses: actions/github-script@v7
  with:
    script: |
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: '✓ Evaluation complete: ${{ steps.coval.outputs.run_url }}'
      })
```

## Configuration Reference

### Inputs

| Parameter         | Type        | Required | Default        | Description                            |
| ----------------- | ----------- | -------- | -------------- | -------------------------------------- |
| `agent_id`        | string      | Yes      | -              | Agent to test (22 chars)               |
| `persona_id`      | string      | Yes      | -              | Simulated persona (22 chars)           |
| `test_set_id`     | string      | Yes      | -              | Test set with test cases (8 chars)     |
| `metric_ids`      | JSON array  | No       | Agent defaults | Metric IDs to evaluate (22 chars each) |
| `iteration_count` | integer     | No       | `1`            | Runs per test case (1-10)              |
| `concurrency`     | integer     | No       | `1`            | Concurrent simulations (1-5)           |
| `metadata`        | JSON object | No       | `{}`           | Custom metadata for tracking           |
| `max_wait_time`   | integer     | No       | `600`          | Max wait time in seconds               |
| `check_interval`  | integer     | No       | `30`           | Status check interval in seconds       |

### Outputs

| Output    | Type   | Description                            |
| --------- | ------ | -------------------------------------- |
| `run_id`  | string | Unique run identifier                  |
| `status`  | string | Final status (COMPLETED, FAILED, etc.) |
| `run_url` | string | Dashboard URL to view results          |

### Environment Variables

| Variable        | Required | Description        |
| --------------- | -------- | ------------------ |
| `COVAL_API_KEY` | Yes      | Your Coval API key |

## API Details

The action uses the Coval v1 Runs API:

### Launch Run

**Endpoint:** `POST https://api.coval.dev/v1/runs`

**Request:**

```json theme={null}
{
  "agent_id": "gk3jK9mPq2xRt5vW8yZaBc",
  "persona_id": "hL4kL0nQr3ySt6vX9zAcDd",
  "test_set_id": "aB1cD2eF",
  "metric_ids": ["iM5lM1oRs4zTu7wY0aBdEe"],
  "options": {
    "iteration_count": 3,
    "concurrency": 2
  },
  "metadata": {
    "campaign": "q4_2025"
  }
}
```

**Response:**

```json theme={null}
{
  "run": {
    "run_id": "8EktrIgaVxn9LfxkIynagX",
    "status": "PENDING",
    "create_time": "2025-10-14T12:00:00Z"
  }
}
```

### Monitor Run

**Endpoint:** `GET https://api.coval.dev/v1/runs/{run_id}`

**Response:**

```json theme={null}
{
  "run": {
    "run_id": "8EktrIgaVxn9LfxkIynagX",
    "status": "IN PROGRESS",
    "progress": {
      "total_test_cases": 10,
      "completed_test_cases": 5,
      "failed_test_cases": 0,
      "in_progress_test_cases": 1
    }
  }
}
```

### Run Statuses

| Status        | Description            |
| ------------- | ---------------------- |
| `PENDING`     | Waiting to start       |
| `IN QUEUE`    | Queued for execution   |
| `IN PROGRESS` | Running test cases     |
| `COMPLETED`   | Successfully completed |
| `FAILED`      | Run failed             |

## Examples

### Environment-Based Testing

```yaml theme={null}
name: Multi-Environment Testing

on:
  push:
    branches: [main, staging, dev]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Set Environment
        id: env
        run: |
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "agent=prodAgentId12345678" >> $GITHUB_OUTPUT
            echo "env=production" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref }}" == "refs/heads/staging" ]]; then
            echo "agent=stgAgentId123456789" >> $GITHUB_OUTPUT
            echo "env=staging" >> $GITHUB_OUTPUT
          else
            echo "agent=devAgentId123456789" >> $GITHUB_OUTPUT
            echo "env=development" >> $GITHUB_OUTPUT
          fi

      - name: Evaluate
        uses: coval-ai/coval-github-action@v1
        env:
          COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
        with:
          agent_id: ${{ steps.env.outputs.agent }}
          persona_id: "hL4kL0nQr3ySt6vX9zAcDd"
          test_set_id: "aB1cD2eF"
          metadata: '{"env": "${{ steps.env.outputs.env }}", "commit": "${{ github.sha }}"}'
```

### Parallel Persona Testing

```yaml theme={null}
name: Multi-Persona Testing

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        persona:
          - { id: "persona1234567890abcd", name: "Friendly" }
          - { id: "persona1234567890efgh", name: "Frustrated" }
          - { id: "persona1234567890ijkl", name: "Technical" }
    steps:
      - name: Test ${{ matrix.persona.name }}
        uses: coval-ai/coval-github-action@v1
        env:
          COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
        with:
          agent_id: "gk3jK9mPq2xRt5vW8yZaBc"
          persona_id: ${{ matrix.persona.id }}
          test_set_id: "aB1cD2eF"
          metadata: '{"persona": "${{ matrix.persona.name }}"}'
```

### Scheduled Regression Testing

```yaml theme={null}
name: Nightly Regression

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM daily

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - name: Run Tests
        uses: coval-ai/coval-github-action@v1
        env:
          COVAL_API_KEY: ${{ secrets.COVAL_API_KEY }}
        with:
          agent_id: "gk3jK9mPq2xRt5vW8yZaBc"
          persona_id: "hL4kL0nQr3ySt6vX9zAcDd"
          test_set_id: "regrTest"
          iteration_count: 5
          concurrency: 3
          max_wait_time: 1800
```

## Troubleshooting

### Invalid API Key

```
Status Code: 401
Error Code: UNAUTHENTICATED
Message: Invalid or missing API key
```

**Solution:** Verify `COVAL_API_KEY` is set correctly in GitHub Secrets.

### Invalid Agent ID

```
Status Code: 400
Error Code: INVALID_ARGUMENT
Message: Invalid agent_id: Agent not found
```

**Solution:** Confirm the agent ID is 22 characters and exists in your organization.

### Validation Errors

```
Status Code: 400
Details:
  - iteration_count: Value must be between 1 and 10
```

**Solution:** Ensure all parameters meet the constraints listed in the Configuration Reference.

### Timeout

**Solution:** Increase `max_wait_time` for larger test sets or check the Coval dashboard for run status.

### Invalid JSON

```yaml theme={null}
# Wrong - will fail
metric_ids: ["id1", "id2"]

# Correct - use single quotes around JSON
metric_ids: '["id1", "id2"]'
```

## Resources

* [API Reference](/api-reference/v1/runs/runs/launch-run)
* [Coval Dashboard](https://app.coval.dev)
* [GitHub Action Repository](https://github.com/coval-ai/coval-github-action)
* [Support](mailto:support@coval.dev)