Run Plune in CI

There are two ways to run Plune in CI, depending on what you want.

Option A — eval-action (recommended for GitHub PRs)

If you’re on GitHub and want a PR diff comment plus regression gating, use the companion eval-action. It runs your suite on the PR head and the base branch, diffs them, and posts a single sticky comment — no baseline file to manage.

.github/workflows/plune-eval.yml

name: Plune eval diff
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
  pull-requests: write
jobs:
  plune:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # full history so the action can diff against the base branch
      - uses: plune-ai/eval-action@v1
        with:
          config: plune.yaml
          fail-on-regression: true

With the safe defaults it runs on the mock provider — no key, zero cost — and only comments. See the eval-action recipes for real providers, monorepos, and fork-PR behaviour.

Option B — the raw CLI in any job

For non-GitHub CI, or when you just want a pass/fail check without a PR comment, run the CLI directly and let its exit code gate the job:

Exit code	Meaning
0	every assertion passed
1	at least one assertion failed
2	configuration or execution error

# A minimal job — exit 1 fails the build
steps:
  - uses: actions/checkout@v4
  - run: npx -y @plune-ai/cli run
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Zero-cost runs with the mock provider

To run on every push without spending tokens or needing a key, set PLUNE_MOCK_PROVIDER=1. The mock provider is deterministic — perfect for a fast smoke check that your plune.yaml is valid and wired up:

PLUNE_MOCK_PROVIDER=1 npx -y @plune-ai/cli run

Caution

Model-graded assertions (llm-judge, semantic-similarity, and the RAG metrics) need a real provider to produce a meaningful score — keep those behind a real-key job, and use the mock only for config/plumbing checks.

Gating on regressions without GitHub

Recompute a baseline from your main branch, then diff (see Build a regression suite + diff):

git show main:plune.yaml > /dev/null   # ensure main is fetched
plune run --format json -o current.json
# (run the suite again from a main checkout to produce baseline.json)
plune diff baseline.json current.json --fail-on-regression

Next steps

eval-action recipes Full PR workflow, real provider vs mock, monorepo/matrix, and fork-PR handling.