Full PR workflow from scratch

This recipe stands up eval-action on a fresh repository, end to end.

Prerequisite: a plune.yaml

The Action wraps the Plune CLI, so it needs a plune.yaml committed at the path you point it at. If you don’t have one yet, run plune init and commit the result — see Getting started.

The workflow

Add .github/workflows/plune-eval.yml:

name: Plune eval diff

on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  pull-requests: write # so the Action can post/update the sticky diff comment

jobs:
  plune:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # full history so the Action can diff the PR against the base branch

      - uses: plune-ai/eval-action@v1
        with:
          config: plune.yaml
          base-ref: main
          # use-mock: 'true'           # default — zero-cost, deterministic, no API key
          # fail-on-regression: 'true' # uncomment to block merges on a pass→fail regression

Open a PR. The Action runs your evals on the PR head and the base branch, diffs them, and posts a single sticky comment.

Why each setting is there

on: pull_request — the Action compares the PR against its base branch, so it’s built for PR events.
permissions: pull-requests: write — required for the sticky comment. Without it the comment step is skipped (the evals still run).
fetch-depth: 0 — the Action checks out the base ref in a detached worktree to recompute the baseline; it needs full history, not a shallow clone.
config / base-ref — default to plune.yaml and main; set them explicitly for clarity.

Next steps

Use a real provider Swap the mock for anthropic / openai / openrouter with a repository secret.

fail-on-regression as a required check Turn the gate into branch protection so regressions can't merge.