Build a regression suite + diff

A single plune run tells you the state now. To catch quality regressions — checks that used to pass and no longer do — compare two runs with plune diff.

How the diff works

plune diff takes two JSON run outputs (a baseline and a current) and classifies every eval by its transition. Only a genuine passed → failed transition is a regression:

Transition	Meaning	Counts as a regression?
passed → failed	a check that used to pass now fails	yes
failed → passed	improvement	no
failed → failed	pre-existing failure	no
errored (either side)	execution error, not a quality change	no

Snapshot a baseline, then diff against it

1. Save a baseline from a known-good state (e.g. main). The diff needs JSON, so write it with --format json:

   plune run --format json -o baseline.json

2. Make your change — edit the prompt, swap the model, tune an assertion — then capture the current run:

   plune run --format json -o current.json

3. Diff the two. --fail-on-regression makes the command exit non-zero when a passed → failed transition appears, so a script or CI step can gate on it:

   plune diff baseline.json current.json --fail-on-regression

A Markdown diff is handy for humans; JSON is handy for tooling:

plune diff baseline.json current.json --format markdown -o diff.md
plune diff baseline.json current.json --format json    -o diff.json

The JSON diff exposes a summary with hasRegression, regressions, and improvements — the same fields the eval-action reads to drive its PR comment and gate.

Don't hand-manage baseline.json

Committing a baseline file goes stale fast. In CI, recompute the baseline from the base branch on every run instead — that is exactly what eval-action does for you. See Run in CI.

Exit codes

plune diff --fail-on-regression exits 1 on a regression and 0 otherwise — mirroring plune run (1 on any failed assertion). 2 is reserved for config/execution errors.

Next steps

Run in CI Hand the diff to eval-action, or drive the raw CLI in a job with exit codes.