Reading reports & exit codes

A plune run reports a pass/fail summary with token cost, and writes its full result to .plune/last-run.json. How you read it depends on whether a human or a machine is looking.

Output formats

plune run and plune report both take --format and an optional -o, --output <file>:

plune run --format console     # default — readable summary in the terminal
plune run --format markdown -o report.md
plune run --format json     -o run.json

The console summary looks like:

3/3 passed · 0 failed · 0 errored · $0.0042

plune report re-renders the most recent run (from .plune/last-run.json) without calling the provider again — useful to turn a local run into a Markdown artifact after the fact:

plune report --format markdown -o report.md

Exit codes drive automation

Every invocation communicates its result through the exit code, so a shell script or CI step can gate on it directly:

Exit code	Meaning
0	every assertion passed
1	at least one assertion failed (or, for diff, a regression with --fail-on-regression)
2	configuration or execution error (bad plune.yaml, unknown prompt variable, provider failure)

if plune run; then
  echo "evals green"
else
  echo "evals failed (exit $?)" && exit 1
fi

Tip

Useful run flags for automation: --only <id|tag> (repeatable) to run a subset, --bail to stop at the first failure, --no-cache to force fresh calls, and --concurrency <n> to parallelize.

Consume the JSON

--format json emits the full run: a summary plus per-row detail (the resolved prompt, the model output, and each assertion’s result). Read the summary for a gate:

plune run --format json -o run.json
node -e 'const r=require("./run.json"); process.exit(r.summary.failed > 0 ? 1 : 0)'

The summary carries total, passed, failed, and errored counts.

Call the engine programmatically

The same engine that powers plune run is exported. Unlike the CLI, the library does not parse argv or auto-load .env — set the provider key in process.env yourself:

import { run } from "@plune-ai/cli";
import type { RunResult } from "@plune-ai/cli";

const result: RunResult = await run({ configPath: "plune.yaml", dryRun: false });
console.log(result.summary); // { total, passed, failed, errored, ... }
if (result.summary.failed > 0) process.exit(1);

Next steps

Run in CI Gate a pipeline on these exit codes, or hand the diff to eval-action.