Real provider vs the mock default

By default the Action runs on the mock provider — use-mock defaults to 'true'. That means no API key, zero cost, and deterministic output, so every PR can run the Action for free.

When the mock is enough

The mock is ideal for verifying that your plune.yaml is valid and the workflow is wired up, and for deterministic checks that don’t depend on a live model. It powers the safe default:

- uses: plune-ai/eval-action@v1
  with:
    config: plune.yaml
    # use-mock: 'true'   # default

Evaluate against a real model

To score against a real provider, set use-mock: 'false' and pass the key as an environment variable from a repository secret — never hard-code it:

- uses: plune-ai/eval-action@v1
  with:
    config: plune.yaml
    use-mock: 'false'
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The key your plune.yaml needs depends on its provider.type:

provider.type	Secret to pass as env
anthropic	ANTHROPIC_API_KEY
openai	OPENAI_API_KEY
openrouter	OPENROUTER_API_KEY

Caution

Keys are read only from the environment and are never printed to logs or included in the PR comment. Add the secret under Settings → Secrets and variables → Actions before the first real run.

Cost-aware pattern

Run the mock on every PR for fast plumbing/config feedback, and reserve a real-provider run for the evals where model quality actually matters — model-graded and RAG assertions only produce a meaningful score against a real provider.

Next steps

Interpreting the diff Read the comment, the transition table, and the action outputs.