Skip to content
Plune Plune Docs

Switch providers

Plune is provider-agnostic. The top-level provider block sets the default backend for the whole suite; the API key is read from the environment based on provider.type and is never written to disk.

The three providers

Section titled “The three providers”
provider:
  type: anthropic            # anthropic | openai | openrouter
  model: claude-3-5-sonnet-latest
provider.typeEnvironment variableExample model
anthropicANTHROPIC_API_KEYclaude-3-5-sonnet-latest
openaiOPENAI_API_KEYgpt-4o-mini
openrouterOPENROUTER_API_KEYanthropic/claude-3.5-sonnet

Switching backends is a one-line change — set type and model, and make sure the matching key is in your environment (or .env):

Terminal window
echo 'OPENAI_API_KEY=sk-...' >> .env

Tune the provider

Section titled “Tune the provider”

The provider block accepts optional knobs (all validated):

provider:
  type: anthropic
  model: claude-3-5-sonnet-latest
  temperature: 0        # 0–2
  max_tokens: 1024
  concurrency: 4        # parallel model calls
  timeout: 60000        # ms
  max_retries: 2

Override per eval

Section titled “Override per eval”

Any eval can override part of the provider — handy for routing one hard eval to a stronger model while the rest of the suite stays cheap:

evals:
  - id: tricky-reasoning
    provider:
      model: claude-3-7-sonnet-latest   # just the model; type is inherited
    prompt: "{{question}}"
    dataset: datasets/hard.jsonl
    assertions:
      - type: llm-judge
        criteria: "The reasoning is correct and shows its steps."

Override the judge

Section titled “Override the judge”

Model-graded assertions (llm-judge) can use their own provider — e.g. a cheaper judge than the model under test:

assertions:
  - type: llm-judge
    criteria: "The answer is concise and on-topic."
    provider:
      type: openai
      model: gpt-4o-mini

Run offline with the mock provider

Section titled “Run offline with the mock provider”

Set PLUNE_MOCK_PROVIDER=1 to bypass real API calls entirely — deterministic, free, and key-less. Ideal for validating that a plune.yaml is wired up, and for zero-cost CI smoke checks:

Terminal window
PLUNE_MOCK_PROVIDER=1 plune run

Next steps

Section titled “Next steps”
Custom assertions Combine the 10 assertion types, tune thresholds, and set suite-wide defaults.