Test a RAG chatbot end-to-end

This guide builds a complete eval for a retrieval-augmented (RAG) support bot: each dataset row carries the user’s question and the context your retriever returned, and Plune checks that the model’s answer is grounded in that context and actually answers the question.

Tip

New to the CLI? Skim Getting started first — this guide assumes @plune-ai/cli is installed and a provider key is set.

1. Scaffold the project

1. Create the starter files:

   plune init

2. Add your provider key to .env (read from the environment, never written to disk):

   echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env

2. Write the dataset

Datasets are JSONL — one JSON object per line, shaped { "vars": { ... }, "expected"?: "..." }. For RAG, each row’s vars carries both the question and the retrieved context. Save this as datasets/support.jsonl:

{"vars":{"question":"How long do refunds take?","context":"Refunds are processed within 5-7 business days to the original payment method."},"expected":"5-7 business days"}
{"vars":{"question":"Can I return a final-sale item?","context":"Final-sale items cannot be returned or exchanged."},"expected":"cannot be returned"}
{"vars":{"question":"Do you ship internationally?","context":"We currently ship within the United States only."},"expected":"United States"}

Where does context come from?

Plune evaluates the model’s output, so you supply the retrieved context yourself — dump your retriever’s results into each row’s vars.context. That keeps the eval reproducible and independent of retriever flakiness.

3. Configure the eval

In plune.yaml, the prompt uses bare {{question}} / {{context}} placeholders filled from each row’s vars. The assertion parameters use the {{vars.x}} and {{expected}} forms:

version: 1
provider:
  type: anthropic
  model: claude-3-5-sonnet-latest
evals:
  - id: support-rag
    description: Answer support questions using only the retrieved context.
    prompt: |
      You are a support assistant. Answer the question using ONLY the context below.
      If the context does not contain the answer, say you don't know.

      Context:
      {{context}}

      Question: {{question}}
    dataset: datasets/support.jsonl
    assertions:
      - type: faithfulness        # output is grounded in the context (RAG)
        context: "{{vars.context}}"
        threshold: 0.8
      - type: answer-relevance    # output actually answers the question (RAG)
        question: "{{vars.question}}"
        threshold: 0.7
      - type: contains            # a key fact from the row is present
        value: "{{expected}}"
        ignore_case: true

Caution

A prompt that references a variable missing from a row is a config error (exit 2) — every {{var}} in the prompt must exist in that row’s vars. Assertion placeholders are more forgiving: an unknown one is left literal.

4. Run and read the result

plune run
# → 3/3 passed · 0 failed · 0 errored · $0.0042

faithfulness and answer-relevance are model-graded (an LLM judge scores each row against the threshold), so they need a real provider — they won’t score on the mock provider. Re-render the last run as Markdown for a PR or a teammate:

plune report --format markdown

Every run also writes its full result to .plune/last-run.json.

Next steps

Build a regression suite + diff Snapshot a baseline and catch pass→fail regressions with plune diff.

Custom assertions Combine the 10 assertion types, tune thresholds, and set suite-wide defaults.