Custom assertions

Every eval lists assertions, and all of them must pass for that row to pass. Mixing cheap deterministic checks with model-graded ones gives you fast, specific feedback plus a quality backstop.

The 10 types, by family

assertions:
  # — Text —
  - type: exact-match
    value: "OK"
    trim: true
    ignore_case: true
  - type: contains
    value: "refund"
    ignore_case: true
  - type: contains-any
    values: ["refund", "return", "exchange"]
  - type: contains-all
    values: ["order", "shipped"]

  # — Structured —
  - type: json-schema
    schema:
      type: object
      required: [status]
      properties:
        status: { type: string }
    extract: auto          # auto | strict — how the JSON is pulled from the output

  # — Model-graded —
  - type: llm-judge
    criteria: "The answer is polite and never promises a refund."
    pass_threshold: 0.7    # 0–1
  - type: semantic-similarity
    reference: "Refunds take 5-7 business days."
    threshold: 0.8         # 0–1, embedding cosine similarity

  # — RAG —
  - type: faithfulness
    context: "{{vars.context}}"
    threshold: 0.7
  - type: answer-relevance
    question: "{{vars.question}}"
    threshold: 0.7
  - type: context-precision
    context: "{{vars.context}}"
    question: "{{vars.question}}"
    threshold: 0.7

Interpolate dataset values

Assertion parameters support two placeholder forms drawn from the current dataset row:

• {{expected}} — the row’s expected field.
• {{vars.NAME}} — a named variable from the row’s vars.

assertions:
  - type: contains
    value: "{{expected}}"        # compare against the row's expected text
    ignore_case: true
  - type: faithfulness
    context: "{{vars.context}}"  # the context you retrieved for this row

Note

This is different from the prompt, which uses bare {{question}}. In assertions, reference variables as {{vars.question}}. An unknown assertion placeholder is left literal (it won’t error).

Tuning model-graded checks

• llm-judge grades the output against your criteria and passes at or above pass_threshold (default applies if omitted). Write the criteria as a single, testable statement — “is concise and cites the policy” beats “is good”.
• semantic-similarity, faithfulness, answer-relevance, context-precision each take a threshold between 0 and 1. Start around 0.7–0.8 and adjust from real runs.
• llm-judge can run on its own cheaper provider — see Switch providers.

Apply assertions to every eval

Repeating the same guardrail (a safety or language check) across every eval is noise. Put it once under defaults.assertions and it applies to the whole suite:

version: 1
provider:
  type: anthropic
  model: claude-3-5-sonnet-latest
defaults:
  assertions:
    - type: llm-judge
      criteria: "The response is in English and contains no profanity."
evals:
  - id: faq
    prompt: "{{question}}"
    dataset: datasets/faq.jsonl
    assertions:
      - type: contains
        value: "{{expected}}"

Next steps

Reading reports & exit codes Console, JSON, and Markdown output — and how to consume results in automation.