Skip to content

Check how a model uses an MCP server's tools

An eval in plune.yaml can give the model the tools of an MCP server and then check what the model did with them: which tool it picked, with which arguments, in which order, how many calls it took and what it cost. This is the agentic check. It is for the person who builds an MCP server, or a prompt that sends a model to one, and wants a failing test when a change makes the model pick the wrong tool.

It is not the page about results of end-to-end tests that an AI agent runs, which is about importing a JUnit report. To test the server itself, without a model, see Test an MCP server.

In the commands below plune stands for the CLI. With npx, write npx -y @plune-ai/cli@X.Y.Z in its place and replace X.Y.Z with the version you have tested.

You need Node.js and @plune-ai/cli. You do not need a Plune account, a Plune token or an API key to follow this section: the model is replaced by a short script, so you can see the loop and the checks before you pay for a model.

  1. Make a folder with the notes server (it is the sample server of Lint an MCP server with its three defects fixed) and install its one dependency:

    server.mjs
    // mcp-sample-server, fixed: the same three tools with the three defects removed.
    import { Server } from "@modelcontextprotocol/sdk/server/index.js";
    import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
    import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
    const notes = [{ id: 1, title: "Release checklist", status: "open" }];
    const readOnly = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
    const tools = [
    {
    name: "list_notes",
    description: "Return the notes with their id, title and status. Do not use it to read one note; use get_note instead.",
    inputSchema: {
    type: "object",
    properties: { limit: { type: "integer", minimum: 1, maximum: 100, description: "How many notes to return at most (default 20)." } },
    required: [],
    },
    annotations: readOnly,
    },
    {
    name: "get_note",
    description: "Read a single note when you know its id. Do not use it to browse; use list_notes instead.",
    inputSchema: {
    type: "object",
    properties: { id: { type: "integer", description: "The id of the note, as list_notes shows it." } },
    required: ["id"],
    },
    annotations: readOnly,
    },
    {
    name: "set_status",
    description: "Change the status of one note. Do not use it to edit the title; this server has no tool for that.",
    inputSchema: {
    type: "object",
    properties: {
    id: { type: "integer", description: "The id of the note to change." },
    status: { type: "string", enum: ["open", "done", "archived"], description: "The new status." },
    },
    required: ["id", "status"],
    },
    annotations: { readOnlyHint: false, destructiveHint: true, idempotentHint: true, openWorldHint: false },
    },
    ];
    const server = new Server({ name: "mcp-sample-server", version: "1.0.0" }, { capabilities: { tools: {} } });
    server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools }));
    server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
    const text = (value) => ({ content: [{ type: "text", text: JSON.stringify(value) }] });
    if (name === "list_notes") return text(notes.slice(0, args.limit ?? 20));
    const note = notes.find((n) => n.id === args.id);
    if (!note) return { isError: true, content: [{ type: "text", text: `No note with id ${args.id}.` }] };
    if (name === "set_status") note.status = args.status;
    return text(note);
    });
    await server.connect(new StdioServerTransport());
    Terminal window
    npm install @modelcontextprotocol/sdk
  2. Write plune.yaml. The top-level mcp block names the server; each eval picks one and lists its checks. The first eval is a positive check (the model must read the note with get_note), the second a negative one (asked only to show a note, the model must not change it):

    version: 1
    provider:
    type: anthropic
    model: claude-haiku-4-5
    mcp:
    servers:
    notes:
    command: node
    args: ["server.mjs"]
    limits:
    max_steps: 6
    evals:
    - id: read-a-note
    description: "The model reads one note by its id"
    prompt: "Read note {{id}} and tell me its title."
    dataset:
    examples:
    - vars: { id: "1" }
    mcp:
    server: notes
    assertions:
    - type: first-tool
    tool: get_note
    - type: tool-args
    tool: get_note
    args: { id: 1 }
    - type: no-tool-errors
    - type: contains
    value: "Release checklist"
    - id: never-change-a-note
    description: "Asked to read, the model must not change anything"
    prompt: "Show me note {{id}}."
    dataset:
    examples:
    - vars: { id: "1" }
    mcp:
    server: notes
    assertions:
    - type: tool-not-called
    tool: set_status
    - type: no-destructive-tool
  3. See what it would cost, with no model call and no server started:

    $ plune run --dry-run
    Plune dry run
    2 row(s) priced, none measured · estimated $0.0615
    PRICED read-a-note
    1 row(s) priced
    PRICED never-change-a-note
    1 row(s) priced
    dry run: an eval with mcp is priced as max_steps model calls of its prompt; tool schemas and tool results are not counted
    $ echo $?
    0
  4. Run it. The model here is a script, script.json: the first step calls get_note, the second answers.

    [
    { "tool_calls": [{ "name": "get_note", "args": { "id": 1 } }] },
    { "text": "Note 1 is titled Release checklist." }
    ]
    $ PLUNE_MOCK_PROVIDER=1 PLUNE_MOCK_SCRIPT=script.json plune run
    Plune run
    2 passed · 0 failed · 0 errored · 2 total · $0.0000 · …ms
    stability: not measured (runs: 1)
    cache: 0 from cache · 2 from model · saved 0 tokens, $0.0000
    PASS read-a-note
    1 row(s) passed
    PASS never-change-a-note
    1 row(s) passed
    $ echo $?
    0
  5. Break it on purpose. A model that changes the note instead of reading it:

    [
    { "tool_calls": [{ "name": "set_status", "args": { "id": 1, "status": "done" } }] },
    { "text": "I marked note 1 as done." }
    ]
    $ PLUNE_MOCK_PROVIDER=1 PLUNE_MOCK_SCRIPT=script.json plune run --only never-change-a-note
    Plune run
    0 passed · 1 failed · 0 errored · 1 total · $0.0000 · …ms
    stability: not measured (runs: 1)
    cache: 0 from cache · 1 from model · saved 0 tokens, $0.0000
    FAIL never-change-a-note
    vars: {"id":"1"}
    output: I marked note 1 as done.
    x tool-not-called: expected "set_status" never called; got 1 call: step 1 set_status {"id":"<id>","status":"done"}; check what in the prompt or the tool list led the model to it
    x no-destructive-tool: expected no call to a tool that may be destructive; got step 1 set_status {"id":"<id>","status":"done"}: "set_status" declares destructiveHint: true; the hints are the server's own claim, so check them on the server
    $ echo $?
    1

Read the failure from the bottom up. tool-not-called says which tool was called and in which step, and what to check; no-destructive-tool says why that tool counts as destructive (the server’s own destructiveHint). The row is failed because the model’s behaviour broke a rule you wrote. Exit 1.

To run with a real model, drop the two PLUNE_MOCK_* variables and set the provider’s key as you do for any eval. The provider must be anthropic, openai or openrouter. By default the rows of an agentic eval run one at a time, since they may share the server’s state; --concurrency or provider.concurrency overrides that. That run costs money, and its output is not shown on this page: it is the same shape as the mock’s, with the model’s own tool choices and a real cost.

For each dataset row plune run starts the server (or connects to the url you name), reads its tools/list to the last page, and offers the model the tools in allowed_tools, or all of them when you leave the field out (an empty list is a config error, not “all”). It runs each call the model asks for, one after another, and gives the model every answer. After a step with a call that worked, it reads tools/list again, so a server that switches tools on or off is followed. It repeats until the model answers without a call or a limit stops it.

What a stdio server process gets is only the variables you name in env, plus a short list such as PATH and HOME. The provider’s API key never reaches it. A value written ${NAME} is read from the environment, and an unset ${NAME} is a config error that names the variable and never its value. A secret written as plain text under a key like token or authorization earns a warning that names the key.

A call to a tool the model was not offered, or with arguments that are not a JSON object, never reaches the server: the model gets an error back, and the call is recorded as not executed.

Every limit has a default, so a loop always ends. A limit is a whole number above 0; there is no “no limit”. An eval’s limits win over the top-level mcp.limits, which win over these:

LimitDefaultWhat it bounds
max_steps10Model turns for one dataset row.
max_tool_calls20Tool calls for one dataset row.
max_total_tokens100,000Input plus output tokens for one dataset row.
timeout_ms120,000Time for one dataset row, in milliseconds.
tool_timeout_ms30,000Time for one tool call, in milliseconds.

A limit is a brake, not a verdict. A row it stopped is judged on the last text the model gave, and the transcript says which limit stopped it. max_total_tokens is checked after each step, so a row can pass it by one step at most.

Each row of an agentic eval has an agent field in last-run.json and in --format json: every tool the server listed, every model step with its text, its tokens and its calls, and for each call its arguments, whether it ran, whether it was an error, how long it took and an excerpt of the answer. The first row of the passing run above, with the annotations of each tool cut to …:

{
"server": "notes",
"tools": [
{
"name": "list_notes",
"offered": true,
"annotations": "…"
},
{
"name": "get_note",
"offered": true,
"annotations": "…"
},
{
"name": "set_status",
"offered": true,
"annotations": "…"
}
],
"steps": [
{
"index": 1,
"text": "",
"finish_reason": "tool_calls",
"usage": {
"input_tokens": 10,
"output_tokens": 5,
"cost_usd": 0
},
"calls": [
{
"id": "call_0_0",
"name": "get_note",
"args": {
"id": "<id>"
},
"executed": true,
"is_error": false,
"duration_ms": "…",
"result": {
"text": "{\"id\":\"<id>\",\"title\":\"Release checklist\",\"status\":\"open\"}",
"bytes": 52,
"truncated": false
}
}
]
},
{
"index": 2,
"text": "Note 1 is titled Release checklist.",
"finish_reason": "stop",
"usage": {
"input_tokens": 10,
"output_tokens": 5,
"cost_usd": 0
},
"calls": []
}
],
"stop": {
"reason": "complete"
},
"collection": "complete"
}

An answer is kept as an excerpt of at most 4 KiB, with its size before the cut. From 0.39.7 the stored transcript keeps no identifier values: in the arguments and in the answer, a value under a key such as id, key or slug, and a UUID anywhere, is stored as "<id>", after the checks have read the real values. From 0.39.8, truncated and the row’s collection say what the first excerpt said about the real answer: an answer the model saw whole stays whole, even when its marks make it a few bytes longer than the excerpt size, and only an answer that was already cut is cut again; bytes is still the size of the real answer. Every value of the server’s env and headers is masked as *** in the transcript, in the output and in errors.

The checks that read the transcript. Each is one entry in an eval’s assertions:

CheckClassParametersPasses when
tool-calleddeterministictool, min_count (default 1)the tool was called at least min_count times, with any arguments
tool-not-calleddeterministictoolthe tool was never called (negative: see Not measured)
tool-argsdeterministictool, args (an object, or a list of objects as alternatives), match (partial default, exact, ignore), min_countat least min_count calls of the tool have arguments that match args
tool-orderdeterministictools, strict (default false)the calls come in this order; other calls may come between them unless strict, where the calls are exactly tools
only-toolsdeterministictools (may be [])no tool off the list was called; [] means no call at all (negative)
first-tooldeterministictoolthe first call is the tool; no call at all fails and says so
no-tool-errorsdeterministictool (optional)no call had an error: isError, a JSON-RPC error, a timeout, a tool that was not offered, arguments that are not an object (negative)
tool-call-countmeasuretool (optional), min, max (at least one of them)the number of calls is within the bounds, inclusive
no-repeated-callheuristictool (optional)no call repeats the one right before it (same name, same arguments). Advice: it never decides the row
token-budgetmeasuremaxthe row used at most max tokens, input plus output
tool-latencymeasuremax_ms, tool (optional)every call that ran took at most max_ms
tool-result-sizemeasuremax_bytes, tool (optional)every answer is at most max_bytes, measured before the transcript cut it to its excerpt
tool-result-schemadeterministicschema, tool (optional)every answer that is not an error parses as JSON that fits the JSON Schema
no-destructive-tooldeterministicnoneno called tool may be destructive by its annotations (a tool counts as safe to call only with readOnlyHint: true or destructiveHint: false)

Why a heuristic does not stop the run. A deterministic check is a rule that is true or false of the transcript; a measure compares a number with a limit you wrote; a heuristic is advice. no-repeated-call guesses that a call that repeats the one before it is a loop, and a model may repeat a call for a good reason. It is filed as advice on the row and no configuration makes it decide the verdict.

A tool name in a check must be a tool the model was offered at some point of the row. A name that is not one (a misspelling) would otherwise let tool-not-called pass forever, so it is an error that lists the tools the model was offered.

no-destructive-tool reads the server’s own annotations: a tool counts as safe to call only with readOnlyHint: true or destructiveHint: false, and a tool with no annotations may be destructive. A pass is what the server declares, not proof that a tool does no harm.

A row that ends errored is not failed. failed means the model’s behaviour broke a rule you wrote. errored means the run could not measure it: the server could not be reached, tools/list failed, the server went away during the loop, the provider failed after its retries, or a reply was cut off in the middle of a call. A fault of the setup is not a regression of your prompt, so it is a different state and a different exit code (2).

A negative check has a second case. tool-not-called, only-tools, no-tool-errors and no-destructive-tool look for something that must not have happened. If a limit stopped the loop, the thing may be in a call that was never made or never seen, so a check that found nothing proves nothing. It is not a pass: the row is errored, with the reason.

$ PLUNE_MOCK_PROVIDER=1 PLUNE_MOCK_SCRIPT=script.json plune run --config plune.limit.yaml
Plune run
0 passed · 0 failed · 1 errored · 1 total · $0.0000 · …ms
stability: not measured (runs: 1)
cache: 0 from cache · 1 from model · saved 0 tokens, $0.0000
FAIL stopped-by-a-limit
vars: {"id":"1"}
error: tool-not-called: not measured because the loop was stopped by max_steps, so a call to "set_status" may be in what was not seen; nothing was found in what was. The assertions after it in this row were not run.
$ echo $?
2

Here max_steps: 1 stopped the loop while the model still wanted to call a tool. A violation that was found still fails the row, whatever stopped the loop: it is there to see. Raise the limit, or shorten the task, until the loop ends on its own.

A mock without a script, or a run with --cache-strict, is refused before anything is called, because an agentic row would measure nothing:

$ PLUNE_MOCK_PROVIDER=1 plune run
Eval "read-a-note" uses MCP server "notes", and this provider cannot call tools, so nothing would be measured (under PLUNE_MOCK_PROVIDER=1, PLUNE_MOCK_SCRIPT names the scripted model steps)
$ echo $?
2
$ PLUNE_MOCK_PROVIDER=1 PLUNE_MOCK_SCRIPT=script.json plune run --cache-strict
Eval "read-a-note" uses MCP server "notes": its tool loop is never cached, so --cache-strict cannot answer it
$ echo $?
2

An agentic row is never cached: a stored answer would not show that the server still answers the way it did.

A model is not deterministic, so one run cannot tell a flaky case from a broken one. runs makes each row run that many times, one run after another, and pass_threshold says what share of them must pass:

runs: 1 # the default for every eval
evals:
- id: read-a-note
runs: 10 # each row runs 10 times
pass_threshold: 0.8 # at least 8 of the 10 must pass

runs is a whole number from 1 to 100. --runs <n> sets it for one run. pass_threshold is above 0 and at most 1 (0 is refused, since it would pass with no passing run). A row needs ceil(pass_threshold × runs) passing runs. Of a row’s runs, let P passed, F failed and E errored, and needed be that number:

ConditionThe row is
P ≥ neededpassed
otherwise, P + E ≥ needederrored
otherwisefailed

Why errored is not failed here. With runs: 10 and pass_threshold: 0.8, 6 passes, 1 fail and 3 errors could have been 9 passes. The verdict would rest on runs that were never measured, so the row is errored. A run that could not be measured is never counted as a pass, and never as a quality fail.

The threshold is coarse on few runs. With runs: 2, a threshold of 0.8 means both runs; with runs: 1, any threshold below 1 is the same as 1. For a threshold below 1, use at least runs: 5. Every run is played to the end, even when the verdict is already settled: the count of passing runs is the measure of how flaky the case is. A run with repeats takes runs times as long and costs runs times as much.

The console and Markdown reports mark a row that passed some of its runs but not all with the share that passed: 3/5 passed (pass_threshold 0.6): passed · flaky 0.6. A row that passed all of its runs, or none, has no mark. A run in which no row repeats prints one line under its summary instead, stability: not measured (runs: 1).

$ plune run --dry-run --config plune.runs.yaml
Config validation failed in plune.runs.yaml
- evals.0.runs: Too big: expected number to be <=100
$ echo $?
2

This pass_threshold is not the pass_threshold of an llm-judge check. That one is the lowest score from the judge that one answer needs, and sits inside the check; the eval’s sits beside runs and counts runs. A config error says which of the two it is about.

plune run --dry-run connects nowhere, starts no server and calls no model. For an eval with an mcp block it prices max_steps model calls of the prompt, and says that tool schemas and tool results are not counted: the estimate is a floor for a loop with many tool results, not a forecast. Repeats multiply it:

$ plune run --dry-run --runs 5
Plune dry run
2 row(s) priced, none measured · estimated $0.3076
PRICED read-a-note
vars: {"id":"1"}
dry run: estimate × 5 runs
1 row(s) priced
PRICED never-change-a-note
vars: {"id":"1"}
dry run: estimate × 5 runs
1 row(s) priced
dry run: an eval with mcp is priced as max_steps model calls of its prompt; tool schemas and tool results are not counted
$ echo $?
0

The number is an estimate from the price table of the CLI. A model with no known price is reported as unpriced, never as free. A row is marked priced: false when any of its steps was unpriced, which is why a cost gate on such a run says “not measured” rather than passing on 0 (see below). The cost gate counts a run replayed from the cache at its full cost, the cost of what it spent plus what its cache hits saved, since a cheap replay is not a cheap run. An agentic row is not cached, so this matters for the other evals in the same file.

plune diff <baseline> <current> compares two plune run --format json results and always prints the comparison. Each gate is off until its flag is given; with none of them the exit is 0.

FlagGateFails when
--fail-on-regressionregressionan eval went from passed to failed
--max-pass-rate-drop-percent <n>pass-ratethe overall pass rate dropped by more than n points (at least 10 attempts on each side, and the same population)
--max-cost-usd <n>costthe current run cost more than n USD
--max-cost-increase-percent <n>cost-increasethe current run cost more than n percent over the baseline
--max-tokens <n>tokensthe current run used more than n tokens, input plus output
--waivers <file>waivers for measured failures, below
ExitMeaning
0Every gate was measured and none failed (a failure lifted by a waiver does not count).
1A measured failure: a regression, or a gate over its limit. It wins over 3.
2An input file is missing or is not a plune run --format json result, a limit is not a number of 0 or more, or the waivers file is malformed.
3Something was not measured: an eval errored in the current run, none of its rows was judged, or a gate’s value is missing.

A regression, with the gate that fails because of it:

$ plune diff baseline.json current.json --fail-on-regression
Plune eval diff: 2 regression(s)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
regression: failed, passed → failed: read-a-note, never-change-a-note: 2 (limit 0)
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
1

A gate over its limit. --max-tokens reads the tokens of the current run:

$ plune diff baseline.json current.json --max-tokens 10
Plune eval diff: 2 regression(s)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
tokens: failed, input + output tokens of the current run: 60 (limit 10)
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
1

The cost gates are measured only when every row of the run has a known price. The mock model has none, so asking for a cost limit gives «not measured» and exit 3, not a green 0:

$ plune diff baseline.json current.json --max-cost-usd 0.05
Plune eval diff: 2 regression(s)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
cost: not measured, current run: eval "read-a-note" has a row with no known price (limit $0.05)
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
3

A run in which nothing was measured is not a pass either. A current run with no evals gives exit 3 under --fail-on-regression:

$ plune diff baseline.json empty.json --fail-on-regression
Plune eval diff: not measured: regression
read-a-note: passed -> absent [removed]
never-change-a-note: passed -> absent [removed]
Gates:
regression: not measured, the current run has no evals
0 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 2 removed · 0 not measured · 0 stable
$ echo $?
3

Same population. The overall pass rate of two runs is compared only when they measured the same thing: the same evals, rows, number of repeats, checks and model. Weakening a check changes the set of checks, so it does not get a clean rate comparison. The text of the prompt is not part of it, because changing the prompt is the very change a pull request is checked for.

A waiver file lifts a measured failure for a limited time. Every entry has a gate, a reason of at most 200 characters, a granted date and an expires date that is after it and at most 30 days after it. eval is required for regression and not allowed for the other gates.

waivers:
- gate: regression
eval: read-a-note
reason: "The prompt was reworded on purpose; the new wording is reviewed in the pull request"
granted: "2026-11-22"
expires: "2026-12-06"
- gate: regression
eval: never-change-a-note
reason: "Same change as read-a-note"
granted: "2026-11-22"
expires: "2026-12-06"
$ plune diff baseline.json current.json --fail-on-regression --waivers waivers.yaml
Plune eval diff: gates passed (1 waived)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
regression: waived, passed → failed, waived: read-a-note, never-change-a-note: 2 (limit 0)
waived: regression of read-a-note until …: The prompt was reworded on purpose; the new wording is reviewed in the pull request
waived: regression of never-change-a-note until …: Same change as read-a-note
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
0

A waiver never lifts a «not measured» (3) and never an input error (2): the gap in what was measured is still a gap. Here a waiver for the cost gate is listed as unused, and the exit is still 3:

$ plune diff baseline.json current.json --max-cost-usd 0.05 --waivers waivers-cost.yaml
Plune eval diff: 2 regression(s)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
cost: not measured, current run: eval "read-a-note" has a row with no known price (limit $0.05)
unused waiver: cost until …: Price of the mock model is unknown here
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
3

An expired waiver no longer applies. It is listed as expired, and the failure counts again:

waivers:
- gate: regression
eval: read-a-note
reason: "The prompt was reworded on purpose; the new wording is reviewed in the pull request"
granted: "2026-10-13"
expires: "2026-11-02"
- gate: regression
eval: never-change-a-note
reason: "Same change as read-a-note"
granted: "2026-10-13"
expires: "2026-11-02"
$ plune diff baseline.json current.json --fail-on-regression --waivers waivers-old.yaml
Plune eval diff: 2 regression(s)
read-a-note: passed -> failed [regression]
never-change-a-note: passed -> failed [regression]
Gates:
regression: failed, passed → failed: read-a-note, never-change-a-note: 2 (limit 0)
expired waiver: regression of read-a-note on …: The prompt was reworded on purpose; the new wording is reviewed in the pull request
expired waiver: regression of never-change-a-note on …: Same change as read-a-note
2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable
$ echo $?
1

Expiry is read against the system clock; no flag or file field moves it. The file changes through pull requests like any other file, so a reviewer sees each waiver in the diff of the change that adds it. plune diff reads the file named by --waivers from the checkout it runs in.

--format markdown is the text of a pull-request comment: it starts with a hidden marker so one comment is updated rather than added each time. This is the output of the command on the two files above:

$ plune diff baseline.json current.json --fail-on-regression --max-tokens 10 --format markdown
<!-- plune-eval-diff -->
## Plune eval diff
### ❌ 2 regression(s)
| Eval | Baseline → Current | Change |
| --- | --- | --- |
| read-a-note | passed → failed | 🔴 regression |
| never-change-a-note | passed → failed | 🔴 regression |
### Gates
| Gate | State | Value | Threshold | Reason |
| --- | --- | --- | --- | --- |
| regression | ❌ failed | 2 | 0 | passed → failed: read-a-note, never-change-a-note |
| tokens | ❌ failed | 60 | 10 | input + output tokens of the current run |
_2 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 0 stable_
$ echo $?
1

And this is a comment the action posted on a live pull request of the plune-ai/eval-action repository, copied as it stands. The action ran from that pull request’s checkout with plune-version: 0.39.3 and use-mock: true (the mock model, so no provider was called), on two models with three repeats each, and nothing changed between the base branch and the pull request. It has no «Gates» section because that run set no gate input (the comment, the run):

<!-- plune-eval-diff -->
## Plune eval diff
### ✅ No regressions
_4 stable passes not listed_
### Advice (does not affect the exit code)
Advice only: nothing here changes the exit code, the state of an eval or a gate.
**Pass rates (at least 5 attempts with a verdict on each side; a guide, not a proof)**
- greet@qwen/qwen3.7-flash: too few attempts: at least 5 attempts with a verdict are needed on each side (current 3, baseline 3)
- greet@z-ai/glm-5.3-flash: too few attempts: at least 5 attempts with a verdict are needed on each side (current 3, baseline 3)
- echo@qwen/qwen3.7-flash: too few attempts: at least 5 attempts with a verdict are needed on each side (current 3, baseline 3)
- echo@z-ai/glm-5.3-flash: too few attempts: at least 5 attempts with a verdict are needed on each side (current 3, baseline 3)
- all evals together: no difference visible: 12/12 against 12/12: 0 pp, 95% interval -24.25 to +24.25 pp
_0 regression(s) · 0 improvement(s) · 0 new-fail · 0 pre-existing-fail · 0 errored · 0 removed · 0 not measured · 4 stable_

An agentic check needs no Plune account. plune run and plune diff do not call the Plune platform: the loop opens a connection only to the model provider’s API and to the server your config names, and the server’s headers go only to its own url. A mock run, as above, opens no connection to a model provider; it still starts your server. You can check this on your own machine by running the commands above under a tool that lists a process’s network connections, or on a machine with no network, where the mock run still passes.

This is such a check, made on 21 November 2026 with @plune-ai/cli 0.39.6 from npm, on Windows with Node.js 24, in an empty folder holding the server.mjs, plune.yaml and script.json of the first section. The CLI and the server’s dependency were installed first (that install talks to the npm registry; it is npm, before plune starts). Each run was watched two ways: the TCP and UDP endpoints of the plune process and of every server process it started, read with Get-NetTCPConnection and Get-NetUDPEndpoint while the run lasted, and a script loaded into plune with node --require that recorded every socket it opened and every DNS lookup. The second run has no mock: its provider URL points at a closed local port and its key is a placeholder, so it shows where a real-model run connects without paying for a call.

RunExitProcessConnections openedDNS lookups
PLUNE_MOCK_PROVIDER=1 PLUNE_MOCK_SCRIPT=script.json plune run0 (2 passed)plunenonenone
node server.mjs, one per eval (2)nonenot recorded
ANTHROPIC_BASE_URL=http://127.0.0.1:59999 plune run2 (2 errored, Connection error.)plune127.0.0.1:59999 only, 6 attempts (3 per eval)none
node server.mjs, one per eval (2)nonenot recorded

In neither run did plune connect to anything but the provider URL: no Plune host, no update check, no other address.

The same evals can run on more than one model. provider may be a non-empty list, and every eval then runs once on each model:

provider:
- type: anthropic
model: claude-sonnet-5-5
- type: openai
model: gpt-4o
temperature: 0
  • A cell is an eval on one model. With several models its result id is <eval id>@<model> (read-a-note@gpt-4o) and it carries a provider field with the type and model it ran on. A list of one model is the same as the object: the same ids, the same report.
  • The report. The console and Markdown reports of plune run and plune report add a Model matrix: a row per eval, a column per model, both in config order. A cell shows its state, the share of passing runs, the cost and the median latency. A cell with an error reads ERROR, one that was never judged NOT MEASURED, and a cell that was not run (--bail, --only) is a dash.
  • The diff. plune diff compares cells by their ids, so a cell that passed and now fails is a regression of that cell alone. --format json adds a matrix field. --format markdown shows the cells that changed or still fail, errored or were not measured, and folds the stable passes into one line with a count. The comment stays within 60,000 characters: over that, whole rows are dropped, least important first, the last line says N more cells in the job summary, and the hidden marker stays the first line.
  • The rules. The same type and model twice is a config error (exit 2). With several models an eval’s own provider may tune temperature or max_tokens, but not set type or model. --only <eval id> selects every cell of that eval. PLUNE_PROVIDER and PLUNE_MODEL name one model, so with a list of several they are refused (exit 2).
  • The judge. An llm-judge check without its own provider is judged by the model of the cell, so each model grades its own answer; plune run warns about it. Give the check a provider to use one judge for all cells.
  • The cost. Each cell is priced at its own model’s rates, and --dry-run sums them and says how many models it covers. The matrix multiplies a run by the number of models, on top of runs.

plune-ai/eval-action runs plune run on the pull request and on the base branch, posts the plune diff --format markdown text as one comment that it updates in place, and passes the gates on to plune diff. It has no comparison of its own: the CLI decides each gate. Each gate is off until its input is set, and the action passes a flag only for an input you set:

InputPassed to plune diff as
fail-on-regression ('true')--fail-on-regression
max-pass-rate-drop-percent--max-pass-rate-drop-percent
max-cost-usd--max-cost-usd
max-cost-increase-percent--max-cost-increase-percent
max-tokens--max-tokens
waivers--waivers, a path relative to working-directory
- uses: plune-ai/eval-action@v1
with:
fail-on-regression: 'true'
max-pass-rate-drop-percent: '5'
max-cost-increase-percent: '20'
waivers: .plune/waivers.yaml

The limits and waivers need a plune-version of 0.38.0 or newer; the action’s default is newer. With an older one the step stops with an error that says the pinned @plune-ai/cli does not know the flag and names it. With any gate set, the check follows the exit code of plune diff: 0 green, and 1 (a measured failure), 2 (an input error) or 3 (not measured) red, each with its own message. The gate step runs after the comment, so the comment and the job summary have their «Gates» section either way, with each gate passed, failed, unmeasured or waived.

OutputValue
has-regressiontrue when at least one eval went from passed to failed.
regressionsThe number of those regressions.
unmeasuredThe number of evals the diff could not judge because a run errored; 0 when everything was measured.
summaryA one-line summary of the diff.
gate-exit-codeThe exit code of plune diff (0, 1, 2 or 3) when fail-on-regression or a gate input is set; empty when none is, or when the base branch has no baseline yet.

On the first pull request that adds evals the base branch has no baseline: there is nothing to compare, no gate runs and the check stays green. The action runs on the mock provider by default (use-mock: 'true'), which writes no price, so a cost gate there is unmeasured and the check is red with exit 3: use the cost gates with a real provider. Required checks and a real provider are set up in the eval action recipes.

Some of what a run and a diff report is advice: it is shown, and nothing decides on it. No eval state, no count of the summary (unmeasured included), no gate and no exit code reads it, so adding or removing it never changes the exit code. There are three kinds, all in @plune-ai/cli 0.38.0 and newer.

Pass rates in plune diff. When an eval has repeats (runs above 1) in both runs, plune diff adds a section «Advice (does not affect the exit code)» to its console and Markdown output, and an advice field to --format json. For each such eval, and for all evals together when the two runs measured the same population (as for the pass-rate gate; otherwise the line says how they differ), it gives the 95 % interval of the difference of the two pass rates, current minus baseline, in percentage points (Newcombe’s method 10 over Wilson intervals), and one of four words:

WordWhen
probably worsethe whole interval is below zero
probably betterthe whole interval is above zero
no difference visiblethe interval holds zero; this is not «no regression»
too few attemptseither side has fewer than 5 attempts with a verdict

An attempt is one run of one row, and only an attempt with a verdict (passed or failed) is in the share: one that errored, was blocked or was skipped is left out, and the line says how many were not measured. A dry run measures nothing, so it is not compared. The runs of one row are related and the interval treats them as independent, so it is a guide, not a proof. In the comment above, each model ran each eval 3 times, so every eval reads «too few attempts», and only all evals together, 12 attempts on each side, get an interval.

Friction signals on an agentic row. plune run files these under the row’s agent.advice, beside the findings of no-repeated-call; the console and Markdown reports show them. Each names the tool, the steps and a count, and each reason says when it may be fine:

SignalWhat it saw
identical-retrya tool called again with the same arguments right after itself
pagination-continuationa tool called again with a new page or cursor; not a repeat
search-after-ida tool called again, in a later step and with other arguments, that returned a record an earlier step had already returned
friction-unmeasuredan answer could not be read for identifiers, so none were looked for; not the same as finding none

search-after-id gives the paths of the keys where it found the identifier, never the identifier itself.

An experiment with tool descriptions. plune run --tool-description <tool>=<text|@file> (repeatable) shows the model another description of a tool for that run only; the server is not changed. Each row records the SHA-256 of the text it was shown (agent.description_overrides). plune diff of a run with it against one without it, on the same population, adds a section «Experiment with descriptions»: the replacements of each run (tool name and SHA-256), the first tool the model called in each eval, and, where the eval has a first-tool check, its hit share on each side with the interval and word above. The text of a description is never printed.

@plune-ai/cli 0.39.5 and newer ship a skill, plune-mcp-evals, in the Agent Skills format: the folder skills/plune-mcp-evals of the npm package and of the CLI repository. An agent with it reads the server’s tool list with plune mcp snapshot, asks you for the 3 to 5 tasks that matter, writes the scenarios into plune.yaml with at least two negative ones, prices them with plune run --dry-run, and runs them only after you agree to the cost. It names only tools the server lists, and fixes a failing scenario by its prompt or its limits, never by weakening the check. It needs a key for the model provider the scenarios run on, and no Plune account.

Copy the folder where your agent reads skills, as for the qa-mission skill. For Claude Code, in a project that has @plune-ai/cli installed:

Terminal window
mkdir -p .claude/skills
cp -r node_modules/@plune-ai/cli/skills/plune-mcp-evals .claude/skills/
  • A real-model run, as said above, is described but not shown.
  • A run of several models is described but not shown: the mock model above is one model.