Skip to content

The model x case matrix of a run, and the trend of each model

GET
/v1/runs/{id}/models
curl --request GET \
--url https://beta-api.plune.ai/v1/runs/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/models \
--header 'Authorization: Bearer <token>'

Counted on the server off the stored RunResult of the run as plune sync sent it (evals[].provider, evals[].measured, rows[].attempts; #2462). Each cell is one case on one model: the share of attempts that got a verdict, the broken ones counted apart and never in the share, the cost and the median latency. A model that never ran a case is a null cell. trend holds this run and the earlier synced runs of the project (at most 20 looked at) whose population - the cases, each with its data, its checks and its repeats - is the same; a run whose population is unknown (an eval that did not say what it measured) has no trend: different populations are never compared.

id
required
string format: uuid

The matrix (empty models for a run that names no model) and the trend

Media type application/json
object
runId
required
string
population
required
object
fingerprint

Sha256 of the population; null when it is unknown.

string | null
reason

Why there is no fingerprint - present only then.

string
Allowed values: unmeasured
minJudgedAttempts
required

The judged attempts a cell needs before a difference is worth stating.

integer
models
required
Array<object>
object
type
string
model
string
rows
required
Array<object>
object
eval
string
testCaseId

The case whose evalId this is; null when none is.

string | null
title
string | null
cells

One per entry of models, in that order; null where that model never ran the case.

Array
One of:
object
resultId

The eval result this cell is: <eval>@<model>, or the eval id of a one-model run.

string
passed
integer
failed
integer
broken

Attempts without a verdict (an error, or a row that said blocked or skipped): counted here, never in share.

integer
share

passed / (passed + failed); null when nothing got a verdict - never 0 or 1 for an empty cell.

number | null
costUsd
number
priced

False: no row had a known price, and costUsd is a placeholder 0.

boolean
latencyMs

Median over the attempts that reported one; null when none did.

number | null
lowSample

Fewer judged attempts than minJudgedAttempts: show the numbers, claim no difference.

boolean
unattributed
required

Evals that name no model, left out of the matrix rather than guessed into a column.

integer
trend
required

Runs of the same population, oldest first, this run last.

Array<object>
object
runId
string
createdAt
string format: date-time
models
Array<object>
object
model
string
passed
integer
failed
integer
broken
integer
share
number | null
Example
{
"runId": "7c0e4a52-6f1d-4d1e-9a51-2f8f0d6c9b11",
"population": {
"fingerprint": "4f2a9c1e6b8d3a7f0c5e2b9d4a6f8c1e3b7d0a5f9c2e4b6d8a1f3c5e7b9d2a4f"
},
"minJudgedAttempts": 5,
"models": [
{
"type": "openai",
"model": "gpt-a"
},
{
"type": "openai",
"model": "gpt-b"
}
],
"rows": [
{
"eval": "lookup",
"testCaseId": "0b6f3c2a-1d4e-4f5a-8b9c-7d6e5f4a3b2c",
"title": "Lookup",
"cells": [
{
"resultId": "lookup@gpt-a",
"passed": 7,
"failed": 1,
"broken": 0,
"share": 0.875,
"costUsd": 0.021,
"priced": true,
"latencyMs": 840,
"lowSample": false
},
{
"resultId": "lookup@gpt-b",
"passed": 3,
"failed": 3,
"broken": 2,
"share": 0.5,
"costUsd": 0.008,
"priced": true,
"latencyMs": 1210,
"lowSample": false
}
]
}
],
"unattributed": 0,
"trend": [
{
"runId": "7c0e4a52-6f1d-4d1e-9a51-2f8f0d6c9b11",
"createdAt": "2026-11-20T09:00:00.000Z",
"models": [
{
"model": "gpt-a",
"passed": 7,
"failed": 1,
"broken": 0,
"share": 0.875
},
{
"model": "gpt-b",
"passed": 3,
"failed": 3,
"broken": 2,
"share": 0.5
}
]
}
]
}

No / invalid token or session

Not found or not yours

Media type application/json
object
error
required
string
Example generated
{
"error": "example"
}