Pilot, scorers & judge
Every run ends with an evaluation that combines three perspectives:
Deterministic scorers
Section titled “Deterministic scorers”Hard, repeatable checks computed on every run: green %, flakiness, semantic checklist coverage. Same inputs → same score, so trends are comparable across runs.
LLM judge
Section titled “LLM judge”A model reviews outputs case by case where rules can’t capture quality — did the generated test actually verify what the case described?
Pilot verdict
Section titled “Pilot verdict”One holistic verdict for the whole run — the thing you’d put in a CI gate. Since v0.3.0 the Pilot
verdict runs on the strong reasoner role (Anthropic claude-opus-4-8) rather than the cheap
judge tier, for sounder go/no-go calls:
Metrics
Section titled “Metrics”Every run scores itself. The numbers appear in the console (=== Metrics ===), in each run’s
report.md (with a one-line meaning per metric), and in Langfuse when configured.
↑ = higher is better, ↓ = lower is better. case_redundancy and flaky_ratio are the only
“lower is better” metrics — every other metric is higher-is-better.
These mirror Cairn’s source of truth,
src/eval/legend.ts (METRIC_LEGEND)
— same metric names, directions, and blurbs.
Deterministic (computed from run data, no LLM):
| metric | direction | meaning |
|---|---|---|
runs_green | ↑ higher is better | Share of generated tests that pass on validation. |
flaky_ratio | ↓ lower is better | Share of tests classified flaky (inconsistent pass/fail). |
verified_ratio | ↑ higher is better | Share of identified elements that resolve to exactly one element (unique locator). |
grounding | ↑ higher is better | Share of cases whose element refs all point to real on-page elements (no hallucinated refs). |
locator_quality | ↑ higher is better | Share of user-facing locators (getByRole/Label/Text…) vs fragile (.locator/getByTestId). |
locator_robustness | ↑ higher is better | Weighted selector strength: role 1.0 > label/text 0.8 > test-id 0.5 > css 0. |
technique_coverage | ↑ higher is better | Distinct test techniques used out of the 6 (ISO/IEC/IEEE 29119-4). |
case_redundancy | ↓ lower is better | Share of cases that are near-duplicates of another (0 = all distinct). |
Judge (LLM-scored):
| metric | direction | meaning |
|---|---|---|
test_case_quality | ↑ higher is better | Holistic quality of the cases (clarity, correctness, usefulness). |
methodology_adherence | ↑ higher is better | How well the cases follow the testing methodology. |
checklist_coverage | ↑ higher is better | Semantic coverage of the provided checklist by the cases. |
The holistic Pilot verdict is separate — a pass / needs-work / fail judgment on the whole run, not a 0–1 score.
Keep-best self-repair
Section titled “Keep-best self-repair”When a generated spec fails, Cairn attempts repair (up to MAX_REPAIR times, default 2) with a
keep-best guarantee: a repair never makes the suite worse than the best state seen so far.
Learning across runs
Section titled “Learning across runs”Best cases accumulate between runs — results-level learning works fully offline from
runs/<id>/report.json, and richer tracing is available with Langfuse
(optional).