Configurations

A configuration is a group of values a run is launched on — Browser: Chrome, Firefox, Safari; Operating system: Windows, macOS, Linux. Pick Chrome and Firefox × Windows and macOS when you launch, and the run gets four combinations — Chrome · Windows, Chrome · macOS, Firefox · Windows, Firefox · macOS — each with its own set of results. It stays one run: one page, one verdict, one row on the Runs list.

Picking values is optional. A run launched with none gets one set of results, as every run did before.

Groups and values

Settings → Configurations, right after Statuses, has a card per group. The card’s head says how many values the group holds and how many runs used it — «3 values · 1 archived · used in 2 runs» — and each value says the same of itself: «used in 1 run».

• Add group opens a dialog: the Group name, then the Values — type one in New value and press Add or Enter — and Save.
• Edit on a group opens the same dialog: rename the group, add a value, or drag a value by its handle to change the order; the arrow keys move it too. That order is the order of the launch form, of the combinations and of the report’s columns.
• Archive on a value takes it out of the launch form. Past runs keep it in their reports and filters; the value stays on the card, grey — «Archived · used in 1 run» — with Restore.
• Archive group does the same for a whole group.

Nothing here is deleted. A value once saved can be archived, not removed: in the dialog, Remove is there only for a value added and not saved yet.

A group is not one of the run’s fields. Settings → Fields does not list it, and it can be neither required nor given a default value: a required group would make every run pick configurations, and a default would multiply runs nobody picked them for.

Only an owner of the project changes groups and values. A member and a reader see the cards, the values and their counts, with «Only an owner of the project can change these.» and nothing to press.

Launch on configurations

The Manual run dialog — New run ▾ › Manual run… on Runs, or Run manually in the cases tree (Starting one) — has a Configurations · optional block: each group with its live values as boxes to tick. An archived value is not offered. Under the boxes the block says, as you tick, what the run will get:

• Nothing ticked — «No configuration picked — the run gets one set of results, as today.» The button stays Launch.
• One group — «Chrome, Firefox, Safari = 3 combinations». A group with nothing ticked does not multiply: «Operating system has no value ticked, so it does not multiply: 1 × 3.»
• Two groups — «Chrome, Firefox × Windows, macOS = 4 combinations», the four listed, and «Each combination gets its own set of results: 12 cases × 4 = 48 results.»

The line at the top of the dialog follows it — «12 cases × 4 configurations → run «…»» — and the button becomes Launch · 4 configurations.

The combinations come in one order: the first group is the outer one, and each group’s values go in their order in Settings → Configurations, not in the order you ticked them. The same pick is always the same list.

Plune multiplies the pick, not the browser: the dialog sends the ticked values, and the run keeps the list of combinations it was launched with, names included. A group renamed, reordered or archived later does not change a run already launched. Nor can the run’s own pick change afterwards: on its page the groups are read-only.

A plan’s Launch run has no configurations block: a run from a plan is launched without them. Launching on several environments at once works with configurations too — Environments.

Limits

What	At most	What the dialog says past it
Combinations in a run	25 — 5 × 5, or 3 × 3 × 2	«A run takes at most 25 combinations — untick some values to launch it.»
Results a run expects: cases × combinations	40 000 — 10 000 cases on 4 combinations	«A run takes at most 40,000 results: 12,000 cases × 4 = 48,000. Untick some values or run fewer cases.»
Groups in a project	50 live ones	—
Values in a group	100	—

Past the first two, Save and Launch wait until the pick fits; the API refuses the same with 400.

Each combination’s results are results like any other: 120 cases on four combinations are 480 results — in the run’s counts, and in the results your plan counts.

The report

A run launched on configurations opens with its combinations. The head says how many — «4 configurations», or «4 configurations, 1 without results» — and the heading counts cases: «4 of 8 cases failed». A case stands at its worst combination, in the order failed, broken, blocked, muted, passed, skipped, so one failure on Firefox · macOS is a failed case. The counts beside it are results, every combination’s: 8 cases on 4 combinations are «32 in total».

• A tile per combination, under Configurations · 4: how many passed out of its results — «3 / 8 passed» — and what did not: «3 failed · 1 broken · 1 skipped».
• The matrix, Results by configuration: a row per case, a column per combination, a cell per result. A cell opens its result in the panel, as a row of the list does; a combination that has no result for a case shows —. Under a case that failed on exactly one combination the row says where: «Fails only on Firefox · macOS», or «Broken only on Firefox · macOS».

Pick one combination — its tile, or Configuration · all and its name — and the page shows its results as the plain list, under «Results · Firefox · macOS», with the filters of the run’s report and that combination’s counts on the status chips. The head and the counts at the top stay the whole run’s. All configurations brings the matrix back. The pick lives in the page’s address, so a link opens the same combination:

/runs/<id>?combination=3

Combinations are counted from 0 in the order of the list: 3 is the fourth.

A combination that sent nothing

A combination with no result has no rows: no «skipped» is made up for it. Its tile says no results — «Nothing reported — not counted as skipped.» — its column stays empty, and a line under the matrix says why: «Firefox · Windows sent no results: its column stays empty and nothing is counted as skipped — 8 «skipped» rows would hide that it never ran.»

Marking a manual run

In an open manual run on configurations, the tiles pick the combination your marks go into. Until one is picked, the page says «Pick a configuration above to mark its cases — each one gets its own results.» A case is marked once per combination: marked on Chrome · Windows, it is still to run on Firefox · Windows.

Finish run counts what is left in pairs — «finished · 5 case-and-configuration pairs not run».

Launch it again

Re-run N failed and Launch a copy (Launch it again) keep the run’s list of combinations as it is — the same combinations, the same names. Re-run N failed takes a case that failed on any combination and runs it on all of them; there is no re-run of one combination alone.

A case’s history

On a case’s page, Result history lists every result, each combination’s too, so a run on four combinations is four lines there.

Stability — the flaky badge on the cases list and the Flaky count in Analytics — reads such a run as one verdict: the worst of its combinations. A case that fails on Firefox · macOS every night and passes on Chrome is not flaky for that — it fails, night after night. A run without configurations is read as before, a result at a time.

Who may do what

	Owner	Member	Reader
See the groups, the values and their counts	yes	yes	yes
Add, edit, archive and restore groups and values	yes	no — 403	no — 403
Launch a run on configurations, mark its cases	yes	yes	no — 403

The API

• GET /v1/configurations — the groups, archived ones too, with their values; each group and each value carries runs, how many live runs hold it. Any member of the project reads it.

• A group is a run field with a role, written through /v1/custom-fields — an owner only:

  { "entity": "run", "type": "list", "role": "configuration", "name": "Browser", "options": [{ "label": "Chrome" }, { "label": "Firefox" }] }

  PATCH /v1/custom-fields/{id} takes the group’s name and its whole options list in the new order, with "archived": true or false on a value. A list that leaves out a saved value answers 400 — option 'Firefox' of a configuration group cannot be removed — archive it instead. A group required or with a defaultValue answers 400. POST /v1/custom-fields/{id}/archive and /restore archive and restore the group; past 50 live groups a new one answers 429.

• POST /v1/runs (schemaVersion: 2) takes the pick in custom — the group’s id and the ids of its ticked values, "custom": { "<group id>": ["<value id>", "<value id>"] } — and answers the run with combinations, the list in its order, each combination’s values with their group and label. Past a limit it answers 400 with the numbers, before anything is written: 3 × 3 × 3 = 27 combinations — a run takes at most 25, or 12000 expected cases × 4 combinations = 48000 results — a run takes at most 40000. An archived value answers 400 too.

• PATCH /v1/runs/{id} that changes a group’s values answers 400 — custom field 'Browser' is a configuration group — configurations are fixed at launch. The same values sent back are accepted.

• POST /v1/runs/{id}/results takes the combination of the batch — one per batch — as combination, its number in the run’s list counted from 0, or as configuration, by names: { "Browser": "Chrome", "Operating system": "Windows" }, matched without regard to case. Into a run on configurations a batch with neither answers 422; so does a combination sent into a run without configurations, or a number outside the list.

• A configuration the run does not have yet joins the end of its list, within the same limits — past them, 422 with the numbers. It names exactly the groups of the run’s combinations, or answers 422. A group or a value that Settings → Configurations does not have is taken as it was sent.

• GET /v1/runs/{id}/results?combination=3 — one combination’s results; every result carries combination. GET /v1/runs/{id}/summary answers byCombination — each combination’s counts, total and notRun, in the order of the list — and caseCounts, the cases by where each stands.