Failed automated results

A failed automated result on a run’s page says what failed without a trip to the CI log. Its row and its panel show what the runner reported, and leave out what it did not:

• The headline — the first line of the error. On a test timeout, the line of the action that was still waiting, not «Test timeout of 30000ms exceeded».
• The steps — the test.steps the test declared, down to the one that failed: Pay › Confirm 3-D Secure. Hooks, fixtures and single actions are not steps, and a test that declares none shows no line at all.
• The place — where it failed, from the repository root: the line in the test’s own file, e2e/checkout.spec.ts:58:43. When that file is not on the error’s stack — a failure inside a fixture, say — it is the line the error was thrown from. With no repository found (below), there is no place.
• The artifacts — what the runner kept as files for the attempt, by the names it gave them: Playwright’s own include screenshot, trace, video, error-context, and a screenshot comparison’s <name>-expected, -actual and -diff. A test’s own attachment goes by the name the test gave it — by the file alone when that name is an absolute path, a ~ path or a file:// address. They are the rows of Attachments in the result’s panel, each with in the CI run ↗ when the run came from CI — or Open, for a JSON or a text file Plune keeps (Files, and where they are).
• The full text — click the row, or press Enter or Space on it: the result’s panel opens beside the list, and under What happened stands every error of the attempt, in order. Nothing is fetched for it; the row already has it.

Every row opens its panel, a passed result’s too: A result in a run says how to open it, where it stands and what it holds. A result from a client that sends no detail — a JUnit XML import, or @plune-ai/playwright before 0.2.6 — shows the first line of its text on the row and the rest in the panel.

The CI run

A run reported from CI links the CI run that ran the tests: CI run ↗ at the end of the run’s header line and of Where it ran in a result’s panel, and in the CI run ↗ beside each file under Attachments. The files stay in CI, and Plune keeps their names — all but the screenshots and the JSON and text files, which reach Plune themselves.

Who can open them, and for how long

CI keeps these for a limited time, and only people it lets see this repository’s artifacts can open them. On GitHub that is the artifact’s retention period — 90 days by default — and read access to the repository.

The link names the CI run that wrote the report, whoever sends it: a report saved in one job and imported by plune run import in another links the first job. A run opened without a link does not gain one from results that join it later.

Only a web address becomes a link. A report from a developer’s machine names no CI run, so its artifacts are listed by name, with no link — and a note under them says that only their names came with the result.

The commit and the branch

A run says which commit it ran on and on which branch. Click a result on the run page and its panel lists them under Where it ran: Branch, and Commit by its first seven characters — a row each, only when the run holds the value. GET /v1/runs/{id}/summary gives the whole commit in run.sha and the branch in run.branch, each null when the run has none.

The client that opens the run sends them, from the CI it ran on: @plune-ai/playwright 0.4.0 or newer, and plune run import in @plune-ai/cli 0.16.0 or newer — what each reads, and where, is in the reporter’s page and the import’s. An older client sends neither, and a run from a developer’s machine or from a CI the client does not know has none: the rows are not there, and nothing is guessed.

Linking to one result

Every row has an anchor: /runs/<run>#result-<result> opens the run with that result’s panel open and its row in the middle of the list. Each line of a case’s Result history links its own result this way. Back, Forward and ↑ ↓ follow the address: A result in a run.

Screenshots

A result that kept screenshots says how many beside its status, and its panel shows them under What happened, a passed result’s too. How they look, what a result keeps, for how long and who sees them is on A result in a run. A picture is not cleaned the way the text is (below), so do not keep one of a screen that shows a password, a token or a customer’s data.

To send them from your suite: Screenshots — @plune-ai/playwright 0.3.0, or plune run import in @plune-ai/cli 0.15.0.

What the text keeps out

• Colour codes. The text is shown as a terminal would have printed it, without the escapes.
• Credentials. A value that looks like a credential becomes a [redacted:<kind>] mark: known token formats; the values of cookie, set-cookie, authorization, proxy-authorization and any header whose name contains key, token, secret, password or session, as a request log prints them; session cookies; and the URL parameters token, code, key, secret, password, signature and sig.
• The machine, in what the reporter and plune run import of Playwright’s JSON send: paths into the repository are relative to its root, and the home folder is ~. Both find the repository as the nearest .git above Playwright’s rootDir — the import only when that folder is on the machine that imports. Without one — a Docker image that leaves .git out, say — only home folders become ~, and the result names no place. JUnit XML’s text is sent as the runner wrote it, paths and all.
• Length. The reporter and the Playwright JSON import send at most 512 KB of text, counted as the text weighs in the request, whole lines from the start and the end around …[omitted N lines]…; a JUnit XML import sends the text whole. The platform keeps at most 256 KB, start and end around …[truncated N bytes]…. The platform looks for credentials before its cut, and the CLI cuts only between whole lines. A chain longer than 20 steps keeps the 10 outer and the 10 inner around «… N steps …», and the artifacts list stops at 50 names and «N more».

A JSON or a text file a test attached goes through the same pass before Plune keeps it — colour codes and credentials out — and a JSON can stop being JSON for it: What Plune keeps.

Sending it from CI

The detail comes from what Playwright records, by either road:

• the reporter, @plune-ai/playwright 0.2.8 or newer, as the run happens;
• plune run import of Playwright’s JSON report, with @plune-ai/cli 0.14.2 or newer. JUnit XML carries the text, not the steps, the place, the files or the CI run.

This page describes those versions. The first to send the detail, 0.2.6 and 0.14.0, could put a path of the machine in the headline and in an attachment’s name, lose a whole batch to an attachment with an empty type, and count the 512 KB in raw bytes; and the 0.2.6 reporter could close the run before its last batches arrived. The next, 0.2.7 and 0.14.1, could still break an address whose host has the name of the repository root or of the home folder (with the root /app, http://app/login became http:login) and, with no repository root, keep the account’s name in a home path written with escaped slashes (\/home\/bob\/…) or in a webpack source map. The CLI changelog lists every fix of 0.14.1 and 0.14.2.

From adapter 0.3.0 and CLI 0.15.0 the screenshots among those files reach Plune as well, each with its own result, a passed test’s too — see Screenshots. From adapter 0.5.0 and CLI 0.17.0 so do the JSON and text files — see JSON and text files. The rest of the files, traces and videos among them, stay in the CI run.

From adapter 0.4.0 and CLI 0.16.0 the run also names the commit and the branch it ran on — see The commit and the branch.

The CI link is Playwright’s own record of the CI run (metadata.ci), which it writes on GitHub Actions and GitLab CI. Upload the files the artifacts name from the job that ran the tests:

- name: Test
  run: npx playwright test

- name: Keep what failed
  if: ${{ !cancelled() }}
  uses: actions/upload-artifact@v4
  with:
    name: playwright-failures
    path: test-results/
    retention-days: 14

!cancelled() rather than failure(): a test that fails and then passes on a retry leaves the job green, and the run page still lists the files of its failed attempt.

A folder without a dot — and mind what is in it

actions/upload-artifact skips files and folders whose names start with a dot, so an outputDir such as .artifacts/ uploads nothing — the step only warns that no files were found, and the job stays green. Keep Playwright’s output in a folder without a leading dot, like the default test-results/. And keep a saved login (storageState) out of that folder: everyone who can read the repository’s artifacts can download what is in it, and a session file is a way in. A trace and a HAR are the same risk — they keep every request with its headers and cookies — and a video shows every screen the test saw, so sign tests in with an account made for testing.