Results of agentic (e2e) tests

The e2e test framework drives an app with an AI agent and can write its results as a JUnit XML file. plune run import reads JUnit, so those results reach Plune with nothing installed into the suite and no provider key: the importer generates nothing, it only records.

This page is the whole recipe. It is also honest about what a JUnit file cannot carry — see What is lost on this path.

The flow

1. Put the case id in the test’s name. An @P<id> anywhere in the name says which Plune case the test is, whatever the rest of the name says. Use the id of the case as it shows in Plune (a UUID such as 3f6c1a52-8d0e-4b7a-9c21-5e4d7a90b1f3):

   import { test } from '@e2e-dev/web';
   
   test('signs in with a valid password @P3f6c1a52-8d0e-4b7a-9c21-5e4d7a90b1f3', async ({ app, agent }) => {
     await app.open('/login');
     await agent.act('sign in as the demo user');
     await agent.assert('the dashboard is shown');
   });

2. Run the tests with the junit reporter. It writes the report beside report.json:

   npx e2e run --reporter list,junit
   # → .e2e/junit.xml

3. Import it. PLUNE_TOKEN is read from the environment, as for any import:

   export PLUNE_TOKEN="plune_xxxxxxxx"
   npx -y @plune-ai/cli run import .e2e/junit.xml

Bring an existing suite explains the output (accepted, unmatched, …), --create and the project’s status map; all of it applies here unchanged.

Does the [targetId] suffix break the @P match?

No. The JUnit reporter writes each case’s name as

<describe > … > test title>[ repeat suffix] [targetId][ [agent]]

so a target id (and an agent name, when it is not default) follows the title in square brackets after a space. The importer reads the token with @P([A-Za-z0-9_-]+) over the whole name. A ], a [ or a space ends the id, so the suffix is never part of it:

Name in junit.xml	Case id read
Login > Sign in @P123 [abc]	123
Login > Sign in @P3f6c1a52-8d0e-4b7a-9c21-5e4d7a90b1f3 [abc]	3f6c1a52-8d0e-4b7a-9c21-5e4d7a90b1f3
Login > Wrong password @P124 [abc] [admin]	124
Login > No token [abc]	none — arrives unmatched

A tag whose id is not a case of the project (a typo, a case of another project, an id with a trailing -) is not “unmatched”: the import counts it as rejected (refused). “Unmatched” is only a test with no @P at all and no classname#name match. Case ids are the long ids Plune shows for a case (UUIDs).

Playwright’s own junit reporter does not add [targetId]: it writes Describe › title with classname="file.spec.ts". The suffix is the e2e framework’s; the @P match is the same in both.

Checked on a small hand-written report with names of exactly this shape, read by the CLI’s JUnit reader without sending anything anywhere. Three things to keep in mind:

• Leave a space or the end of the title after the id. The id may contain - and _, so a title such as …@P3f6c1a52-8d0e-4b7a-9c21-5e4d7a90b1f3- would read the trailing dash as part of the id. The suffix e2e adds always starts with a space, so this only matters for what you type yourself.
• The suffix stays in the name that is sent, as the token is (nothing is cut out of a name). A test that is not tagged is matched by classname#name, which then includes the suffix: renaming the target renames the test for that fallback. The @P token is what makes a name safe to change.
• One case, several targets. A test that runs against two targets gives two JUnit cases with the same @P; both are recorded against the same Plune case.

Why a JUnit file from e2e is not read as a Playwright one

The importer treats a report as Playwright’s own when classname ends in .spec.ts or .test.ts. e2e puts the test file there (tests/login.e2e.ts), so it is read as any other runner’s JUnit: the token is looked for in the whole name. A file named login.test.ts would be read the Playwright way, and the token is then taken from the last › part of the name — e2e joins its titles with >, so that is still the whole name and the result is the same.

In CI

- name: Test
  run: npx e2e run --reporter list,junit

- name: Report the run to Plune
  if: always()
  continue-on-error: true
  env:
    PLUNE_TOKEN: ${{ secrets.PLUNE_TOKEN }}
    PLUNE_ENV: ci
  run: npx -y @plune-ai/cli run import .e2e/junit.xml

if: always() because a failed run is the one worth recording; continue-on-error so a hiccup on our side never turns your build red. Add --create on the first import to offer the untagged tests to the review queue.

What is lost on this path

JUnit has fewer outcomes than e2e’s own report, so three things change on the way:

In e2e	In the JUnit file	In Plune
flaky (failed, then passed on a retry)	a passed case with a note in system-out	passed — the failed attempts are not visible
interrupted (the run was stopped before a verdict)	a <skipped> case with the reason in its message	skipped, not blocked: Plune’s blocked means «the product was not judged», and a JUnit file cannot say that
timed-out, a configuration or an infrastructure error	a <failure> or an <error>, by the error’s category	whatever your project’s status map gives failure and error — check that an infrastructure error is not counted as a product failure
steps, the agent’s explanations	not in the file	no steps on the result
tokens, model, cost	not in the file	no usage on the result

The first two matter for trust: a flaky test looks green, and a stopped run looks like a skipped one, so a number of «skipped» in a Plune run can hide a run that was cut short. Read the e2e output, or the report.json next to the JUnit file, when that number is not what you expect.

Screenshots and files can still travel: a test that prints a [[ATTACHMENT|path]] line brings its file along — see Files from JUnit & Allure.

What a native importer would give

Reading e2e’s own report-1 JSON instead of JUnit would keep what this path drops: flaky with its attempts, interrupted as blocked, timed-out as broken, the steps with their labels, and the tokens and cost of each result. There is no such importer today and no open request for one; if you need it, say so on the issue tracker.

What was checked

The import was run end to end (@plune-ai/cli 0.28.0, plune run import) against a local Plune instance, on the JUnit file of a real Playwright run and on a file in the e2e shape (> joins, a [targetId] suffix, an extra [agent]). @P matched the right cases (passed, failed, skipped), the suffix did not interfere and stayed in the stored name, and two targets of one @P gave two results on one case. A passed case with a system-out note is stored as passed, a <skipped> as skipped, an <error> as broken.

Not done: a run of the e2e framework itself, which needs its own model provider. The exact XML it writes and the flaky and interrupted rows are taken from the reporter’s source, not observed.

Bring an existing suite The import itself: how a case is matched, what the output means, --create, and one run from several jobs.