Monorepo & matrix

One package in a monorepo

Point the Action at a subdirectory with working-directory; config is resolved relative to it:

- uses: plune-ai/eval-action@v1
  with:
    working-directory: packages/chatbot
    config: plune.yaml          # i.e. packages/chatbot/plune.yaml
    base-ref: main

Several suites with a matrix

To evaluate multiple packages (or multiple configs) in one workflow, fan out with a matrix:

jobs:
  plune:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        package: [chatbot, search, summarizer]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: plune-ai/eval-action@v1
        with:
          working-directory: packages/${{ matrix.package }}
          config: plune.yaml
          fail-on-regression: true

Each leg runs independently, and fail-on-regression gates each one — so a regression in any package turns that leg (and the check) red.

One sticky comment per PR

The Action’s PR comment is sticky on a single fixed marker, so every matrix leg updates the same comment — the last leg to finish wins, and you only see one package’s diff. To keep the per-package gate while avoiding the clobber, turn the comment off on the matrix legs:

      with:
        working-directory: packages/${{ matrix.package }}
        comment: 'false'        # gate only; no comment collision
        fail-on-regression: true

If you want a visible diff comment too, run that one package in its own non-matrix job with comment: 'true'.

Only run when a package changes

Pair the matrix with paths filters (or a path-aware trigger) so each suite runs only when its package changes — cheaper CI and less comment churn:

on:
  pull_request:
    paths: ['packages/chatbot/**']

Next steps

fail-on-regression as a required check Make the gate enforce on protected branches.