Skip to content

MCP server checks in CI

From eval-action v1.9.0 the repository holds a second action, in its mcp/ folder: plune-ai/eval-action/mcp@v1. On every pull request it runs plune mcp check, lint and diff against your server and gives you:

  • an annotation on the run for every finding;
  • one pull request comment, updated in place on every run, apart from the eval-diff comment;
  • the same report in the job summary, also when the check is red;
  • the reports as an artifact: <command>.json, <command>.xml (JUnit) and <command>.md;
  • a gate: the check is red when something failed, when the server could not be tested, and when nothing was measured.

The action holds no check of its own. Each command runs once in @plune-ai/cli, which writes every report form in that one run, and the action posts what the CLI wrote. Like the commands, it needs no Plune account and does not talk to the Plune platform.

The action starts the server itself from server-command. Copy this into .github/workflows/mcp.yml:

name: MCP server
on:
pull_request:
branches: [main]
permissions:
contents: read
pull-requests: write # only for the PR comment; with comment: 'false' remove this line
jobs:
mcp:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v6
with:
node-version: '20'
- run: npm ci && npm run build
- uses: plune-ai/eval-action/mcp@v1
with:
steps: lint,check,diff
# Split on spaces, no shell: for anything more, point it at a script in your repository.
server-command: node dist/server.js
snapshot-file: mcp-snapshot.json

diff compares the server with a snapshot you created on your machine and committed (see Snapshot and diff):

Terminal window
npx -y @plune-ai/cli mcp snapshot --out mcp-snapshot.json -- node dist/server.js

No secret is involved, so this job runs on pull requests from forks too.

A step starts the server in the background; check keeps trying to reach it for wait, so no sleep is needed, and the action runs check before lint and diff. The credentials go from a secret to an env: variable of the action’s step, and the action names the variable, never the value:

name: MCP server
on:
pull_request:
branches: [main]
permissions:
contents: read
pull-requests: write # only for the PR comment; with comment: 'false' remove this line
jobs:
mcp:
if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v6
with:
node-version: '20'
- run: npm ci && npm run build
- name: Start the server
run: node dist/http-server.js > "$RUNNER_TEMP/server.log" 2>&1 &
env:
PORT: '3000'
MCP_AUTH_TOKEN: ${{ secrets.MCP_AUTH_TOKEN }}
- uses: plune-ai/eval-action/mcp@v1
with:
steps: lint,check
server-url: http://127.0.0.1:3000/mcp
wait: 30s
headers-from-env: Authorization=MCP_AUTHORIZATION
env:
# The secret holds the whole header value ("Bearer <token>").
MCP_AUTHORIZATION: ${{ secrets.MCP_AUTHORIZATION }}
# Fork and Dependabot pull requests: no secret, so test a server that needs none.
mcp-fork:
if: github.event.pull_request.head.repo.full_name != github.repository || github.actor == 'dependabot[bot]'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v6
with:
node-version: '20'
- run: npm ci && npm run build
- name: Start the server without auth
run: node dist/http-server.js > "$RUNNER_TEMP/server.log" 2>&1 &
env:
PORT: '3000'
- uses: plune-ai/eval-action/mcp@v1
with:
steps: lint,check
server-url: http://127.0.0.1:3000/mcp

The action reads no secret itself. A header’s value travels from your secret to the env: of the step and from there to the CLI by the variable’s name (headers-from-env: Header=ENV_VAR), so it is never on a command line, and the reports mask it wherever the server repeats it.

Keep the whole header value in the secret — Bearer <token> — and pass it as it is. Written as Bearer ${{ secrets.TOKEN }}, a missing secret would leave the variable at Bearer , which looks set. When a variable named in headers-from-env is empty or unset, the action stops with an error naming the variable before it starts or contacts the server.

A pull request from a fork gets no secrets, and neither does one opened by Dependabot: its runs get Dependabot secrets only, not Actions secrets (GitHub docs). That is why the HTTP workflow above has two jobs: the one with the secret runs only for branches of this repository, and the other tests a server started without credentials. On a fork the comment is skipped, because the token is read-only; the annotations, the job summary, the artifacts and the gate still work. The action never uses pull_request_target.

ExitMeaningDefaultfail-on: never
0Passed, and everything was measured.greengreen
1A finding or a check failed (the annotations and the report name it). A server that started and exited before it answered initialize is a failed check: the log has the last lines of its stderr, secrets masked.redgreen, with a notice naming the command
2The server could not be tested: its command was not found or could not be started, it could not be reached, refused the credentials, or a flag is wrong. The log has the last lines of its stderr, secrets masked.redgreen, with a warning; measured=false
3Nothing was measured: no tools listed, nothing to compare, a check that could not run.redgreen, with a warning; measured=false

A wrong input — both server-command and server-url, an unknown step, diff without snapshot-file, an empty headers-from-env variable — stops the action with an error before anything starts, whatever fail-on says.

fail-on: never turns the whole action into a report: nothing fails the check, diff included.

- uses: plune-ai/eval-action/mcp@v1
with:
server-command: node dist/server.js
fail-on: never

“Nothing measured” stays visible all the same. A server that did not start (exit 2) or listed nothing (exit 3) gives a warning on the run, not a notice, and the measured output is false for a later step to read. A green check from a server that was never tested would look like a healthy one, so the action does not let the two look alike.

For an advisory lint with a blocking diff, use the action twice: one step with steps: lint and fail-on: never, one with steps: diff, each with its own artifact-name, and comment: 'false' on one of them.

Each finding is an annotation on the run. This is what plune mcp lint --format github prints for the sample server, the lines the runner turns into annotations:

$ npx -y @plune-ai/cli@0.39.9 mcp lint --format github -- node server.mjs
::warning title=desc-missing get_note::get_note: The tool has no description.%0Aexpected: a description%0Aactual: none%0Afix: Describe what the tool does, when to use it and when not to: the model chooses tools by name and description alone. The specification does not require a description; this is advice on quality.%0Aspec: https://modelcontextprotocol.io/specification/2025-11-25/schema#tool-description
::warning title=list-no-paging list_notes::list_notes: "list_notes" returns a list by its name, but cannot be asked for less.%0Aexpected: one of: limit, cursor, page, offset, pagesize, perpage, per_page, page_size, pagetoken, page_token%0Aactual: parameters: none%0Afix: Let the caller bound the answer: add a limit and a cursor (or page) parameter.
::warning title=param-enum-in-text set_status.status::set_status.status: The parameter description lists a closed set of values, but the schema does not.%0Aexpected: an enum%0Aactual: a free string described as "The new status: one of open, done, archived."%0Afix: List the allowed values in "enum" rather than in the description, so the client can check them.
plune mcp lint: 21 passed, 0 failed, 0 could not run, 7 advisory or not applicable; exit 0.
$ echo $?
0

The comment is the Markdown each command writes. For the same server, the lint part:

$ npx -y @plune-ai/cli@0.39.9 mcp lint --format markdown -- node server.mjs
<!-- plune-mcp-lint -->
### MCP lint
**Exit 0** — 21 passed, 0 failed, 0 could not run, 7 advisory or not applicable
| State | Level | Check | Target | Expected | Actual | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| failed | warn | desc-missing | get_note | a description | none | Describe what the tool does, when to use it and when not to: the model chooses tools by name and description alone. The specification does not require a description; this is advice on quality. |
| failed | warn | list-no-paging | list_notes | one of: limit, cursor, page, offset, pagesize, perpage, per_page, page_size, pagetoken, page_token | parameters: none | Let the caller bound the answer: add a limit and a cursor (or page) parameter. |
| failed | warn | param-enum-in-text | set_status.status | an enum | a free string described as "The new status: one of open, done, archived." | List the allowed values in "enum" rather than in the description, so the client can check them. |
$ echo $?
0

Server text in a cell cannot add HTML, a mention, a link or an image to the comment: those characters are written as character references.

InputDefaultWhat it does
stepslint,checkCommands to run, comma-separated: lint, check, diff. They run in the order check, lint, diff.
server-commandCommand that starts a stdio server, split on spaces: no quotes, no shell. Not with server-url.
server-urlURL of a Streamable HTTP server started by an earlier step. Not with server-command.
wait30sHow long check keeps trying to reach server-url.
headers-from-envHeader=ENV_VAR, comma-separated; each value is read from that variable. Only with server-url.
fail-onerrorLint threshold: error, warn, info or never. never makes the whole action advisory.
ignore-fileThe lint exceptions file.
snapshot-fileThe committed snapshot diff compares with. Required when steps has diff.
diff-fail-onbreakingDiff threshold: breaking, any or none.
smokefalsetrue adds --smoke to check.
commenttruePost and update one PR comment. On a second use of the action in one PR, set 'false'.
debug-artifactfalseAlso upload the frame journal of each command. It can hold what the tools returned.
artifact-nameplune-mcp-reportsName of the reports artifact; give each use its own.
working-directory.Where server-command, ignore-file and snapshot-file are found.
plune-version0.39.9Version of @plune-ai/cli to install from npm.

Outputs: lint-exit, check-exit, diff-exit (empty when the command did not run), measured (true when no command that ran exited 2 or 3), summary and report-dir. Only contents: read is needed; add pull-requests: write for the comment.