MCP server checks in CI
From eval-action v1.9.0 the repository holds a second action, in its
mcp/ folder: plune-ai/eval-action/mcp@v1. On every pull request it runs
plune mcp check, lint and diff against your server and gives
you:
- an annotation on the run for every finding;
- one pull request comment, updated in place on every run, apart from the eval-diff comment;
- the same report in the job summary, also when the check is red;
- the reports as an artifact:
<command>.json,<command>.xml(JUnit) and<command>.md; - a gate: the check is red when something failed, when the server could not be tested, and when nothing was measured.
The action holds no check of its own. Each command runs once in @plune-ai/cli, which writes every
report form in that one run, and the action posts what the CLI wrote. Like the commands, it needs no
Plune account and does not talk to the Plune platform.
A stdio server
Section titled “A stdio server”The action starts the server itself from server-command. Copy this into
.github/workflows/mcp.yml:
name: MCP server
on: pull_request: branches: [main]
permissions: contents: read pull-requests: write # only for the PR comment; with comment: 'false' remove this line
jobs: mcp: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4
- uses: actions/setup-node@v6 with: node-version: '20' - run: npm ci && npm run build
- uses: plune-ai/eval-action/mcp@v1 with: steps: lint,check,diff # Split on spaces, no shell: for anything more, point it at a script in your repository. server-command: node dist/server.js snapshot-file: mcp-snapshot.jsondiff compares the server with a snapshot you created on your machine and committed (see
Snapshot and diff):
npx -y @plune-ai/cli mcp snapshot --out mcp-snapshot.json -- node dist/server.jsNo secret is involved, so this job runs on pull requests from forks too.
An HTTP server started in the same job
Section titled “An HTTP server started in the same job”A step starts the server in the background; check keeps trying to reach it for wait, so no
sleep is needed, and the action runs check before lint and diff. The credentials go from a
secret to an env: variable of the action’s step, and the action names the variable, never the value:
name: MCP server
on: pull_request: branches: [main]
permissions: contents: read pull-requests: write # only for the PR comment; with comment: 'false' remove this line
jobs: mcp: if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4
- uses: actions/setup-node@v6 with: node-version: '20' - run: npm ci && npm run build
- name: Start the server run: node dist/http-server.js > "$RUNNER_TEMP/server.log" 2>&1 & env: PORT: '3000' MCP_AUTH_TOKEN: ${{ secrets.MCP_AUTH_TOKEN }}
- uses: plune-ai/eval-action/mcp@v1 with: steps: lint,check server-url: http://127.0.0.1:3000/mcp wait: 30s headers-from-env: Authorization=MCP_AUTHORIZATION env: # The secret holds the whole header value ("Bearer <token>"). MCP_AUTHORIZATION: ${{ secrets.MCP_AUTHORIZATION }}
# Fork and Dependabot pull requests: no secret, so test a server that needs none. mcp-fork: if: github.event.pull_request.head.repo.full_name != github.repository || github.actor == 'dependabot[bot]' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v6 with: node-version: '20' - run: npm ci && npm run build - name: Start the server without auth run: node dist/http-server.js > "$RUNNER_TEMP/server.log" 2>&1 & env: PORT: '3000' - uses: plune-ai/eval-action/mcp@v1 with: steps: lint,check server-url: http://127.0.0.1:3000/mcpSecrets
Section titled “Secrets”The action reads no secret itself. A header’s value travels from your secret to the env: of the
step and from there to the CLI by the variable’s name (headers-from-env: Header=ENV_VAR), so it is
never on a command line, and the reports mask it wherever the server repeats it.
Keep the whole header value in the secret — Bearer <token> — and pass it as it is. Written as
Bearer ${{ secrets.TOKEN }}, a missing secret would leave the variable at Bearer , which looks set.
When a variable named in headers-from-env is empty or unset, the action stops with an error naming
the variable before it starts or contacts the server.
Fork pull requests
Section titled “Fork pull requests”A pull request from a fork gets no secrets, and neither does one opened by Dependabot: its runs get
Dependabot secrets only, not Actions secrets
(GitHub docs).
That is why the HTTP workflow above has two jobs: the one with the secret runs only for branches of
this repository, and the other tests a server started without credentials. On a fork the comment is
skipped, because the token is read-only; the annotations, the job summary, the artifacts and the gate
still work. The action never uses pull_request_target.
What each exit code does
Section titled “What each exit code does”| Exit | Meaning | Default | fail-on: never |
|---|---|---|---|
0 | Passed, and everything was measured. | green | green |
1 | A finding or a check failed (the annotations and the report name it). A server that started and exited before it answered initialize is a failed check: the log has the last lines of its stderr, secrets masked. | red | green, with a notice naming the command |
2 | The server could not be tested: its command was not found or could not be started, it could not be reached, refused the credentials, or a flag is wrong. The log has the last lines of its stderr, secrets masked. | red | green, with a warning; measured=false |
3 | Nothing was measured: no tools listed, nothing to compare, a check that could not run. | red | green, with a warning; measured=false |
A wrong input — both server-command and server-url, an unknown step, diff without
snapshot-file, an empty headers-from-env variable — stops the action with an error before anything
starts, whatever fail-on says.
Make the gate advisory
Section titled “Make the gate advisory”fail-on: never turns the whole action into a report: nothing fails the check, diff included.
- uses: plune-ai/eval-action/mcp@v1 with: server-command: node dist/server.js fail-on: never“Nothing measured” stays visible all the same. A server that did not start (exit 2) or listed
nothing (exit 3) gives a warning on the run, not a notice, and the measured output is false
for a later step to read. A green check from a server that was never tested would look like a
healthy one, so the action does not let the two look alike.
For an advisory lint with a blocking diff, use the action twice: one step with steps: lint and
fail-on: never, one with steps: diff, each with its own artifact-name, and comment: 'false' on
one of them.
What the pull request gets
Section titled “What the pull request gets”Each finding is an annotation on the run. This is what plune mcp lint --format github prints for
the sample server, the lines the runner turns into
annotations:
$ npx -y @plune-ai/cli@0.39.9 mcp lint --format github -- node server.mjs::warning title=desc-missing get_note::get_note: The tool has no description.%0Aexpected: a description%0Aactual: none%0Afix: Describe what the tool does, when to use it and when not to: the model chooses tools by name and description alone. The specification does not require a description; this is advice on quality.%0Aspec: https://modelcontextprotocol.io/specification/2025-11-25/schema#tool-description::warning title=list-no-paging list_notes::list_notes: "list_notes" returns a list by its name, but cannot be asked for less.%0Aexpected: one of: limit, cursor, page, offset, pagesize, perpage, per_page, page_size, pagetoken, page_token%0Aactual: parameters: none%0Afix: Let the caller bound the answer: add a limit and a cursor (or page) parameter.::warning title=param-enum-in-text set_status.status::set_status.status: The parameter description lists a closed set of values, but the schema does not.%0Aexpected: an enum%0Aactual: a free string described as "The new status: one of open, done, archived."%0Afix: List the allowed values in "enum" rather than in the description, so the client can check them.plune mcp lint: 21 passed, 0 failed, 0 could not run, 7 advisory or not applicable; exit 0.$ echo $?0The comment is the Markdown each command writes. For the same server, the lint part:
$ npx -y @plune-ai/cli@0.39.9 mcp lint --format markdown -- node server.mjs<!-- plune-mcp-lint -->### MCP lint
**Exit 0** — 21 passed, 0 failed, 0 could not run, 7 advisory or not applicable
| State | Level | Check | Target | Expected | Actual | Fix || --- | --- | --- | --- | --- | --- | --- || failed | warn | desc-missing | get_note | a description | none | Describe what the tool does, when to use it and when not to: the model chooses tools by name and description alone. The specification does not require a description; this is advice on quality. || failed | warn | list-no-paging | list_notes | one of: limit, cursor, page, offset, pagesize, perpage, per_page, page_size, pagetoken, page_token | parameters: none | Let the caller bound the answer: add a limit and a cursor (or page) parameter. || failed | warn | param-enum-in-text | set_status.status | an enum | a free string described as "The new status: one of open, done, archived." | List the allowed values in "enum" rather than in the description, so the client can check them. |$ echo $?0Server text in a cell cannot add HTML, a mention, a link or an image to the comment: those characters are written as character references.
Inputs and outputs
Section titled “Inputs and outputs”| Input | Default | What it does |
|---|---|---|
steps | lint,check | Commands to run, comma-separated: lint, check, diff. They run in the order check, lint, diff. |
server-command | Command that starts a stdio server, split on spaces: no quotes, no shell. Not with server-url. | |
server-url | URL of a Streamable HTTP server started by an earlier step. Not with server-command. | |
wait | 30s | How long check keeps trying to reach server-url. |
headers-from-env | Header=ENV_VAR, comma-separated; each value is read from that variable. Only with server-url. | |
fail-on | error | Lint threshold: error, warn, info or never. never makes the whole action advisory. |
ignore-file | The lint exceptions file. | |
snapshot-file | The committed snapshot diff compares with. Required when steps has diff. | |
diff-fail-on | breaking | Diff threshold: breaking, any or none. |
smoke | false | true adds --smoke to check. |
comment | true | Post and update one PR comment. On a second use of the action in one PR, set 'false'. |
debug-artifact | false | Also upload the frame journal of each command. It can hold what the tools returned. |
artifact-name | plune-mcp-reports | Name of the reports artifact; give each use its own. |
working-directory | . | Where server-command, ignore-file and snapshot-file are found. |
plune-version | 0.39.9 | Version of @plune-ai/cli to install from npm. |
Outputs: lint-exit, check-exit, diff-exit (empty when the command did not run), measured
(true when no command that ran exited 2 or 3), summary and report-dir. Only
contents: read is needed; add pull-requests: write for the comment.