mcp_server_precheck benchmark

Zero false positives on our re-vet legitimate sets

Q: Is “zero false positives” forever?

No. It means zero FPs on the legitimate sets exercised across the t1892–t1938 re-vet rounds of Version c62b814d, after mid-round FPs were fixed. New rules can introduce new FPs; we intend the suite to become a CI gate as the public corpus is assembled.

Q: Does a clean grade mean the server is safe?

No. It means our static checks did not fire. Runtime behavior, auth posture, paraphrased instructions, and deferred languages can still be wrong. Treat the grade as a pre-trust screen, not a clearance.

Fairquery · Published November 3, 2026 · Updated November 3, 2026

A free, no-install pre-check for MCP servers: static rules that flag poisoning and related risks. On the legitimate tool sets in our multi-round Claude re-vet (t1892–t1938), after mid-round fixes, false positives hit zero—not a forever claim for every everyday tool.

Try mcp_server_precheck on the hosted MCP server Privacy Notice

claude mcp add --transport http fairquery https://mcp.getfairquery.com/mcp

Short answer: Zero false positives on every legitimate set we ran across the multi-round Claude re-vet of free mcp_server_precheck on fairquery-mcp Version c62b814d (P2.5, Claude PASS t1938u, November 3, 2026). Earlier rounds did find false positives; those were fixed before the PASS. The re-vet used a multi-round regression suite (~60 cases)—an estimate from that review, not a counted public corpus with a published attack/legitimate split.

The scanner is static and deterministic. Semantic analysis is not run (no LLM/classifier judge on the free tier). The remaining known miss is paraphrased attacks (PARAPHRASE_GAP). Some languages (RU/ZH/JA) are deferred. Auth posture is out of scope.

Try it free, no account: connect to https://mcp.getfairquery.com/mcp and call mcp_server_precheck.

What we measured

We re-vetted the free tip on fairquery-mcp Version c62b814d through multiple Claude rounds (t1892 through t1938). Legitimate sets in those notes include a 10-tool clean set, a 9-tool set (including non-English email/upload and invoice→billing), and benign “scary” tools (e.g. password-manager paths, ~/.aws/credentials, “Important:”, silent-retry wording). False positives that appeared mid-stream were fixed (including chaining/shadowing and PNG/base64 noise, and a send_mail≈send_email typosquat false alarm) before the final PASS recorded zero FPs.

We do not publish a detection rate or an exact attack-versus-legitimate count. Claude described the suite as roughly 60 cases across rounds; the public corpus is not yet tagged attack/legit. Internal regression assets that are counted: 160 precheck unit tests in the tip tree (full Worker vitest 207/207), plus live smokes such as t1935u 35/35 on https://mcp.getfairquery.com/mcp.

Categories covered (verified in re-vet)

Category	What the checks cover
Duplicate names	Same tool name more than once → `DUPLICATE_TOOL_NAME` (high)
Hidden / encoded instructions	Zero-width, Unicode tags, bidi, HTML comments; tag decode-and-rescan; base64 decode-and-rescan → `ENCODED_INSTRUCTION` / `HIDDEN_INSTRUCTION`
Concealment	Phrases that tell the model not to mention something to the user
Prompt injection / poisoning	Keyword-style instruction overrides; multilingual ES/FR/DE/PT among them
Shadowing	Same-list and cross-server tool shadowing (cross-server treated critical when warranted)
Exfil destinations	BCC, paste sites, email/upload patterns (including non-English stems); medium prose exfil in prompts/resources/SKILL when send-verbs meet fixed destinations
Install / pipe-to-shell	`INSTALL_PIPE_TO_SHELL` on MCPB install commands; SKILL.md curl\|bash → critical
Overbroad permission	Dangerous capability shapes vs benign broad tools
Typosquat	Look-alike and letter-swap names; synonym pairs (e.g. send_mail / send_email) kept clean after the FP fix
Toxic flow	Multi-tool “read untrusted → send elsewhere” chains; evidence names the tools
Baselines / fingerprints	`TOOL_CHANGED` / `ADDED` / `REMOVED` vs a prior list; stable fingerprints across rounds
SARIF	Optional SARIF 2.1.0 with security-severity and OWASP tags; notes on unknown params
Live fetch	Free one-off public `serverUrl` fetch with rate limits; localhost/http refused; `shared_externally` disclosed
Prompts / resources / SKILL.md	Surfaces beyond tools
Credential hygiene	Hardcoded secrets / config PII in mcpServers env and related fields (`HARDCODED_SECRET`, `CONFIG_PII`); `MCP_SERVERS_NOT_SCANNED` when a config can’t be walked
Source attribution	Findings tagged `remote:` / `pasted:`; `grades_by_source`; `INPUT_MIXED_SOURCES` refuses mixing serverUrl with pasted tool lists in one call
npm data-flow honesty	Accurate `package_name_to_npm_registry` disclosure when that path is used

How to read the result

Grade A–F is hedged: static checks only—“semantic analysis not run.” It is a decision aid, not a certification.
PARAPHRASE_GAP: paraphrased attacks (for example, quietly asking to append API keys without classic injection phrases) can miss.
shared_externally: states what left the request (tool text vs optional npm registry vs OSV Actor).
source / grades_by_source: remote URL findings are not blended with pasted content in one call.
Findings include evidence you can replay for the same input (deterministic rules).

Methodology (honest limits)

We do	We don’t
Static, deterministic pattern and structure checks; same input → same result	Run a semantic LLM/classifier judge on the free tier
Scan tools, optional prompts/resources/SKILL.md, credential hygiene on pasted configs, one-off public `serverUrl`	Claim we catch paraphrased or novel wording outside our rules
Disclose optional external shares (npm name lookup, OSV Actor)	Walk mcpServers nested at arbitrary depth (we emit a note instead)
Refuse private/localhost-style `serverUrl` targets where applicable	Cover auth posture, RU/ZH/JA multilingual stems (ASCII word-boundary limits; deferred), or replace your own review

Known residual (not a false positive): benign lines like “send feedback to support@vendor” in resource or SKILL prose can still raise a medium prose-exfil finding by design.

Free vs paid

Per our product split (Claude notes, t1932u):

Free today: deterministic engine + credential hygiene + one-off public serverUrl (rate-limited) + this benchmark page.

Paid (planned / not built on this tip): semantic classifier; authenticated, bulk, or scheduled fetch; change monitoring; custom rules; deeper OSV.

Try it

Add the hosted server (Claude Code example):

claude mcp add --transport http fairquery https://mcp.getfairquery.com/mcp

Call mcp_server_precheck with a tools list, or a public serverUrl alone—not mixed with pasted tools in the same call.
Read meta.grade, findings, PARAPHRASE_GAP notes, and shared_externally before you pin trust on a new MCP server.

Full connect steps: Hosted MCP server. Privacy: text you send to hosted free tools is processed on our Workers; see the Privacy Notice.

FAQ

Is “zero false positives” forever?

No. It means zero FPs on the legitimate sets exercised across the t1892–t1938 re-vet rounds of Version c62b814d, after mid-round FPs were fixed. New rules can introduce new FPs; we intend the suite to become a CI gate as the public corpus is assembled.

Do you publish a detection percentage?

No. We do not invent or advertise a recall/%. The re-vet described a multi-round suite of ~60 cases; attack vs legitimate tagging for a public corpus is still to be done.

Does a clean grade mean the server is safe?

No. It means our static checks did not fire. Runtime behavior, auth posture, paraphrased instructions, and deferred languages can still be wrong. Treat the grade as a pre-trust screen, not a clearance.

Do you send my tool descriptions to a third-party AI?

Not for the free static scan. Optional paths that contact npm or run the OSV Actor are called out in shared_externally. Hosted processing still runs on Fairquery’s infrastructure—see Privacy.

Fairquery LLC. Corrections: info@getfairquery.com. Benchmark tied to fairquery-mcp Version c62b814d (checked 2026-11-03; inventory BENCHMARK-INVENTORY-t1939u). Claude, ChatGPT, Cursor, npm, and MCP are trademarks of their owners. Fairquery is not affiliated with, endorsed by, or sponsored by them.