97% of MCP Tool Descriptions Are Broken — How to Test Yours

Your MCP server works. Every tool returns valid JSON, the schemas validate, the handshake succeeds. And the agent still picks the wrong tool.

When that happens, the instinct is to blame the model. The model is rarely the problem. The problem is usually the one artefact you wrote by hand and never tested: the tool description.

A model does not see your code. It sees a name, a description, and a JSON Schema. That is the entire basis on which it decides whether to call your tool.

The MCP specification is deliberately quiet on description quality — it defines the wire format, not the prose. That leaves the highest-impact field in your server completely unguarded.

In February 2026 a team from Queen's University and Huawei published an empirical study of MCP tool description quality across the public ecosystem. The numbers are worse than most people assume, and the fix is more complicated than the headline suggests.

This post covers what the study found, the six defect classes it identified, why naively expanding your descriptions can make things worse, and how to get a real measurement instead of a guess.

What the research actually found

The paper is Model Context Protocol (MCP) Tool Descriptions Are Smelly! by Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams and Ahmed E. Hassan. First submitted 16 February 2026, revised 31 May.

They examined 856 tools spread across 103 MCP servers and assessed each description against a defect taxonomy borrowed from code-smell research.

The headline result:

Read the bottom three rows again. That is the part almost every summary of this paper leaves out, and it changes what you should do about it.

The six ways an MCP tool description fails

Glama built a Tool Definition Quality Score on top of this research, scoring each tool 1–5 across six dimensions. They map cleanly onto the defect classes.

1. Purpose clarity

The description does not say what the tool does. This is the 56% case — the single most common defect in the ecosystem.

A description like Panel runner is a noun phrase. It names a component in your architecture, not an action a user wants.

2. Usage guidelines

89% of descriptions never say when you should or should not use the tool. This is what causes tool confusion on servers with overlapping capabilities.

If you ship search_docs and get_page, and neither says which to reach for first, the model guesses. Sometimes it guesses wrong.

3. Behavioural transparency

Does the tool mutate anything? Is it idempotent? Does it hit a rate limit? A model that does not know a call is destructive will treat it like a read.

4. Parameter semantics

The tool description is fine, but the parameter descriptions are empty. So the model knows to call get_issue and has no idea what format issueId takes.

This is the defect that produces JSON-RPC -32602 invalid params errors at runtime.

5. Conciseness

Longer is not better. Every character of every description sits in the context window on every single turn, for every tool you expose.

A 60-tool server with 400-word descriptions burns real budget before the user has typed anything. I wrote about this in how MCP tools silently eat your context window.

6. Contextual completeness

The description assumes knowledge the model does not have. Internal entity names, project-specific jargon, an ID scheme documented only in your wiki.

The uncomfortable part: fixing descriptions has a cost

Here is where most write-ups of this study go wrong. They quote 97%, quote the success gain, and conclude: write longer descriptions.

The paper does not support that conclusion. Augmenting descriptions raised execution steps by 67.46% — the agent did more work to get to the answer.

And in 16.67% of cases the augmented description made performance worse. One time in six, the rewrite was a regression.

So the actionable version is not "expand everything". It is: say the right things, cut the rest, and measure whether it worked.

That last clause is the one people skip. A 1-in-6 regression rate means you cannot assume your rewrite helped.

Three real descriptions, rewritten

These patterns come from tools I have pointed evals at. The names are changed; the shapes are exactly what shows up in the wild.

Scenario 1: the noun phrase

{
  "name": "panel_runner",
  "description": "Panel runner."
}

What goes wrong: the model has no verb to match against the user's request. Asked to "survey our target customers", it never connects that to panel_runner.

{
  "name": "run_research_panel",
  "description": "Run a customer research panel against a target
audience and return per-question response breakdowns. Use for
new survey questions. To re-read a panel that already ran, use
get_panel_results instead."
}

Lead with the verb the user would use. Then name the sibling tool it gets confused with.

Scenario 2: the overlapping pair

A docs server ships search and fetch_page. Both descriptions are accurate. Neither mentions the other.

The observable symptom: the agent calls search, gets a list of URLs, calls fetch_page on the first one, does not find the answer, and calls search again with a slightly different query.

It gets there eventually. It burns four calls doing what should take two. On a metered API, that is your bill.

The fix is one clause in each description: "Returns ranked page URLs and snippets — call fetch_page on a result to read the full page."

Scenario 3: the undocumented ID format

{
  "name": "get_issue",
  "description": "Get an issue.",
  "inputSchema": {
    "type": "object",
    "properties": { "issueId": { "type": "string" } },
    "required": ["issueId"]
  }
}

The model has to invent an ID format. It tries "1234". Your server wants "PROJ-1234". You get a -32602 or an empty result.

Add one line to the parameter and the failure disappears: "issueId": { "type": "string", "description": "Issue key in PROJ-123 form, as returned by list_issues." }

That final clause — as returned by list_issues — tells the model where to get a valid value. It is the highest-leverage sentence you can write.

Why your test suite cannot see this

This is the core of the problem. A description defect is invisible to every form of testing that calls tools directly.

Run your server through an inspector and every tool returns 200. Run your integration tests and they pass. Validate against the 2026-07-28 spec and you are conformant.

None of that touches the question that matters: given these descriptions, does a model pick the right tool?

The reason is structural. When you call a tool yourself, you have already made the choice the model has to make. You skipped the only step that can fail.

The failure mode this produces is the nastiest one in MCP: every call succeeds and the final answer is still wrong. Nothing in your logs is red.

The annotations problem nobody talks about

The MCP spec lets a server declare readOnlyHint, idempotentHint and destructiveHint on each tool. These are part of how a model reasons about safety, and the 2026-07-28 spec release leaned further into annotations as a risk vocabulary.

Most servers in the wild declare none of them. That is a description defect with real consequences, and it is one you can fix in an afternoon.

It matters more than it looks. When I built the eval engine behind MCP Evals, missing annotations forced a hard design decision.

A functional eval calls tools with arguments designed to succeed. That is the whole point of it. So pointing one at an unannotated delete_record would really delete a record.

The only safe posture is fail-closed: an unannotated tool gets treated as destructive and excluded from a default run.

Note that "unknown" and "destructive" are not the same situation, even though both get excluded. An unannotated tool may well be read-only. You just cannot prove it, and guessing is not good enough when the call is designed to work.

How to actually measure description quality

Given a 1-in-6 regression rate, you need a measurement, not an opinion. There are three practical levels.

Level 1: read them against the six dimensions

Cheap and genuinely useful. Score each tool 1–5 on purpose clarity, usage guidelines, behavioural transparency, parameter semantics, conciseness and contextual completeness.

Anything scoring 1 or 2 on purpose clarity is a certain problem. Fix those first.

Level 2: make a model choose, once

Connect your server to a real model, ask a question a user would actually ask, and watch which tool it reaches for. This is the fastest way to find a confused pair.

You can do this in a browser with any of the models on MCP Playground without writing a harness.

Level 3: run it as an eval suite

The difference between level 2 and level 3 is repeatability. An eval is a task plus a scored outcome, run the same way every time your descriptions change.

This is what the academic benchmarks measure at ecosystem scale. MCP-Atlas runs 1,000 human-authored tasks across 36 real servers and 220 tools, with a public leaderboard. MCP-Bench exposes 250 tools across 28 servers to test cross-server orchestration.

Those tell you how good a model is. You want the same method pointed at your server.

The signal to watch is not just pass or fail. It is how many calls the model needed.

A task that passes in two calls and later passes in five did not stay the same. Call count is your description-quality metric, and it moves before pass rate does.

That is why the eval engine I built fixes the call budget at generation time. If the budget moved per run, the comparison would be meaningless.

The checklist

Frequently asked questions

Where this leaves you

The 97% number is real, peer-reviewed and worth taking seriously. The naive response to it is not.

Description quality is a measurable property of your server, it degrades quietly, and the only instrument that reads it is a model being made to choose. Call count tells you before pass rate does.

Score your tools against the six dimensions, declare your annotations, add the "as returned by" clause, then check that the rewrite actually helped.

New to the protocol? Start with what the Model Context Protocol actually is, then how to test an MCP server step by step.