What Is an MCP Eval? Why Your Server Passes Every Test and Still Fails

You built an MCP server. The handshake works, every tool returns valid JSON, your integration tests are green. Then someone connects it to Claude and it is useless.

That gap has a name now. An MCP eval is what closes it.

The distinction is simple once you see it. Your test suite proves the protocol works. It says nothing about whether a model can use what you built.

Those are genuinely different questions, and only one of them is the reason your server exists.

The MCP specification has nothing to say about the second one, and it should not — conformance is not usability. The 2026-07-28 release tightened the protocol considerably and did not change this at all.

I have spent the last few months building an eval engine for arbitrary MCP servers. This post covers what an eval is, the four outcomes worth distinguishing, why the results are non-deterministic, and when you should bother.

What an MCP eval actually is

An MCP eval is a task, phrased the way a user would ask it, that a model has to complete using only your server's tools.

Not an assertion. Not a mocked conversation. A question, and a score for whether the model got there.

A test looks like this:

assert(callTool("list_issues", { status: "open" }).length > 0)

An eval looks like this:

Task:        "Which open issue in the billing project has been
              waiting longest, and who is it assigned to?"
Budget:      4 tool calls
Expectation: names a specific issue and an assignee

Nothing in the eval names a tool. Choosing the tool is the thing being tested.

The model gets your tool list, your descriptions and your schemas, and has to work out the rest. That is exactly what happens in production.

MCP eval vs testing vs inspecting

These three get used interchangeably and they measure different things.

The key structural point: when you write a test, you have already made the choice the model has to make.

You picked the tool. You picked the arguments. You skipped the only step that can fail in the way that matters.

All three are worth having. Testing catches protocol and correctness bugs, and it is faster and cheaper than an eval. Evals catch a class of defect testing structurally cannot reach.

The failure that has no red log line

Here is the scenario that made me build this.

A docs server exposes search and get_page. A user asks which regions a product supports.

The agent calls search("regions"). It gets back five ranked results. It calls get_page on the first one. That page mentions two regions in an example snippet.

The agent answers: "It supports us-east and eu-west."

Every call succeeded. Valid JSON, no errors, no timeouts. Your logs are clean and your dashboard is green.

The real answer was eleven regions, listed on a page the search ranked fourth.

Nothing in your stack is lying. The protocol worked perfectly. The server was just hard to use, and the model did what it could.

The four outcomes and what each one means

A binary pass/fail throws away most of the signal. Four outcomes are worth distinguishing.

Pass

The agent answered, stayed within its call budget, and the answer was correct. This is the only outcome that requires judging content, which means it is the only one a model decides.

Wrong answer

The agent produced an answer and it was not right. Either it could not find the information, or your tools gave it something misleading.

This is the outcome from the scenario above, and it is the most valuable one an eval produces.

Too many calls

The agent got there, but it took eight calls when the budget was four. This is the tool-description signal, and it is the one to watch over time.

The agent was guessing. It tried a tool, got something unhelpful, tried another. Every one of those calls cost you latency, tokens and — if your API is metered — money.

Call count degrades before pass rate does. A task that passed in two calls last month and passes in six today is a regression, even though the number in the pass column did not move.

I covered the root cause of this in why 97% of MCP tool descriptions are broken.

Untestable

Something outside your server broke. The connection dropped, the driver model rate-limited, the harness fell over.

This is not a failure and it must never be counted as one. It gets its own bucket for a reason I will come to.

Why "untestable" deserves its own outcome

This is the design decision I would defend hardest, and it took me a while to get right.

A failure only means something if you know the input was valid.

Say an eval calls get_issue("PROJ-4821") and gets a 404. Is that a defect in your server?

It depends entirely on where that ID came from.

• If list_issues returned it a second earlier, the ID demonstrably exists and your server cannot fetch it. That is a confirmed defect, with both calls as evidence.
• If a model invented it, a 404 proves nothing. The issue probably just does not exist.

Same error code, opposite meaning. The only thing that separates them is provenance.

This is why a serious eval engine harvests real values from your server before planning anything that needs them. Failures on harvested values are defects. Failures on invented values are untestable.

Collapsing those two into "fail" gives you a report full of noise that developers correctly learn to ignore.

Evals are not deterministic — and that is fine

Models write the tasks. A model drives the tools. A model grades the answer. Run the same suite twice and you can get different numbers.

People find this disqualifying. I think that reaction comes from expecting an eval to be a test, which it is not.

The research benchmarks live with the same constraint. MCP-Atlas grades 1,000 tasks across 36 real servers using a rubric-driven judge, and MCP-Bench pairs rule-based checks with LLM scoring for exactly this reason. Nobody has found a way to make this deterministic, because the thing being measured is not.

An eval is evidence about how an agent behaves against your server. It is not a pass/fail certificate, and any tool that presents it as one is overselling.

Two rules make non-determinism workable:

Reproduce before you act. A single failure is a lead, not a finding. Run it again. A defect that reproduces is real; one that does not is variance.

Read the transcript, not the score. The number tells you where to look. The transcript — which tools were called, with what arguments, what came back — tells you whether the verdict was fair.

There is a third rule I only arrived at by getting it wrong: the grader must fail open.

If the judge model errors out or returns nothing for an item, that item stays passing. A missing verdict is not evidence of a defect. Failing closed would let an unrelated API hiccup invent bugs in your server.

When you need evals, and when you do not

Evals cost model calls. They are slower and pricier than tests. Some servers do not need them.

That last one matters more than it looks. Fix your annotations before you run evals, or safe tooling will skip most of your tools. The spec repository documents all three hints, and they take minutes to add.

An eval calls tools with arguments designed to succeed. An unannotated delete_record would really delete a record, so anything responsible has to treat unannotated tools as destructive.

How to run your first one

Start smaller than you think.

Pick your three most-used tools. Write three tasks a real user would ask, in their words, with no tool names in them.

Connect a model to your server and give it each task. Any of the models available in the browser will do for a first pass — you do not need a harness to learn something.

Watch three things: did it answer, was the answer right, and how many calls did it take.

You will almost always find one of two things. A tool the model never reaches for, or a pair it keeps confusing. Both are description problems, and both are fixable in an afternoon.

Once that loop is useful, make it repeatable so a description change gets checked instead of hoped about.

Frequently asked questions

The short version

A test proves your server responds. An eval proves it can be used.

The gap between those is where the expensive failures live — wrong answers assembled from successful calls, and agents burning four extra calls because two of your tools read alike.

Start with three tasks and three tools. Watch the call count as closely as the pass rate. Reproduce before you act.

Related: what the Model Context Protocol is · how MCP agents drive tools · which model is best at MCP tool calling · testing an AI agent without burning tokens