I Built a Deterministic MCP Eval Engine, Then Deleted Half of It

I spent a few weeks building an MCP eval framework where no model touched the pass/fail decision. Then I deleted the part I was proudest of.

This is the writeup of why. It is mostly about a mistake, because the mistake is the useful part.

If you are building anything that evaluates MCP servers you did not write, you will hit the same wall. It is not obvious until you are past it.

The design that looked obviously right

The premise was sound and I would still defend it in isolation: a model fumbling an argument produces an error indistinguishable from a server defect. So keep models away from the verdict.

That instinct is not unusual. Most eval frameworks built on the Model Context Protocol start here, and the research benchmarks like MCP-Bench pair deterministic rule checks with judging for the same reason.

The architecture followed directly. A writer model emits data — concrete calls plus the checks to run against them. Code does the calling and the deciding.

{
  "kind": "direct",
  "tool": "list_issues",
  "args": [{ "name": "status", "json": "\"open\"", "source": "literal" }],
  "checks": [
    { "type": "succeeds" },
    { "type": "output_schema" },
    { "type": "max_items", "limit": 50 }
  ]
}

Nine check types, every one machine-decidable:

Plus is_error, max_items and latency_under. Schema validation ran through Ajv in non-strict mode, because MCP servers ship hand-written schemas full of harmless non-standard keywords.

A check could also come back skipped. Asserting output-schema conformance against a tool that declares no outputSchema proves nothing, so that is neither a pass nor a failure.

On servers I knew, this worked beautifully. Every failure was a real finding, with the request and response sitting right there as evidence.

Where it broke

Then I pointed it at servers I had never seen.

The planner has your tool list, your descriptions and your schemas. What it does not have is any idea what data actually exists inside your system.

So it has to guess arguments. And every defect I hit lived in that guessing.

Cross-entity ID matches

The planner sees list_teams returns objects with an id, and get_project takes a projectId. It is a plausible leap. It is also wrong, and the resulting 404 looks exactly like a broken lookup.

Calls that failed for unrelated reasons

A planner invents { "region": "us-west-3" }. Your server supports four regions and that is not one of them. You get a clean -32602.

Is that your bug or mine? From the outside, the two are identical. That question has no answer from the response alone, and a report full of unanswerable questions is worthless.

The empty-arguments disaster

This one was my fault and I will come back to it in the landmines section. An entire suite of calls went out with {} as arguments, and every single one "failed".

The harvest pass, and the rule it bought

The first fix was to stop guessing. Before planning anything, collect real values from the server.

The harvest pass calls read-only tools in a deliberate order — list and search shapes first, fewest required parameters first — and only when every required parameter can already be satisfied. Nothing is invented at this stage.

It walks the responses for ID-shaped fields and keeps where each came from:

{
  "key": "issueId",
  "value": "PROJ-4821",
  "from": "list_issues({ status: 'open' })",
  "path": "items[0].id"
}

That provenance is the whole point. It gives you a rule that makes failures mean something:

Every argument carries a source: harvested, synthesized or literal. The report reads completely differently once that distinction exists.

Untestable is not a softer failure. It is an admission that the eval could not establish anything, which is far more honest than a red row a developer will correctly ignore.

What replaced it

Harvesting helped. It did not fix the underlying problem: the planner was still choosing arguments for a system whose semantics it cannot see.

So I stopped having it choose. The plan became agentic-only.

Instead of scripting get_issue("PROJ-4821") and checking the result, the planner writes a task:

Task:        "Which open issue in the billing project has been
              waiting longest, and who is it assigned to?"
Budget:      4 tool calls
Expectation: names a specific issue and an assignee

A driver model gets your real tools and works it out. It picks arguments with your actual schema in front of it, and adapts when a call fails — exactly like a real client.

The agent loop runs on the Vercel AI SDK, with plan schemas defined in Zod and tools loaded straight from the server's own tools/list response.

That generalises to servers nobody anticipated, which scripted calls never did.

The tradeoffs are real and I will not pretend otherwise. Results are non-deterministic, so you reproduce before acting. And coverage is whatever the agent exercises, not a guaranteed sweep of every tool.

I took that deal because a narrow honest signal beats a broad meaningless one.

Keeping the model out of the verdict

Going agentic did not mean giving up on determinism where it still applies. The outcome is derived in code, from the transcript.

no final answer            -> failed_answer
answered, over the budget  -> too_many_calls
answered, within budget    -> pass (subject to the judge)
harness or transport error -> untestable

Only the last step consults a model, and only about whether the answer is right. Never about whether a call succeeded.

One detail I got wrong first time: the agent loop has to be allowed to exceed its budget.

My first version capped the loop at exactly the budget. Every over-budget run then looked like a failure to answer — a different and much less useful finding.

Now the ceiling sits above the budget, so going over is observable rather than truncated. You learn the agent needed seven calls, which is the signal you actually wanted.

The budget is also fixed at generation time, not per run. If the budget moved between models, a cross-model comparison would not be a fair test.

The judge fails open

The judge answers exactly one question: given the task, the expectation and the answer, is the answer correct?

It never sees whether a call succeeded. It cannot turn a pass into a failure on any ground other than wrong content.

The rule that matters most is what happens when it breaks:

// A missing verdict must not silently become a failure — an
// unjudged answer is treated as correct, since the deterministic
// layer already passed it.

If the judge errors out, every item stays passing. Losing your grader should never invent defects in someone's server.

This feels wrong the first time you write it. Surely an unjudged item is unknown, not passing? But consider the alternative: a rate limit on your judge provider turns into a page of red rows and someone spends an afternoon chasing bugs that do not exist.

Judging is also batched — one call per slice rather than per case. Per-case judging tripled the recurring cost of every run for no extra signal.

Fail-closed tool safety

This constraint shapes more of the engine than anything else, and it is worth stating plainly.

A functional eval calls tools with arguments designed to succeed. That is the entire point of it.

Which means pointing one at an unannotated delete_record would really delete a record. Not a mock. Not a dry run. Someone's actual data.

So classification is fail-closed. An unannotated tool is treated as destructive and excluded from a default run.

But unknown and destructive must not read the same in the UI, even though both get excluded:

Since most MCP servers in the wild declare no annotations at all, collapsing these two would leave the common case with an empty selection and no explanation of what to do about it.

One more rule I had to enforce against myself: a tool's name has no authority over whether it mutates. Name patterns like list_ or search_ only rank which read-only tools to harvest from first. They never upgrade a tool's safety classification.

Three structured-output landmines

These cost me real days and I have not seen them written down anywhere, so here they are.

1. Anthropic rejects numeric range keywords on integers

Write z.number().int().positive() in a schema for structured output and the request fails outright, before the model ever runs.

The validator rejects exclusiveMinimum, then minimum, and by extension maximum.

The fix is better design anyway: enforce bounds after generation, in code. A schema constraint makes the provider reject the whole response. Code can clamp one bad field and keep an otherwise good plan you already paid a frontier model to write.

2. OpenAI requires every property in required

This one only surfaced when I changed the writer model, which made it maddening to track down.

'required' is required to be supplied and to be an array
including every key in properties. Missing 'uncovered'.

A Zod .default() makes a field optional, and OpenAI and Azure structured outputs reject any object whose required array does not list every property. Anthropic accepted it happily.

No .default() and no .optional() anywhere in a structured-output schema. Handle the empty case in code.

3. Open records silently produce empty objects

This is the one that produced that whole suite of empty-argument calls.

z.record(z.unknown()) compiles to an object schema with no declared properties. Structured output cannot invent keys for such a schema.

The model has no way to express {"query": "Workers KV"}, so it returns {}. No error. No warning. Just empty arguments across every case.

The fix is a list of explicit entries instead of an open map:

args: [
  { name: "query", json: "\"Workers KV\"", source: "literal" }
]

The json field carries the JSON encoding of the value, so strings, numbers, booleans, objects and arrays all travel through one string field unambiguously. Fully expressible, no guessing.

If you are building anything with structured outputs, that pattern is worth stealing on its own.

What I would tell you to copy

Five things generalise beyond MCP.

1. Provenance decides meaning. The same 404 is a confirmed defect or proof of nothing, depending entirely on where the input came from. Track it.

2. "Untestable" is a real outcome. Any evaluator that only has pass and fail will report noise, and developers will learn to ignore the whole report.

3. Let models produce data, not decisions. Even in the agentic design, outcome derivation stayed in code. The one genuinely subjective question is scoped to a judge that can only affect that question.

4. Fail open on graders, fail closed on side effects. A broken judge must not invent bugs. An unannotated tool must not get called.

5. Make the thing you are measuring observable. Capping the loop at the budget hid the exact signal I was trying to capture.

And the meta-lesson: a design can be internally correct and still measure the wrong thing. Deterministic scripted evals were rigorous. They were rigorously grading my planner's guesses.

The check evaluators are still in the codebase. They work, they are well tested, and nothing calls them any more. That felt bad for about a day.

Frequently asked questions

Where it landed

The engine that shipped is smaller than the one I designed and it tells you fewer things. Every one of those things is true, which the original could not claim.

If you are building an evaluator for systems you did not write: track provenance, give yourself an untestable outcome, keep models away from verdicts, and be suspicious of any design where you cannot tell your bugs from theirs.

Related: what an MCP eval is · why 97% of tool descriptions are broken · what the Model Context Protocol is · building on the 2026-07-28 spec · every error code change in the 2026 spec