# I Built a Deterministic MCP Eval Engine, Then Deleted Half of It

> Scripted MCP evals looked obviously correct: no model near the verdict, every check machine-decidable. Then I ran them against servers I had not seen, and almost every failure turned out to be my own planner guessing arguments. Here is what broke, what replaced it, and the structured-output landmines I hit on the way.

**Source:** https://mcpplaygroundonline.com/blog/mcp-eval-engine-design  
**Author:** Nikhil Tiwari  
**Published:** 2026-08-15  
**Updated:** 2026-08-15  
**Category:** Development  
**Reading time:** 14 min read

---

TL;DR

-   **I built scripted MCP evals with nine deterministic check types and no model in the verdict.** It was the right design for servers I already understood.
-   **It did not generalise.** A planner guessing arguments produces a -32602 indistinguishable from a real server defect.
-   **Harvesting real values first is what makes a failure mean something** — and it created the failed-vs-untestable rule.
-   **Agentic evals replaced it**, because an agent picks arguments with the real schema in front of it and adapts when a call fails.
-   **The model still never decides an outcome.** Code derives pass/fail from the transcript; a judge only grades answer correctness, and fails open.
-   **Three structured-output landmines cost me days**: Anthropic rejecting numeric bounds, OpenAI requiring every key in `required`, and open records silently producing empty arguments.

Table of contents

-   [The design that looked obviously right](#v1)
-   [Where it broke](#broke)
-   [The harvest pass, and the rule it bought](#harvest)
-   [What replaced it](#agentic)
-   [Keeping the model out of the verdict](#code)
-   [The judge fails open](#judge)
-   [Fail-closed tool safety](#safety)
-   [Three structured-output landmines](#landmines)
-   [What I would tell you to copy](#lessons)

I spent a few weeks building an **MCP eval framework** where no model touched the pass/fail decision. Then I deleted the part I was proudest of.

This is the writeup of why. It is mostly about a mistake, because the mistake is the useful part.

If you are building anything that evaluates MCP servers you did not write, you will hit the same wall. It is not obvious until you are past it.

## The design that looked obviously right

The premise was sound and I would still defend it in isolation: **a model fumbling an argument produces an error indistinguishable from a server defect.** So keep models away from the verdict.

That instinct is not unusual. Most eval frameworks built on the [Model Context Protocol](https://modelcontextprotocol.io/) start here, and the research benchmarks like [MCP-Bench](https://openreview.net/forum?id=fe8mzHwMxN) pair deterministic rule checks with judging for the same reason.

The architecture followed directly. A writer model emits _data_ — concrete calls plus the checks to run against them. Code does the calling and the deciding.

```
{
  "kind": "direct",
  "tool": "list_issues",
  "args": [{ "name": "status", "json": "\"open\"", "source": "literal" }],
  "checks": [
    { "type": "succeeds" },
    { "type": "output_schema" },
    { "type": "max_items", "limit": 50 }
  ]
}
```

Nine check types, every one machine-decidable:

`succeeds`  
no error, no isError

`output_schema`  
validates against declared schema

`error_code`  
expected JSON-RPC code

`pagination_terminates`  
cursor loop ends, no dupes

`idempotent`  
same call, same result

`no_mutation`  
witness tool proves nothing changed

Plus `is_error`, `max_items` and `latency_under`. Schema validation ran through [Ajv](https://ajv.js.org/) in non-strict mode, because MCP servers ship hand-written schemas full of harmless non-standard keywords.

A check could also come back **skipped**. Asserting output-schema conformance against a tool that declares no `outputSchema` proves nothing, so that is neither a pass nor a failure.

On servers I knew, this worked beautifully. Every failure was a real finding, with the request and response sitting right there as evidence.

## Where it broke

Then I pointed it at servers I had never seen.

The planner has your tool list, your descriptions and your schemas. What it does not have is **any idea what data actually exists inside your system.**

So it has to guess arguments. And every defect I hit lived in that guessing.

### Cross-entity ID matches

The planner sees `list_teams` returns objects with an `id`, and `get_project` takes a `projectId`. It is a plausible leap. It is also wrong, and the resulting 404 looks exactly like a broken lookup.

### Calls that failed for unrelated reasons

A planner invents `{ "region": "us-west-3" }`. Your server supports four regions and that is not one of them. You get a clean -32602.

**Is that your bug or mine?** From the outside, the two are identical. That question has no answer from the response alone, and a report full of unanswerable questions is worthless.

### The empty-arguments disaster

This one was my fault and I will come back to it in the landmines section. An entire suite of calls went out with `{}` as arguments, and every single one "failed".

The realisation

I had built something that measured **my planner's ability to guess arguments** and reported the result as a verdict on someone else's server. The determinism was real. It was just deterministically measuring the wrong thing.

## The harvest pass, and the rule it bought

The first fix was to stop guessing. Before planning anything, **collect real values from the server.**

The harvest pass calls read-only tools in a deliberate order — list and search shapes first, fewest required parameters first — and only when every required parameter can already be satisfied. Nothing is invented at this stage.

It walks the responses for ID-shaped fields and keeps where each came from:

```
{
  "key": "issueId",
  "value": "PROJ-4821",
  "from": "list_issues({ status: 'open' })",
  "path": "items[0].id"
}
```

That provenance is the whole point. It gives you a rule that makes failures mean something:

**If `get_issue("PROJ-4821")` 404s on an ID that `list_issues` returned a second earlier, that is a confirmed defect** — the ID demonstrably exists and both calls are the evidence. If the ID was invented, a 404 proves nothing. So anything built from synthesized values is reported **untestable**, never **failed**.

Every argument carries a source: `harvested`, `synthesized` or `literal`. The report reads completely differently once that distinction exists.

**Untestable is not a softer failure.** It is an admission that the eval could not establish anything, which is far more honest than a red row a developer will correctly ignore.

## What replaced it

Harvesting helped. It did not fix the underlying problem: **the planner was still choosing arguments for a system whose semantics it cannot see.**

So I stopped having it choose. The plan became agentic-only.

Instead of scripting `get_issue("PROJ-4821")` and checking the result, the planner writes a task:

```
Task:        "Which open issue in the billing project has been
              waiting longest, and who is it assigned to?"
Budget:      4 tool calls
Expectation: names a specific issue and an assignee
```

A driver model gets your real tools and works it out. **It picks arguments with your actual schema in front of it, and adapts when a call fails** — exactly like a real client.

The agent loop runs on the [Vercel AI SDK](https://ai-sdk.dev/), with plan schemas defined in [Zod](https://zod.dev/) and tools loaded straight from the server's own `tools/list` response.

That generalises to servers nobody anticipated, which scripted calls never did.

The tradeoffs are real and I will not pretend otherwise. Results are non-deterministic, so you reproduce before acting. And coverage is whatever the agent exercises, not a guaranteed sweep of every tool.

I took that deal because **a narrow honest signal beats a broad meaningless one.**

See what this produces

A finished eval run, replayed step by step — tools classified, tasks written, transcripts and verdicts. No sign-up.

[Walk through MCP Evals →](https://mcpplaygroundonline.com/mcp-evals)

## Keeping the model out of the verdict

Going agentic did not mean giving up on determinism where it still applies. **The outcome is derived in code, from the transcript.**

```
no final answer            -> failed_answer
answered, over the budget  -> too_many_calls
answered, within budget    -> pass (subject to the judge)
harness or transport error -> untestable
```

Only the last step consults a model, and only about whether the _answer_ is right. Never about whether a call succeeded.

One detail I got wrong first time: **the agent loop has to be allowed to exceed its budget.**

My first version capped the loop at exactly the budget. Every over-budget run then looked like a failure to answer — a different and much less useful finding.

Now the ceiling sits above the budget, so going over is _observable_ rather than truncated. You learn the agent needed seven calls, which is the signal you actually wanted.

The budget is also fixed at generation time, not per run. **If the budget moved between models, a cross-model comparison would not be a fair test.**

## The judge fails open

The judge answers exactly one question: given the task, the expectation and the answer, is the answer correct?

It never sees whether a call succeeded. It cannot turn a pass into a failure on any ground other than wrong content.

The rule that matters most is what happens when it breaks:

```
// A missing verdict must not silently become a failure — an
// unjudged answer is treated as correct, since the deterministic
// layer already passed it.
```

**If the judge errors out, every item stays passing.** Losing your grader should never invent defects in someone's server.

This feels wrong the first time you write it. Surely an unjudged item is unknown, not passing? But consider the alternative: a rate limit on your judge provider turns into a page of red rows and someone spends an afternoon chasing bugs that do not exist.

Judging is also batched — one call per slice rather than per case. Per-case judging tripled the recurring cost of every run for no extra signal.

## Fail-closed tool safety

This constraint shapes more of the engine than anything else, and it is worth stating plainly.

**A functional eval calls tools with arguments designed to succeed.** That is the entire point of it.

Which means pointing one at an unannotated `delete_record` would really delete a record. Not a mock. Not a dry run. Someone's actual data.

So classification is fail-closed. An unannotated tool is treated as destructive and excluded from a default run.

But **unknown and destructive must not read the same in the UI**, even though both get excluded:

Classification

What it means

**destructive**

The server told us it mutates. Opting in is a real risk.

**unknown**

The server told us nothing. It may well be read-only; we just cannot prove it.

Since most MCP servers in the wild declare no annotations at all, collapsing these two would leave the common case with an empty selection and no explanation of what to do about it.

One more rule I had to enforce against myself: **a tool's name has no authority over whether it mutates.** Name patterns like `list_` or `search_` only rank which read-only tools to harvest from first. They never upgrade a tool's safety classification.

## Three structured-output landmines

These cost me real days and I have not seen them written down anywhere, so here they are.

### 1\. Anthropic rejects numeric range keywords on integers

Write `z.number().int().positive()` in a schema for structured output and the request fails outright, before the model ever runs.

The validator rejects `exclusiveMinimum`, then `minimum`, and by extension `maximum`.

**The fix is better design anyway:** enforce bounds after generation, in code. A schema constraint makes the provider reject the whole response. Code can clamp one bad field and keep an otherwise good plan you already paid a frontier model to write.

### 2\. OpenAI requires every property in `required`

This one only surfaced when I changed the writer model, which made it maddening to track down.

```
'required' is required to be supplied and to be an array
including every key in properties. Missing 'uncovered'.
```

A Zod `.default()` makes a field optional, and OpenAI and Azure structured outputs reject any object whose `required` array does not list every property. Anthropic accepted it happily.

**No `.default()` and no `.optional()` anywhere in a structured-output schema.** Handle the empty case in code.

### 3\. Open records silently produce empty objects

This is the one that produced that whole suite of empty-argument calls.

`z.record(z.unknown())` compiles to an object schema with **no declared properties**. Structured output cannot invent keys for such a schema.

The model has no way to express `{"query": "Workers KV"}`, so it returns `{}`. No error. No warning. Just empty arguments across every case.

The fix is a list of explicit entries instead of an open map:

```
args: [
  { name: "query", json: "\"Workers KV\"", source: "literal" }
]
```

**The `json` field carries the JSON encoding of the value**, so strings, numbers, booleans, objects and arrays all travel through one string field unambiguously. Fully expressible, no guessing.

If you are building anything with structured outputs, that pattern is worth stealing on its own.

## What I would tell you to copy

Five things generalise beyond MCP.

**1\. Provenance decides meaning.** The same 404 is a confirmed defect or proof of nothing, depending entirely on where the input came from. Track it.

**2\. "Untestable" is a real outcome.** Any evaluator that only has pass and fail will report noise, and developers will learn to ignore the whole report.

**3\. Let models produce data, not decisions.** Even in the agentic design, outcome derivation stayed in code. The one genuinely subjective question is scoped to a judge that can only affect that question.

**4\. Fail open on graders, fail closed on side effects.** A broken judge must not invent bugs. An unannotated tool must not get called.

**5\. Make the thing you are measuring observable.** Capping the loop at the budget hid the exact signal I was trying to capture.

And the meta-lesson: **a design can be internally correct and still measure the wrong thing.** Deterministic scripted evals were rigorous. They were rigorously grading my planner's guesses.

The check evaluators are still in the codebase. They work, they are well tested, and nothing calls them any more. That felt bad for about a day.

## Frequently asked questions

**Why not keep both scripted and agentic evals?+**

Scripted evals work well when you wrote the server and know what data exists — that is a genuinely good use for them. They stop working when the planner has to guess arguments for a system it cannot see into, which is every server a hosted tool encounters. Keeping both would have meant shipping a mode that produces unanswerable failures.

**Does the driver model see the eval expectation?+**

No. The driver gets the task and the tools, nothing else. The expectation exists only for the judge to score against afterwards. Showing it to the driver would leak the answer and turn the eval into a formatting exercise.

**Why run each eval case on its own connection?+**

A legacy stateful server may serialise or misbehave on concurrent calls over a single session, and that would look like a defect in the server rather than in the scheduling. A handshake per case is a cheap price for isolation, and one dead connection then kills one case instead of the whole batch.

**How much tool description do you give the planner?+**

Up to 4,000 characters, deliberately generous. A good description is the richest source of test material available — Cloudflare's docs tool lists every product it covers, and its search tool ships literal usage examples. Those are exactly the realistic arguments you want, so truncating to a couple of hundred characters throws away the best input you have.

## Where it landed

The engine that shipped is smaller than the one I designed and it tells you fewer things. Every one of those things is true, which the original could not claim.

If you are building an evaluator for systems you did not write: **track provenance, give yourself an untestable outcome, keep models away from verdicts, and be suspicious of any design where you cannot tell your bugs from theirs.**

Point it at your own server

Connect any MCP server in the browser and see how a real model handles your tools.

[Test any MCP server free →](https://mcpplaygroundonline.com/mcp-test-server) [See MCP Evals →](https://mcpplaygroundonline.com/mcp-evals)

Related: [what an MCP eval is](/blog/what-is-an-mcp-eval) · [why 97% of tool descriptions are broken](/blog/mcp-tool-description-quality) · [what the Model Context Protocol is](/blog/what-is-model-context-protocol) · [building on the 2026-07-28 spec](/blog/build-mcp-server-2026-spec) · [every error code change in the 2026 spec](/blog/mcp-error-32602)

## Frequently asked questions

### Why did you replace deterministic scripted MCP evals with agentic ones?

Scripting concrete calls meant the planner had to guess arguments for a server whose semantics it cannot see. A guessed argument produces a JSON-RPC -32602 that is indistinguishable from a real server defect, so failures became unanswerable. An agent picks its own arguments with the real schema in front of it and adapts when a call fails, which generalises to servers nobody anticipated.

### What is a harvest pass in an MCP eval engine?

It collects real argument values from the server by calling read-only tools before any parameterised call is planned, recording which call produced each value and where it sat in the response. That provenance is what lets a failure on a harvested value be reported as a confirmed defect, while the same failure on an invented value is reported as untestable.

### Should an LLM judge fail open or fail closed?

Fail open. If the judge errors or returns no verdict for an item, that item should stay passing, because the deterministic layer already passed it. Failing closed means a rate limit on your judge provider turns into a page of red rows and someone chases bugs that do not exist.

### Why does Anthropic reject Zod numeric constraints in structured output?

The structured-output validator rejects numeric range keywords on integers — exclusiveMinimum, then minimum, and by extension maximum. Using .positive(), .min() or .max() makes the request fail before the model runs. Enforce bounds in code after generation instead, which also lets you clamp one bad field rather than losing the whole response.

### Why does z.record() produce empty arguments with structured outputs?

An open record compiles to an object schema with no declared properties, and structured output cannot invent keys for such a schema. The model has no way to express a key-value pair, so it silently returns an empty object. Use a list of explicit {name, json, source} entries instead, where json carries the JSON encoding of the value.

### Why must an eval agent be allowed to exceed its call budget?

Capping the loop at exactly the budget makes every over-budget run look like a failure to answer, which is a different and much less useful finding. Setting the hard ceiling above the budget makes going over observable, so you learn the agent needed seven calls instead of four.


---

_Canonical page: https://mcpplaygroundonline.com/blog/mcp-eval-engine-design — MCP Playground (mcpplaygroundonline.com), the free browser-based tool for testing MCP servers and building AI agents._