A passing connection test proves your server speaks the protocol. It does not prove that a model handed your tools can work out which one to call, or that the answer it comes back with is right. An eval is a real task — “what is the total for order 1041?” — given to a model that has nothing but your tool names and descriptions to go on.
The most valuable thing it finds is the failure no schema check can see: every tool call succeeded, the arguments validated, and the answer was still wrong. Below is a finished run on a made-up orders server, replayed exactly as the product renders it.
Detects your tools, writes real tasks against them, then hands each task to a model that has to complete it using your server. Tells you whether your server is actually usable — and which models can use it.
Point it at any MCP server you can reach. It detects the tools, harvests real values from the read-only ones so the tasks are grounded in your actual data, writes the evals, then runs them. Tools that write are left out unless you opt in.
Run evals on your server