MCP Observability: How to Trace Every Tool Call in Production

A tool call failed in production last week. The agent said "I wasn't able to retrieve that."

That was the entire signal. No stack trace, no timing, no tool name.

This is the MCP observability gap, and almost every team building agents hits it around month three. Your app has traces. Your MCP server has logs. Neither knows the other exists.

The worst part is not the outage. It is the failure that looks like success — a confident, wrong, expensive answer that nobody flags for a week.

Two things changed in 2026 that make this fixable. The 2026-07-28 MCP specification deprecated protocol-level Logging. And OpenTelemetry's GenAI semantic conventions grew a proper MCP section with defined span names, attributes and metrics.

So there is now one vendor-neutral vocabulary for agent, tool and MCP spans.

I'll walk through the exact field names, how trace context survives a stateless request, what to alert on, and the five mistakes that quietly produce useless traces.

If you are new to the protocol itself, start with my introduction to the Model Context Protocol first.

What Is MCP Observability, Exactly?

MCP observability is the ability to answer "what did the agent actually do" from telemetry alone.

That means three linked questions, not one.

Which tool did the model pick, and why that one? Model-side. Lives in your LLM spans.

What did the server do with that call? Server-side. Latency, downstream calls, errors.

Did the result actually help? Outcome-side. Did the agent retry, give up, or hallucinate around a bad result?

Traditional APM answers the middle question well and the other two not at all.

An MCP tool call is a distributed transaction that crosses a model, a client, a transport and your backend. Treat it like one.

The three signals map cleanly:

• Traces — the causal path of one request across client and server.
• Metrics — call rate, duration percentiles, error rate per tool.
• Logs — the detail you attach to a span, not a separate stream.

That last point is the one that changed this year.

The 2026-07-28 Spec Deprecated Logging — Here Is What Replaced It

The 2026-07-28 specification deprecated four things at once: Roots, Sampling, Logging, and the legacy HTTP+SSE transport.

Each gets a minimum twelve-month support window. These were the first deprecations to follow MCP's new formal feature lifecycle policy.

Logging is the one that matters here. MCP servers were inventing their own log formats and shipping them over the protocol. That never aggregated well.

The replacement is not another MCP feature. It is OpenTelemetry.

The spec also documented W3C Trace Context propagation with fixed keys — traceparent, tracestate and baggage. Distributed traces now correlate across SDKs and render as one span tree in your backend.

The rest of the release pushed the same direction. Stateless core, header-based routing via Mcp-Method and Mcp-Name, and cacheable list results carrying ttlMs and cacheScope.

All of that assumes you can see across many short-lived requests rather than one long session. Which is exactly what tracing gives you.

If you have not made that move yet, my guide to migrating to the 2026-07-28 stateless spec covers the breaking changes.

Why Your Agent Traces Break at the MCP Boundary

Here is the failure, concretely.

Your agent framework emits a span for the LLM call. Good. It emits a child span for "tool use". Still good.

Then the request crosses into your MCP server and a brand-new, parentless trace begins.

You now have two traces for one logical operation. Neither is searchable by the other's ID.

Ask "why did the checkout agent take 40 seconds" and you get an agent trace showing 40 seconds of "tool call" and a server trace showing 38 seconds of something. Joining them is manual archaeology.

The root cause is that JSON-RPC over HTTP has nowhere obvious to put trace headers. MCP is a protocol on top of a transport, and the transport headers are not always yours to control — gateways rewrite them, stdio has none at all.

This got worse with the stateless spec, not better. Without Mcp-Session-Id, you lost the one accidental correlation key many teams were quietly relying on.

Every request now travels alone, carrying its own protocol version, client identity and capabilities in _meta.

Which turns out to be the hint. _meta is also where the trace context goes.

The OpenTelemetry MCP Semantic Conventions, Field by Field

OpenTelemetry's GenAI Special Interest Group formed in April 2024 and has since expanded to cover agent orchestration, MCP tool calling, content capture and quality evaluation.

The MCP conventions live in the semantic-conventions-genai repository.

The headline rule: when instrumenting MCP calls, follow the MCP conventions rather than the generic RPC conventions. MCP spans record things RPC spans cannot — tool identity, protocol version, message exchanges inside streaming calls.

Span names and kinds

Both sides use the same name format: {mcp.method.name} {target}.

The target matches gen_ai.tool.name or gen_ai.prompt.name where one applies. So a tool call becomes tools/call get_invoice, not a generic POST /mcp.

• Client span — kind CLIENT
• Server span — kind SERVER

That naming alone fixes most dashboards. You can group by tool without parsing a request body.

The attributes that matter

Those opt-in attributes are opt-in for a reason. Tool arguments routinely carry customer records, tokens and internal IDs. Turn them on in staging, sample them hard in production, and scrub before export.

mcp.protocol.version deserves a callout. Once you support two spec revisions, it becomes the single most useful attribute you have — it tells you which clients are still on the old path.

The four metrics

• mcp.client.operation.duration
• mcp.server.operation.duration
• mcp.client.session.duration
• mcp.server.session.duration

Client and server duration are split on purpose. The gap between them is your transport, gateway and queueing cost — and it is usually where the surprise lives.

How Context Propagation Works Through params._meta

This is the mechanism that stitches the trace back together.

Instrumentations inject trace context into the MCP request's params._meta property bag using the configured OpenTelemetry propagators. The keys are unprefixed: traceparent, tracestate, baggage.

So a tools/call on the wire carries something like this:

{
  "jsonrpc": "2.0",
  "id": "42",
  "method": "tools/call",
  "params": {
    "name": "get_invoice",
    "arguments": { "invoice_id": "inv_123" },
    "_meta": {
      "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
    }
  }
}

On the receiving side the rule is explicit. Server instrumentation should use the context extracted from params._meta as the parent for the server span, and should link the ambient context if one is present.

Read that twice, because the ordering is the whole trick.

The MCP client span becomes the parent of the MCP server span regardless of transport. Span links then record the transport context separately, where one exists.

Why parent from _meta rather than from HTTP headers? Because _meta rides inside the protocol. It survives stdio, gateways that rewrite headers, and the stateless routing layer. HTTP headers do not.

That is also why this works for local stdio servers, where there is no HTTP request to hang a header on at all.

How to Instrument an MCP Server With OpenTelemetry

The order matters. Do it in this sequence and you get a usable trace on the first try.

1. Install the SDK and set a real service name. The service.name value shows up as the box in every trace diagram. unknown_service helps nobody.

2. Wrap the request handler, not the transport. You want one SERVER span per JSON-RPC request. Name it {mcp.method.name} {target} and set the kind explicitly.

3. Extract the parent from params._meta before you create the span. Pull traceparent out and pass it to your propagator's extract call. If it is missing, start a fresh root span rather than throwing.

4. Set mcp.method.name on every span. It is the one required attribute. Everything downstream groups by it.

5. Add gen_ai.operation.name set to execute_tool on tool calls. This is what makes your MCP spans join the wider GenAI span tree instead of sitting off to the side.

6. Record errors as error.type, and set rpc.response.status_code when the JSON-RPC response carries an error code.

7. Propagate outward. If your tool calls a database or a third-party API, those become child spans automatically once the context is active.

8. Keep payload capture opt-in. Gate gen_ai.tool.call.arguments behind an environment variable. Default it off.

One caution before you wire this into a build pipeline: the GenAI conventions are still under active development and not stable. Pin your instrumentation library version and read the changelog on upgrade. Attribute names have moved before.

Coding agents including VS Code Copilot, OpenAI Codex and Claude Code already emit OTel GenAI traces, so the client half is increasingly free.

What to Actually Alert On

Dashboards are easy. Useful alerts are not.

Alert on tool-level error rate, not server-level. One broken tool out of thirty will never move an aggregate. Group by mcp.method.name plus gen_ai.tool.name and alert per tool.

Alert on the client-server duration gap. If client duration p95 runs far above server duration p95, your server is fine and your gateway is not.

Alert on p95, never on mean. Agent latency distributions are brutally long-tailed. One tool doing a cold database connect will hide entirely in an average.

Alert on retry storms. Same tool name, same arguments, three or more calls inside one trace means the agent is stuck in a loop and burning tokens.

Alert on protocol version drift. A sudden rise in old mcp.protocol.version values means a client rolled back, or a new one connected on the deprecated path.

What not to page on: tools/list volume. Cacheable list results arrived in the 2026-07-28 spec with ttlMs and cacheScope, so list traffic now varies with client cache behaviour, not with demand.

One thing tracing will not tell you: whether the answer was correct. Latency and error rate say nothing about quality — that gap is what MCP evals exist to close. Run both.

How to Test MCP Observability Before You Ship

Instrumentation that emits nothing fails silently. That is the whole problem with it.

Check these four before you call it done.

1. Does a real client call produce two spans, not one? A client span and a server span, with the server span's parent set to the client span.
2. Does _meta actually carry traceparent on the wire? Read the raw frame.
3. Does a deliberately failing tool set error.type? Force a bad argument and look.
4. Does a stdio server still correlate? network.transport should read pipe and the parent should still resolve.

The fastest way to check all four is to drive the server with a real client and watch the JSON-RPC traffic. For the wider checklist, my step-by-step guide to testing MCP servers covers the functional side.

Five MCP Observability Mistakes I Keep Seeing

1. Treating a JSON-RPC error as a success. The transport returns 200. The call failed. If your error rate comes from HTTP status codes alone, it is wrong.

2. Using RPC semantic conventions instead of MCP ones. You get a span called POST with no tool name. Technically valid, operationally useless.

3. Logging tool arguments by default. gen_ai.tool.call.arguments is opt-in because arguments carry customer data and credentials. Shipping it to a third-party backend by default is a data-handling incident waiting to happen.

4. Instrumenting the transport instead of the handler. A span per HTTP request looks fine until one request carries a batch, or the transport is stdio and there is no HTTP request at all.

5. Assuming the conventions are frozen. They are not stable. An unpinned instrumentation library can rename an attribute in a patch release and quietly break every dashboard that depends on it.

If you are auditing what your server exposes more broadly, my complete MCP server security guide covers the data-handling side of mistake three.

Instrument One Tool First

MCP observability comes down to three moves. Emit MCP-specific spans rather than generic RPC ones, propagate trace context through params._meta, and alert per tool instead of per server.

The spec deprecating Logging was not a loss. It pushed MCP onto a vocabulary that every observability backend already speaks.

Start with one tool. Confirm the client span actually parents the server span. Then expand.

And if you run several servers behind one entry point, the gateway itself becomes a span worth watching — my MCP gateway guide covers that routing layer.

FAQ