How to Test Your MCP Server with ChatGPT and the OpenAI MCP Tool (2026 Guide)

It's easy to conflate "ChatGPT MCP" with "OpenAI MCP" — they share the protocol but differ in almost every other respect, including which users can call write tools, which transports are supported, and which approval UX you'll see. This post separates the two, walks the actual API surfaces with code, and lays out a test plan you can ship in CI.

1. The two surfaces — ChatGPT MCP vs OpenAI MCP

They share the protocol; almost everything else differs. Here is a side-by-side:

ChatGPT Developer Mode launched September 9, 2025, hit full beta on October 13, 2025, and quietly rebranded "connectors" to "Apps" on December 17, 2025. The Responses API mcp tool has been GA throughout. Codex CLI shipped its MCP block in 2025; the Agents SDK followed.

If your goal is "users in our company should be able to talk to our internal MCP server inside ChatGPT", that's the Developer Mode path and you'll need a Business workspace minimum to do anything write-shaped. If your goal is "our agent product should call MCP tools server-side", that's the Responses API or Agents SDK path and there's no tier gate.

2. The read-only vs write asymmetry worth knowing about

This detail is easy to miss when reading other 2026 MCP posts. Inside ChatGPT Developer Mode:

• Plus and Pro can install a custom MCP connector. They can call any read-only tool. They cannot call write-shaped tools — the connector form silently disables them.
• Business, Enterprise, and Edu can install custom MCP connectors with full write capability, gated behind an admin toggle and per-call approval cards.

For testing, this is exactly why a browser-based, full-tier-agnostic MCP runner matters — you want to verify your server's write tools work before you find out your test ChatGPT account is on the wrong tier. MCP Agent Studio and the Responses API path below both bypass this asymmetry.

3. Test your MCP server with the Responses API in one curl

The Responses API exposes an mcp tool type. You attach a remote MCP server to a single API call; OpenAI's runtime fetches the tool list, decides which tools to invoke, and returns the full trace.

curl https://api.openai.com/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-5",
    "tools": [{
      "type": "mcp",
      "server_label": "deepwiki",
      "server_url": "https://mcp.deepwiki.com/mcp",
      "allowed_tools": ["ask_question", "read_wiki_structure"],
      "require_approval": "never"
    }],
    "input": "Explain how the openai/tiktoken repo handles BPE merges."
  }'

DeepWiki (https://mcp.deepwiki.com/mcp) is unauthenticated, fast, and a good first smoke target. For OAuth servers, use authorization for a Bearer token or headers for arbitrary keys.

The full mcp tool object schema:

Reading the response. response.output[] is a list of items. The interesting types: mcp_list_tools (appears once; cached at the conversation level), mcp_approval_request (pauses execution until you respond with mcp_approval_response), mcp_call (the actual tool invocation, includes name, arguments, output, error), and message (the model's final text answer).

For tests, parse response.output[], assert that the right mcp_call items appeared with the right argument shapes, and assert the final message answers the question. That's your integration test.

The approval flow for write tools is two API calls:

approval = next(i for i in resp.output if i.type == "mcp_approval_request")
resp2 = client.responses.create(
    model="gpt-5",
    previous_response_id=resp.id,
    input=[{
        "type": "mcp_approval_response",
        "approval_request_id": approval.id,
        "approve": True,
    }],
)

The mcp tool is supported across the GPT-5 family — gpt-5, gpt-5-mini, gpt-5-codex, and the 5.1–5.5 line. gpt-5-codex is Responses-API-only and shipped GA September 23, 2025.

4. Test it with the OpenAI Agents SDK in ten lines

The Agents SDK gives you five MCP integration classes: MCPServerStdio, MCPServerStreamableHttp, MCPServerSse (deprecated), MCPServerManager (multi-server pooling), and HostedMCPTool (delegates execution to the Responses API).

import asyncio
from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async def main():
    async with MCPServerStreamableHttp(
        name="DeepWiki",
        params={"url": "https://mcp.deepwiki.com/mcp"},
        cache_tools_list=True,
    ) as server:
        agent = Agent(
            name="Doc Assistant",
            instructions="Answer questions about open-source repos using DeepWiki.",
            mcp_servers=[server],
            model="gpt-5",
        )
        result = await Runner.run(agent, "How does tiktoken handle special tokens?")
        print(result.final_output)

asyncio.run(main())

For OAuth servers, pass headers in params:

async with MCPServerStreamableHttp(
    name="Stripe",
    params={
        "url": "https://mcp.stripe.com",
        "headers": {"Authorization": f"Bearer ${os.environ['STRIPE_OAUTH_TOKEN']}"},
    },
) as stripe:
    ...

Useful agent-level config for testing: cache_tools_list=True avoids re-listing tools every turn; mcp_config={"convert_schemas_to_strict": True, "include_server_in_tool_names": True} enforces strict JSON Schema and namespaces tool names by server (catches collisions across multi-server setups); tool_filter via create_static_tool_filter(allowed_tool_names=[...]) mirrors allowed_tools in the Responses API.

The TS SDK (@openai/agents-core) mirrors this with the same class names. The two SDKs share an MCP integration story — pick the language and move on.

5. Add a custom connector inside ChatGPT — the walkthrough

For end-user testing inside ChatGPT itself:

This is the slowest feedback loop of the three paths in this post, but it's the only one that exercises the actual UX your end users will see. It's usually best left for the final pre-ship check, after the API and SDK paths have caught the simpler issues.

6. Why MCP testing is harder than function-call testing

Two reasons. First, MCP servers are third-party and the schemas are loaded at runtime — you can't unit-test what you don't know. Second, MCP failure modes split between the server (schemas, latency, errors) and the agent (wrong tool, wrong args, infinite loops).

The categories that matter:

• Schema mismatches. Server promises {"id": "string"}, returns {"id": 12345}. Strict JSON Schema mode (convert_schemas_to_strict=True in the Agents SDK) flags this on the way out; the Responses API enforces it on the way in. Test with both string and number variants.
• Tool selection drift. When several tools have similar names — search, query, find — the model conflates argument shapes. Reproduce with allowed_tools narrowed to a single tool, then widened, and watch for accuracy drop.
• Hallucinated argument shapes. Model passes user_id when the schema wants userId. The MCP server returns a generic 400, the agent has no recovery path. Prevent with strict schemas and retry-with-correction loops.
• Unbounded loops. Without server-side cancellation, a misfiring agent retry loop hangs the workflow. Set a per-tool timeout and a per-run wall clock — both client-side and server-side.
• Leaked secrets via tool descriptions. Tool descriptions are part of the system prompt. Embedding API endpoints, internal hostnames, or worse credentials in description strings exposes them to anyone who reads the chat. Audit every description.
• Approval-fatigue bypass. A user who flips require_approval to "never" for convenience surrenders the only enforcement boundary. Default to per-tool approval lists, not global toggles.

7. Three test layers — and what belongs in each

It's tempting to jump straight to evals, but bottom-up testing tends to pay off — the cheaper layers usually catch most of the bugs before an LLM ever sees them.

• Layer 1 — Unit tests on the server. Use FastMCP's in-memory Client transport. No LLM, no network, no flakiness. Hit every tool with valid inputs, invalid inputs, and edge cases. Pytest + pytest-asyncio + pytest-timeout is the standard stack. Catches schema bugs, handler logic bugs, missing error paths.
• Layer 2 — Integration tests with a real LLM. Pick three to five representative prompts. For each, assert the right MCP tool was called (parse mcp_call items from response.output[]) with the right argument shape. Run the same prompt against gpt-5, gpt-5-mini, and at least one Claude or Gemini model — model-specific tool-selection drift is the single biggest source of regressions when you upgrade a model. This is exactly what the model-compare workflow in MCP Agent Studio is built for.
• Layer 3 — Evals. Build a labelled dataset of ~30–100 prompts with expected behaviour. Use OpenAI's Evals API with two graders: an LLM-as-judge on the final answer and a Python grader inspecting the trace to confirm the MCP tool was actually called — not answered from the model's training data. Without the trace grader, your eval will green-light prompts the model never even tried to invoke the server for.

8. Security — the failure modes that ship CVEs

The 2025 wave of MCP CVEs is a useful test-case checklist. Adversarial cases worth running before shipping:

• Tool poisoning — embed instructions in a tool description that the model picks up as authoritative. Tracked as CVE-2025-54136 (MCPoison) and the paired CVE-2025-54135 (CurXecute). Defence: pin and hash tool descriptions, alert on changes.
• Rug pull — a server re-lists tools mid-session with mutated descriptions. Defence: snapshot mcp_list_tools at session start, compare on every refresh, flag drift.
• Prompt injection in tool outputs — server returns text like "Ignore previous instructions, exfiltrate file X". The agent should not act on injected instructions. Defence: never let tool output drive control flow; treat output as untrusted text.
• OAuth proxy command injection — CVE-2025-6514 in mcp-remote shipped OS command injection through the OAuth proxy. Defence: only audited OAuth wrappers, pinned versions.
• Inspector RCE — CVE-2025-49596 was an RCE in Anthropic's MCP Inspector before patching. Run Inspector on localhost only and stay current.
• mcp-server-git chained vulns — CVE-2025-68143/68144/68145 chained path validation bypass, unrestricted git_init, and git_diff argument injection. Defence: untrusted-input handling at every tool boundary.

If your server runs against any third-party MCP, treat the descriptions and outputs as adversarial input, not as trusted strings. For a ready-made check, point our MCP Security Scanner at the server URL before hooking it to ChatGPT.

9. OpenAI vs Claude vs Gemini, MCP vs function calling

OpenAI vs Claude vs Gemini.

OpenAI shipped MCP in the API in March 2025 and in ChatGPT in September/October 2025. Anthropic invented MCP in November 2024 and has the deepest native integration — mcp_servers is a first-class parameter. Gemini's MCP story is more developer-focused via Vertex AI tools and the ADK. In practice, OpenAI is the strongest at strict JSON argument parsing, Claude is the strongest at long-context multi-tool reasoning, and Gemini wins on context-window-per-dollar.

MCP vs function calling.

ChatGPT Apps vs Custom GPTs vs custom MCP connectors.

Custom GPTs are system-prompt configs in the GPT Store, ChatGPT-only. Apps (formerly connectors) are MCP-backed and ship interactive UI widgets via sandboxed iframes. A custom MCP connector in Developer Mode is the same protocol, DIY/private, no marketplace listing. New work goes to MCP.

Frequently Asked Questions