← Writing

Building an MCP Server for a Production SaaS: Lessons from Exposing an Experimentation Platform to LLMs

20 July 2026

MCPLLMAgentic WorkflowsTypeScriptAPI DesignAI Safety

Building an MCP Server for a Production SaaS: Lessons from Exposing an Experimentation Platform to LLMs

What actually changes when you put a real product behind Model Context Protocol (not a demo filesystem server)

By Muhammad Zia | Full Stack AI Engineer

Why this post exists

In 2025 I shipped an MCP server that exposes Convert's experimentation platform to LLM clients as tools. Recruiters now ask about MCP in screens. Most engineers have only read the Anthropic announcement or wired up the reference filesystem server. Very few have put a production SaaS behind it, with real auth, write actions, schema versioning, and the prompt-sensitivity problems that show up the moment a model starts calling your tools.

This is that write-up. I'll keep the MCP primer short and spend the space on decisions that matter in production: tool-first design, what becomes a tool vs what stays behind confirmation, auth for LLM clients acting on behalf of users, idempotency for writes, and the failures we hit (schema versioning and tool-description prompt sensitivity).

I also open-sourced mcp-policy-guard, the policy middleware I built after wishing it existed while shipping this work. More on that at the end.

MCP in one paragraph (for people who already ship APIs)

Model Context Protocol is a standard way for LLM hosts (Claude Desktop, Cursor, custom agents) to discover and call tools, read resources, and pull prompts from a server you run. Think OpenAPI for agents, but with a session lifecycle and a transport layer (stdio or HTTP/SSE) instead of "here's a REST URL, good luck."

If you already design APIs, the mental model is: each tool is a typed RPC method with a JSON Schema for inputs and a structured content result. The host lists tools, the model picks one, the host calls your server, you execute, you return content. That's the whole loop.

What MCP does not give you: auth, rate limits, audit logs, confirmation flows, or PII redaction. Those are on you. That gap is why most public MCP servers are demo-grade, and why production ones look different.

Tool-first beats chat-first

The first design mistake is treating MCP as "chat with your product." Chat UIs bury capability behind natural language. Tools force you to name the actions.

At Convert, the platform already had a clear action surface: list experiments, get results, create variants, start/stop tests, read goals. Mapping those to tools was mostly a product decision, not an engineering one:

| Platform action | MCP tool? | Why | |---|---|---| | List experiments | Yes | Read-only, high value for agents | | Get experiment results | Yes | The reason anyone opens the product | | Create draft experiment | Yes, with confirmation | Write, but reversible | | Start / stop experiment | Yes, require confirmation | Irreversible traffic impact | | Delete experiment | Deny or confirm + narrow RBAC | Destructive | | Bulk update billing | Never a tool | Out of scope for LLM clients |

Tool-first design forces the hard question early: which verbs is an LLM allowed to conjugate on behalf of a user? Chat-first hides that question until something goes wrong in production.

Architecture

flowchart LR
  Host[LLM Host] -->|stdio / HTTP| Guard[mcp-policy-guard]
  Guard --> Auth[Auth + tenant scope]
  Auth --> Tools[MCP Tool Handlers]
  Tools --> API[Convert Platform API]
  Guard -->|audit JSONL| Logs[Audit sink]

The MCP server sits as a thin adapter over existing platform APIs. We did not reimplement experimentation logic inside tool handlers. Handlers validate, authorize, call the same services the web app uses, and shape the response for the model (short, structured, no HTML).

That thinness matters. When the platform API changes, you update one adapter, not a parallel agent stack that drifts.

Auth: LLM clients acting on behalf of users

An MCP session is not a user typing in a browser with a cookie. The host process holds credentials and calls tools. You need:

  1. Who is the principal? OAuth token or API key bound to a user/org, established at session start.
  2. What is the tenant scope? Every tool call must carry org/project context. Never trust the model to "remember" which workspace it is in. Put it in the tool args or the session metadata and enforce server-side.
  3. What can this principal do? Reuse your existing RBAC. Do not invent a second permission system for agents. An agent with a user's token should not exceed that user's permissions.

We bind the session to a scoped token at connect time. Tool handlers never accept a raw "act as user X" argument from the model. If the model wants a different workspace, it has to go through a tool that re-scopes with an explicit, auditable step.

Idempotency for write tools

Models retry. Hosts retry. Networks drop mid-call. Write tools without idempotency keys will create duplicate experiments, duplicate variants, or double-start a test.

Pattern that worked:

server.registerTool(
  "create_experiment",
  {
    description: "Create a draft experiment. Pass the same idempotencyKey to safely retry.",
    inputSchema: {
      name: z.string(),
      projectId: z.string(),
      idempotencyKey: z.string().uuid(),
    },
  },
  async ({ name, projectId, idempotencyKey }) => {
    const experiment = await experiments.createDraft({
      name,
      projectId,
      idempotencyKey, // server stores and dedupes
    });
    return {
      content: [{ type: "text", text: JSON.stringify({ id: experiment.id, status: experiment.status }) }],
    };
  }
);

Put the idempotency key in the schema and in the tool description. Models follow descriptions more reliably than you'd like, and ignore them when the wording is vague. Be explicit: "Pass the same idempotencyKey to safely retry."

What went wrong

1. Schema versioning

We shipped tools, then renamed fields in the platform API. The MCP schema lagged. Models kept sending the old field names because cached tool lists and old system prompts still described the previous shape.

Fix: treat tool schemas as a public API. Version tool names when the contract breaks (create_experiment_v2) or keep aliases during a deprecation window. Document breaking changes the same way you would for a REST API used by third parties, because that is what this is.

2. Tool-description prompt sensitivity

This one surprised us. Changing a tool description from "Start an experiment" to "Start an experiment. This sends traffic to variants immediately." cut accidental start calls dramatically. Adding "Prefer list_experiments before mutating" reduced wrong-project writes.

Tool descriptions are prompts. They compete with the system prompt and the user's message. Short, imperative, consequence-aware descriptions outperform clever marketing copy. We A/B'd description wording the way product teams A/B CTAs, because the failure mode (wrong write) is expensive.

3. Over-exposing admin surface

Early versions exposed internal admin tools "for completeness." A curious model will find them. Default deny is the right default. If a tool is not needed for the agent use case, do not register it.

Confirmation for dangerous writes

MCP does not have a universal "are you sure?" primitive across all hosts. We implemented a two-phase pattern at the guard layer:

  1. First call to a dangerous tool returns a structured "confirmation required" result with a one-time token.
  2. The model (or host UX) retries with the token.
  3. Token is single-use and expires in five minutes.

That pattern is now the core of mcp-policy-guard. Wrapping looks like this:

import { guard, allow, deny, requireConfirmation } from "mcp-policy-guard";

const guarded = guard(server, {
  policies: [
    allow("list_*", "get_*", "search_*"),
    requireConfirmation("create_*", "start_*", "stop_*", "update_*"),
    deny("admin_*", "delete_*"),
  ],
  rateLimit: { windowMs: 60_000, maxCalls: 30, perTool: true },
  audit: { sink: "stdout" },
  redact: { patterns: ["email", "phone", "creditCard"] },
});

Trade-offs / what I'd do differently

  • I'd version tools from day one. We treated schemas as internal and paid for it when hosts cached old lists.
  • I'd invest earlier in evaluation. Golden transcripts of "user asks X → expected tool sequence" would have caught description regressions faster than production anecdotes.
  • I'd keep the tool surface smaller. Twenty sharp tools beat forty mediocre ones. Models choose worse as the menu grows.
  • I'd ship guardrails as middleware, not per-handler checks. Per-handler auth and confirmation is how gaps appear. Central policy is how you sleep.

Closing

A production MCP server is mostly API design with sharper failure modes. Auth, idempotency, confirmation, audit, and ruthless tool selection matter more than protocol trivia. If you are exposing a SaaS to LLMs in 2026, start with the threat model and the write path, not the hello-world stdio transport.

Questions or war stories: mozia.dev / LinkedIn.