§ war story

Our MCP server authenticated everyone as me

2026-05-12 · by Dennis Gubsky · ~7 min read

We added MCP to our agent stack for one of the cleanest reasons you can move to MCP: to stop putting bearer tokens in our agents' system prompts. We had every reason to feel good about the decision. The model surface got typed, the bearer left the LLM's line of sight, hallucination rates on tool calls dropped sharply. It was a real architectural win.

Some days later we discovered that our shiny new MCP integration had quietly authenticated every user's agent calls as me. Documents that should have belonged to a second user were stored against my user_id. The bug had been live in production for a stretch of days before we caught it. The reason we found it was that the second user kept complaining about flaky agents and we eventually believed her.

Why we moved to MCP in the first place

Once we'd shipped the basic agentic toolset — Read, WebFetch, WebSearch, HTTP — our agents could finally do real work. They could fetch data from our REST API, transform it, save results back. The agentic-runtime substrate was producing useful output.

Two problems showed up almost immediately:

The simpler models hallucinated parameters. Given the built-in HTTP tool's free-form {url, method, headers, body} input schema, agents had to compose REST calls from whatever the system prompt told them about our API. Tool-trained LLMs are reasonably good at this — they almost never invent paths from scratch — but they would occasionally swap query-param names, use the wrong content-type on PATCH, or miss a required field in the body. Production-acceptable failure rate at v0.3.x, not at v1.0.
The bearer token had to live somewhere the model could see. For agents to hit authenticated endpoints, the bearer needed to end up in the HTTP tool's headers arg — which means in the system prompt, which means in the LLM provider's request, the assistant's echoed turns, the tool-call records in our events table, and every operator transcript replay. The model could literally paste the bearer into a follow-up assistant turn and we'd have no way to scrub it after the fact.

The clean answer to both problems is the same: wrap the REST API as an MCP server. Each REST operation becomes a typed tool with its own JSON Schema — the model picks a tool by name and fills typed parameters rather than composing HTTP calls. And the bearer moves out of the model's line of sight entirely, into operator yaml as a substitution template that the runtime injects at request-build time.

We built it. Hallucinations dropped. The bearer disappeared from every model-facing surface. Agents reliably called mcp__jobs__getAgentContext() with empty parameters and got real data back. The shape of the win was exactly the shape we expected.

The developer-token shortcut

For development simplicity, we authorized the MCP server with a single bearer token — one I'd generated for my own developer account in our user registry. The operator yaml looked roughly like this:

mcp_servers:
  jobs:
    transport: http
    url: http://localhost:3000/api/mcp
    headers:
      Authorization: "Bearer "

It worked perfectly. My agents called the MCP server, the MCP server validated the bearer against our auth registry, resolved it to my developer user_id, and forwarded calls to the underlying REST API as me. Every roundtrip behaved exactly as intended. The model didn't see the token. The token authorized the calls. We shipped.

Everything worked for us. We were the only ones using it.

The second user

Then a second user joined. She started running her own agents against the system, and she started complaining. The agents worked sometimes and not other times. She could fetch her profile sometimes and not other times. Sometimes the agent would complete the run and the resulting document would simply not appear in her account. The complaints were vague and the failure mode was inconsistent.

This coincided with our larger migration to multi-provider routing — we'd just started running some agents on deepseek-v4-pro, others on gemini-2.0-flash-lite, others on the original Anthropic baseline. We naturally blamed the symptoms on hallucinations from the new models. Of course they were producing inconsistent outputs; we'd just changed the model layer. We tightened prompts, narrowed allowed_tools, watched the resolver routing logs, and our test runs got reliable again.

The second user kept complaining anyway. After a few days of blaming-the-model-and-tuning, with our own runs visibly working and hers visibly not, we eventually accepted that the model hypothesis didn't explain the residual. Something else was going on.

The database transaction analysis

So we did the boring thing and pulled a transaction log out of the database. Specifically: every INSERT and UPDATE that had landed in the last 72 hours against the documents table, joined to the user_id that owned each row.

The pattern jumped out within about thirty seconds of looking. Documents that, by every other piece of metadata, should have belonged to the second user — created during runs she had initiated, referencing her profile, fetched in response to her prompts — were quietly stamped with my user_id. Every single one. There was no flakiness; there was no inconsistency. Every agent call from every user's run was authenticating as me, the developer, and writing results into my account.

The reason was sitting in our operator yaml in plain sight. The bearer token we'd used to authorize the MCP server was a bearer I had generated for my user. Our MCP server's job was to validate the inbound bearer, resolve it to a user, and forward calls to the REST API as that user. The system was working exactly as designed. It was just resolving to the wrong user on every call that wasn't mine.

For our own runs, this was invisible — the bearer happened to resolve to the same user_id that triggered the run, so document ownership matched and nothing looked wrong. For the second user's runs, her prompts hit the agent, the agent called the MCP server, the MCP server saw a bearer that resolved to my user, and her documents were created under my account. The "flakiness" she saw was every call quietly succeeding into the wrong account.

The middle-of-the-night fix

The fix had to be: give every agent run its own per-user bearer, one that resolves to the user who initiated the run. Naturally this took a couple of sunset-to-dawn workdays, because that's when these realizations consolidate.

What we built — and what eventually became loomcycle's ${run.user_bearer} substitution mechanism (v0.8.14) — was a per-run bearer field on the POST /v1/runs request shape. The caller supplies a token bound to the authenticated user. The runtime attaches it to ctx, propagates it through sub-agent inheritance, and when an MCP tool fires, the HTTP client substitutes that per-run bearer into the outbound Authorization header per the operator yaml template:

mcp_servers:
  jobs:
    transport: http
    url: http://localhost:3000/api/mcp
    headers:
      Authorization: "Bearer ${run.user_bearer}"   # was: "Bearer <my-developer-token>"

The substitution happens per-request against a local copy of the headers map, not against the shared one — so two concurrent runs against the same MCP server send distinct bearers simultaneously without coordination. The model still never sees the token; the runtime still does the unsafe work outside the model's view. The only change is which token gets injected into which run's outbound header. Per-user instead of per-runtime.

The shared-developer-token approach is now structurally impossible in the operator yaml schema — the substitution form ${run.user_bearer} is the only way to thread a runtime-resolved bearer through to an MCP call. If you don't pass a user_bearer on the run request, loomcycle drops the Authorization header entirely and the MCP server returns a clean 401. Far easier to debug than the silent wrong-user resolution we had before.

Details of the substitution mechanism, sub-agent inheritance, and what the model sees on which surfaces are documented at docs/MCP_INTEGRATION.md §3 — the boundary table in particular is the canonical artifact for "what surfaces does the bearer appear on" auditing.

Two lessons

Lesson 1: be very careful about authorization paths in agentic systems. They look different from human auth paths. In a human-facing system, the bearer is bound to the session of the person clicking the button. In an agentic system, the bearer has to be bound to the run that the person triggered, which means it has to thread through the runtime layer, the tool dispatcher, the MCP transport, and back into your REST API — and at every hop, the temptation to just paste a static token "for now" is enormous. Each "for now" is a multi-tenancy bug waiting to be born.

Lesson 2: one worried second user is always better than one self-confident developer. The second user's complaints were the only signal that anything was wrong. We had our own runs working perfectly and a coherent-sounding hypothesis ("must be the new models") that explained away the noise. If she'd been less persistent, or if we'd been less willing to eventually believe her, the bug would have lived until we noticed our own database growing impossibly fast with content we didn't recognize.

This wasn't the first time an auth-leak pattern cost us something. See the $80 prequel for an earlier and more expensive incident in the same family — a forgotten code path that bypassed the runtime and inherited an API key it shouldn't have. Different shape of mistake; same lesson about how multi-tenant authorization is *not* what single-tenant authorization looks like with another user added.