MCP Gateway Architecture for Enterprise AI Agents
Where an MCP gateway sits between AI agents and enterprise systems, what it should and should not do, and how to put one in front of existing APIs and integration flows.
The Model Context Protocol (MCP) gives AI agents a standard way to discover and call tools. That is exactly what makes it attractive for enterprise integration, and exactly what makes it risky: once an agent can call a tool, it can create sales orders, change employee records or release payments as fast as it can reason. An MCP gateway is the control point that lets you offer those tools to agents without handing every agent a direct line into ERP, HR and finance.
Why agents need a gateway
Integration teams have solved this problem before. Twenty years ago point-to-point interfaces gave way to middleware; later, API gateways put a managed front door on business services. Agentic integration repeats the pattern with a new kind of consumer, one that chooses its own calls at runtime instead of following a fixed interface specification.
Without a central point, each agent team ends up wiring its own MCP servers, credentials and logging. You get the familiar symptoms: duplicated connectors, inconsistent security, no single view of what agents actually did, and no fast way to switch a misbehaving tool off. A gateway gives you one place to answer the questions auditors and operations teams will ask: which agent, acting for which user, called which tool, with what result?
MCP in one minute
MCP is an open protocol built on JSON-RPC 2.0. An MCP client (inside an agent or AI application) connects to an MCP server, which exposes three kinds of capability:
- Tools: functions the model can invoke, each with a name, description and JSON Schema for its input (for example
get_order_status). - Resources: read-only context the client can fetch, such as a document or a configuration record.
- Prompts: reusable prompt templates the server offers.
Clients discover tools with tools/list and invoke them with tools/call. Servers run locally over standard input/output, or remotely over HTTP (the Streamable HTTP transport). For enterprise use the remote HTTP form matters most, because that is the one you can put behind a gateway, secure with OAuth and operate centrally.
initialize handshake or Mcp-Session-Id, every request carries its protocol version and client capabilities in _meta, and HTTP requests carry Mcp-Method and Mcp-Name headers a gateway can route and authorize on. Servers that need more input mid-call use Multi Round-Trip Requests instead of calling back into the client. Details in What MCP 2026-07-28 changes for integration architects.Reference architecture
The key design decision is the bottom half of the diagram. MCP servers should be thin adapters over your existing managed APIs and integration flows, not new back doors into the database. The mappings, retries, error handling and monitoring your integration platform already provides keep working, and the agent becomes one more governed consumer.
What the gateway should do
| Capability | Why it matters for agents |
|---|---|
| Server registry and routing | One endpoint for agents; the gateway decides which MCP servers each agent may even see. |
| Per-tool authorization | Access is granted per tool, per agent and per user, not per server. Reading an order and cancelling one are different permissions. |
| Filtered discovery | Return only permitted tools from tools/list. A tool the agent cannot see is a tool it cannot be tricked into calling. |
| Approval gates | Pause high-impact calls (payments, master-data changes) for human confirmation. |
| Rate limits and quotas | Agents retry and loop. Limits protect back-end systems from runaway reasoning. |
| Payload inspection and redaction | Block secrets and mask personal data in both arguments and results before they reach the model or the logs. |
| Audit trail | Record agent, user, tool, arguments summary, decision and outcome with a correlation ID. |
| Kill switch | Disable one tool or one agent instantly, without redeploying anything. |
Equally important is what the gateway should not do. It should not hold business logic, transform payloads between systems, or become the only authorization check. Those belong in the integration layer and the systems of record.
Identity and authorization
MCP's authorization specification for HTTP transports is based on OAuth 2.1 (hardened in 2026-07-28 with issuer validation and Client ID Metadata Documents; see agent identity and authorization): the MCP server (or the gateway in front of it) acts as an OAuth resource server and validates access tokens issued by your identity provider. In practice, design for three identities on every call:
- The user the agent is acting for. Their permissions should cap what the agent can do, so an agent should never be able to read a salary its user could not.
- The agent itself, registered as a distinct client so you can rate-limit, audit and revoke it independently.
- The downstream service identity the MCP server uses to call your APIs, scoped to the minimum the tool needs.
Avoid passing the user's token straight through to back-end systems. Exchange it at the gateway or MCP server for a narrowly scoped downstream token, so a leaked token from an agent session cannot be replayed against unrelated APIs. And keep enforcing authorization in the downstream APIs too: the gateway is one layer of defence, not the only one.
Worked example: an order-status tool on existing APIs
Suppose sales copilots need to answer "where is order 4500012345?". The order data lives in ERP, and an integration flow already exposes it as a managed REST API used by the customer portal. Rather than letting the agent query ERP, wrap that API as an MCP tool.
The tool definition the server advertises:
{
"name": "get_order_status",
"description": "Return delivery and billing status for one sales order. Read-only.",
"inputSchema": {
"type": "object",
"properties": {
"orderId": { "type": "string", "pattern": "^[0-9]{10}$" }
},
"required": ["orderId"],
"additionalProperties": false
},
"annotations": { "readOnlyHint": true }
}The request flow:
Agent ──tools/call get_order_status──▶ MCP gateway
1. validate OAuth token → user = j.smith, agent = sales-copilot
2. policy: sales-copilot may call get_order_status for sales org US01 ✔
3. schema check: orderId matches ^[0-9]{10}$ ✔
4. route ──▶ MCP server "orders"
└─▶ API gateway GET /orders/4500012345/status
└─▶ integration flow ──▶ ERP
5. redact fields not needed for the answer
6. audit: correlationId, user, agent, tool, decision, latency, outcome
◀── structured result returned to the agentNotice what the agent never sees: ERP hostnames, technical users, or any field the portal API does not already expose. If you are on SAP BTP, the same shape applies with an SAP Cloud Integration flow behind SAP API Management; on other stacks, substitute your own iPaaS and API gateway.
Designing enterprise tools agents can use safely
- Model business tasks, not endpoints.
get_order_statusis better than a genericcall_erp_api. Narrow tools are easier to authorize, test and explain. - Separate read from write. Put read-only and side-effecting operations in different tools, and mark them with annotations such as
readOnlyHintanddestructiveHint. Treat annotations as hints for the client, never as security controls. - Make writes idempotent. Require an idempotency key on create operations so an agent's retry does not post the same payment twice.
- Constrain inputs tightly. Use patterns, enums and
additionalProperties: false. Free-text fields are where prompt injection travels. - Return structured, minimal results. Give the model what it needs to answer, not the full record.
- Write descriptions for the model. Say what the tool does, when to use it and what it will not do. Clear descriptions reduce wrong-tool calls more than any prompt trick.
Observability and operations
Run agent traffic like any other production integration. Propagate a correlation ID from the agent session through the gateway, MCP server, API gateway and integration flow so one search reconstructs the whole chain. Track per-tool error rates, latency and denied calls; a sudden rise in denials often means a prompt-injection attempt or a mis-scoped agent. Set timeouts shorter than the agent's own, so slow back ends fail predictably instead of leaving the model waiting. Version tools explicitly, and when a schema changes, publish a new tool rather than silently breaking agents that cached the old definition.
Gateway products in 2026
Most enterprises do not build a gateway from scratch. Commonly evaluated options include SAP Integration Suite's MCP Gateway (the endorsed path for agents reaching SAP systems; see our SAP guide), Amazon Bedrock AgentCore Gateway, Azure API Management, Kong AI Gateway, the open-source agentgateway project hosted by the Agentic AI Foundation, IBM ContextForge and Microsoft's open-source MCP Gateway. Several of these also proxy A2A agent-to-agent traffic. Choose the one that sits closest to the APIs you already govern, and make sure whichever you pick supports the 2026-07-28 revision's header-based routing and per-user filtered tool lists.
Common pitfalls
- One super-tool with broad credentials. It works in a demo and fails every security review.
- Trusting tool output as instructions. Data returned from systems, such as a customer note that says "ignore previous instructions", must be treated as data. Keep high-impact actions behind approval regardless of what the model concludes.
- Logging full payloads. Agent logs quickly become the largest store of personal data in the landscape. Log metadata and summaries, and redact.
- Gateway as the only check. If the downstream API trusts anything that comes from the gateway, one misconfigured policy exposes everything.
- Skipping the integration layer. Direct database or system access throws away years of mapping, validation and monitoring work.
Implementation checklist
- Inventory every MCP server, the tools it advertises and the systems each tool reaches.
- Register each agent as its own OAuth client; carry user context on every call.
- Authorize per tool, filter
tools/listto permitted tools, and re-check authorization downstream. - Back MCP tools with existing managed APIs or integration flows, not direct system access.
- Validate inputs strictly; require idempotency keys for writes.
- Gate high-impact operations behind human approval.
- Apply rate limits, timeouts and a per-tool kill switch.
- Redact secrets and personal data in results and logs; record audit metadata with correlation IDs.
- Require idempotency keys on write tools: under the stateless spec, retries are routine.
- Version tools and test them with adversarial prompts before go-live.