What is an LLM gateway?

“Our teams already call OpenAI and Anthropic directly. Why put a gateway in the middle?”

The short answer: an LLM gateway is a proxy that sits between your applications and model providers. Apps get one endpoint and a virtual key instead of a raw provider key, and every request picks up access control, a budget, logging, and failover on the way through. You need one as soon as more than one team, app, or provider is involved.

Direct calls are the right way to start. One app, one provider, one key in a secrets manager: putting infrastructure in front of that would be ceremony. The trouble is that LLM usage inside a company doesn’t stay that shape for long, and the failure modes of “everyone calls the provider directly” arrive on a fairly predictable schedule.

What does an LLM gateway actually do?

Five jobs, in the order most teams discover they need them:

1. One API across providers. The gateway speaks the OpenAI and Anthropic wire formats and translates behind the scenes. Most AI tools already speak OpenAI’s /chat/completions; an app written against it can be pointed at Claude, Gemini, or a local vLLM model by changing a model name, not by rewriting the client.
2. Virtual keys instead of raw provider keys. Each user, app, and agent gets its own key, issued and revoked at the gateway. Provider keys live in exactly one place. Cutting off a departed contractor no longer means rotating a credential that’s pasted into nine repos.
3. Budgets and attribution. Spend is metered per key as requests happen, so a team can be capped at a number and overspend gets blocked before it becomes an invoice. The provider’s bill tells you what you spent; it can’t tell you who.
4. Routing and failover. When a provider has an outage or a rate-limit cliff, the gateway retries against a fallback instead of surfacing errors to every app at once.
5. A policy point. Once all model traffic flows through one place, that place can log it, decide which teams see which models, and run guardrails that strip PII and secrets before anything leaves your network.

None of these are model problems. They’re the same governance problems you already solved for databases and cloud accounts, showing up for LLM traffic.

Job five has a quiet corollary: data ownership. Once every prompt and completion crosses a point you control, the full history of how your company uses LLMs can accumulate in your own storage instead of living only with a provider. That history starts as an audit trail and grows into an asset — an eval base for the next model migration, and training material if you ever fine-tune a model of your own. It’s also the raw material for what’s increasingly called a company brain: an agent with access to your corporate knowledge and the accumulated record of how your people actually use AI. You can’t build one on history you never kept. We unpack both in the dedicated post.

What breaks without one?

The shared key. A provider key gets created for one project, then travels: into CI, into a notebook, onto a teammate’s laptop. Six months later nobody can say what breaks if it’s rotated, so it isn’t.

The invisible invoice. Finance sees one line per provider. When that number doubles in a month, “which team, which app, which model” has no answer, because attribution was never captured at request time — and it can only be captured at request time.

The Friday agent. Scripts and coding agents retry. A loop a human would notice in a minute runs unattended for hours, and without a per-key cap the first alert is the invoice.

Agents multiply request volume without multiplying the attention watching it.

In our view, that’s the strongest argument for putting a gateway in before the coding agents arrive, not after.

When do you actually need one?

Not on day one. A single application on a single provider, with spend small enough to eyeball, doesn’t need the extra moving part.

The realistic triggers, any one of which is enough:

• A second provider. Claude for code, a lighter model for bulk extraction: model diversity is normal practice now, and per-app provider wiring stops scaling right there.
• A second team starts calling models, and you’d rather not hand them your key.
• You’re rolling out IDE assistants or coding agents, which turn every developer into a high-volume API caller.
• Finance or security asks who spent this, what data left the network, or which models were used, and your current logs can’t answer. A first shadow AI detection pass tends to surface exactly that question.

Most companies we talk to hit the agent trigger first, and hit it faster than they planned to.

Is this the same as an API gateway? What about MCP gateways?

HTTP-layer gateways. A generic API gateway (Kong, nginx, a cloud load balancer) works at the HTTP layer: routing, auth, rate limits by request count. It doesn’t know what a token is. An LLM gateway is model-aware — it meters cost in tokens and dollars, understands provider wire formats and streaming, and can read the request body to apply guardrails. You won’t get that from an HTTP-layer product without building it yourself.

MCP gateways. An MCP gateway governs the other half of agent traffic: the tools agents call via the Model Context Protocol, rather than the models themselves. Different traffic, same governance instinct. If you’re running agents seriously, you’ll end up wanting both; MCP gateways deserve their own post, and we’ll write it.

Should you go without, buy hosted, or self-host?

The open-source route deserves a fair word. LiteLLM in particular is capable software: we liked it enough to build SecondGate, SecondStack’s gateway, on a fork of it — the differences are laid out in SecondStack vs LiteLLM. The caveats are operational, not functional — the gateway itself is maybe a third of the work, and the SSO integration, spend dashboards, guardrail plumbing, and upgrade discipline around it are the part that doesn’t fit in anyone’s spare time.

Hosted gateways solve the operations problem and introduce a data-flow one: every prompt and completion now transits someone else’s cloud. For plenty of companies that’s acceptable. If it isn’t for yours, self-hosting stops being a preference and becomes the requirement.

Where the gateway sits in SecondStack

SecondStack ships the gateway as one layer of a self-hosted platform. SecondGate is our LiteLLM fork, exposing OpenAI-compatible /chat/completions and /responses plus Anthropic-compatible /messages, with /embeddings, /images, and /audio alongside — so IDEs, scripts, and agents authenticate with virtual keys against endpoints they already speak.

Around it sits the part that usually gets hand-built: budgets per user, team, and key with hard cutoffs, so a runaway script hits its own limit instead of locking out the team. Access groups decide which models each team sees. Usage analytics live in ControlTower. And SecondGuard checks requests and responses for PII and secrets in two stages (a fast heuristic, then a context-aware LLM classifier), so the common case stays fast. All of it, gateway included, runs on your infrastructure, with logs and history in your own PostgreSQL.

That bundling is the reason SecondStack exists at all; the longer story is in why we built SecondStack.

If you’re weighing wiring up a gateway yourself against running the whole governed stack, write to [email protected] — we’ll show you our own gateway config.