The Self-Hosted LLM Gateway Pattern: How Mantis and LiteLLM Turn Model Routing Into a $10 Container

You are already paying the model provider for every token. Then you pay someone else for permission to call that provider.

That second bill — the gateway fee — is small enough to ignore at 10 million tokens a month and large enough to dominate infra decisions at 500 million. A self-hosted LLM gateway kills it entirely: the same routing, fallback, and cost tracking you get from OpenRouter or Portkey, running as a single container beside your apps for $10 a month instead of a 5.5% tax on every token.

TL;DR — The Math Up Front: At 100M tokens/month (~$500 in model spend at GPT-4o-class pricing), OpenRouter's 5.5% fee adds $27.50, Portkey's Production tier adds $49, and Cloudflare adds $0. A self-hosted LiteLLM or Mantis gateway adds ~$10 in container cost — flat, forever, no per-token meter. At 500M tokens the managed tax is $137 (OpenRouter) vs. still $10 self-hosted. Below ~20M tokens the managed fee is actually cheaper than operating anything yourself — the breakeven is honest, and early.

This is the self-hosted LLM gateway pattern, and Mantis — a 2026 Show HN that deploys with mantis deploy — is the latest, most approachable entry in a category that already includes LiteLLM, Portkey's open-source gateway, and a dozen Go/Rust proxies. The pattern matters more than any single tool: model routing, fallback, and cost tracking are just another tenant workload, not a billable service category.

What an LLM Gateway Actually Does​

Every team that touches more than one model reinvents the same middleware. An LLM gateway centralizes it:

Unified API. One OpenAI-compatible endpoint that fronts 100+ providers — OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, vLLM — so clients call POST /v1/chat/completions once and the gateway translates the request. LiteLLM's proxy, Mantis, and Portkey all expose this. No per-provider SDK churn.

Routing and load balancing. Policy-based routing by model, cost, latency, or tenant. gpt-4o from OpenAI, fallback to Azure OpenAI on rate limit, round-robin keys on 429/5xx. LiteLLM's Router and Mantis's combo aliases both support key rotation and combo fallback chains.

Fallback and retries. Automatic retries with timeout escalation, circuit breakers, and provider fallback. If Anthropic returns a 529 or OpenAI times out after 30 seconds, the gateway replays the request against the next provider without the client ever knowing. This is the feature that justifies a gateway on its own — without it, every app carries bespoke retry logic.

Cost tracking and budgets. Per-request token counting, per-model pricing (4,000+ models via models.dev or LiteLLM's pricing table), per-key and per-team budgets, spend alerts. The gateway knows that claude-3-5-sonnet input costs differ from gpt-4o-mini output and enforces budget: $500/mo before workloads burn through it.

Virtual keys and access control. SHA-256-hashed API keys per user or service, scoped to specific models, with usage broken down by key. One team, one key, one budget — without sharing raw provider keys.

That is the whole feature set. None of it requires a managed service. It requires a stateful proxy with a config file and a database.

The Cost Math: Why the Gateway Fee Scales and the Container Doesn't​

Here is the part every managed gateway comparison buries below the fold: the fee is additive to model spend, and it compounds.

Assume a blended model cost of $5 per million tokens — a reasonable midpoint between gpt-4o-mini ($0.15/$0.60), claude-3-5-sonnet ($3/$15), and hosted open-weight models. The gateway fee is the only variable:

* Portkey's self-hosted gateway is MIT open source and free to operate. Its managed control plane starts at $49/mo (Production) and scales with log retention and governance features. TokenMix's 2026 comparison pegs comparable SaaS tiers at $99–$499 for higher volume.

Two takeaways that survive sensitivity analysis:

At low volume, managed wins. At 10M tokens, OpenRouter's $2.75 is cheaper than operating a container you have to monitor. LiteLLM's own community guidance — "under $2K/mo LLM spend, OpenRouter or Portkey Free; over $10K, LiteLLM is the clear cost winner" — is honest about this. If you run a single prototype, pay the 5.5%.

At scale, self-hosted wins by 3–14x on gateway overhead alone. At 100M tokens, the self-hosted premium is roughly one-third the cheapest managed fee. At 500M, it is one-fourteenth of OpenRouter's take. And the self-hosted number is flat — it does not grow with tokens, models, or request count. The container costs the same whether you route 50M or 500M tokens; it is bounded by CPU and memory, not by someone else's meter.

Add a second dimension: throughput cost is not the only meter. Managed gateways charge for the control plane (logging, guardrails, caching observability) on separate tiers. Self-hosted, those are disk and Postgres rows. Helicone and Langfuse Cloud charge $50–$200/mo after traffic for observability that a self-hosted Langfuse + LiteLLM + Redis stack runs on the same $10 droplet.

Cloudflare AI Gateway deserves a callout: at $0 routing fee and 0% markup, it is the managed gateway that already concedes the core argument. If a hyperscaler can afford to route tokens for free, the routing itself was never the scarce resource — the data path and the pricing power were.

Three Contenders: Mantis, LiteLLM, and the Managed Alternatives​

LiteLLM is the incumbent self-hosted choice. Its proxy server is the reference implementation most teams copy. Python SDK for in-process use, proxy server for centralized gateway, Rust core for performance, and the largest model-pricing dataset in the ecosystem. If you want the most connectors and the most production hours, LiteLLM is the default.

Mantis is the approachable challenger. Pitched explicitly for small teams on early-stage products, Mantis trades LiteLLM's breadth for a single-command deploy story. The HN launch positioned it as "designed for small teams, deployed to AWS with mantis deploy" — less config surface, fewer moving parts, same unified API. For a team that has never operated a gateway, Mantis is the on-ramp.

OpenRouter and Portkey are the managed counterfactual. OpenRouter's value prop is zero ops and 0% markup on model prices — you pay providers' sticker and a flat 5.5% platform fee. Portkey's is observability and governance, with a gateway you can self-host for free and a managed control plane for teams that want it. Both are excellent if you prefer to outsource. That is the trade the cost table prices out.

The honest summary from the community: under $2K/mo model spend, pick managed and reclaim the ops time. Between $2K and $10K, any option is defensible — OpenRouter for simplicity, Portkey for observability, LiteLLM/Mantis if you have DevOps capacity. Above $10K, self-hosted is the cost winner because the gateway tax grows linearly and the container does not.

Running the Gateway as a Tenant Workload Beside Your Apps​

The self-hosted gateway pattern's insight is architectural: the gateway is not infrastructure you provision separately. It is just another service on your platform — same deploy path as any tenant app.

Mantis: One Command​

# Install and deploy — single command to AWS
mantis deploy
 
# Your apps now call the gateway, not providers directly
curl https://gateway.your-domain.com/v1/chat/completions \
  -H "Authorization: Bearer $MANTIS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "Explain the gateway pattern"}]
  }'

Mantis handles key management, logging, and provider routing behind that endpoint. Fallback and model aliases are config, not code changes.

LiteLLM: A Config File and a Container​

LiteLLM's self-hosted proxy is a config.yaml and a container — the shape every PaaS already knows how to run:

# litellm-config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-4o-fallback
    litellm_params:
      model: azure/gpt-4o
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
 
router_settings:
  routing_strategy: latency-based-routing
  num_retries: 3
  timeout: 30
 
litellm_settings:
  fallback_strategy: ["gpt-4o", "gpt-4o-fallback"]
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

FROM ghcr.io/berriai/litellm:main-latest
COPY litellm-config.yaml /app/config.yaml
CMD ["--config", "/app/config.yaml", "--port", "4000"]

Virtual keys, per-team budgets, and rate limits are API calls, not redeploys:

# Create a scoped virtual key for the frontend team
curl -X POST https://gateway.your-domain.com/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{
    "models": ["gpt-4o", "claude-3-5-sonnet"],
    "max_budget": 500,
    "budget_duration": "30d",
    "metadata": {"team": "frontend"}
  }'

On a platform like Bex.co — push a git repo, get a running HTTPS service — this is a bex.yml beside your app's:

# bex.yml for the gateway
app: llm-gateway
image: ghcr.io/berriai/litellm:main-latest
port: 4000
env:
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  DATABASE_URL: ${{ secrets.DATABASE_URL }}

The gateway scales like any other service. The marginal cost is disk for logs and a small always-on container — not a per-token meter that follows your usage curve.

Latency note: Self-hosted proxy overhead is sub-3ms for Go gateways (Bifrost measures ~0.62ms, Portkey's gateway ~2.65ms). LiteLLM's Python proxy adds a few milliseconds more. Against LLM inference latency (hundreds of milliseconds to seconds), the gateway is noise.

Data Ownership: Whose Infrastructure Sees Your Prompts​

The cost math is only half the argument. The other half is who holds the data path.

A managed gateway is a third-party proxy. Every prompt, completion, API key, and usage log transits someone else's infrastructure. OpenRouter's docs are explicit that they route through their network; Portkey's control plane observes every request for governance. Their retention policies, logging granularity, and subprocessors become your compliance surface, whether you chose them or not.

A self-hosted gateway keeps that surface inside your VPC:

• API keys stay with you. No sharing raw provider keys with a proxy vendor. The gateway holds them; clients hold virtual keys scoped to your policy.
• Prompts and completions don't egress to a third party. For regulated workloads — healthcare, finance, anything with PII in the prompt — "no data leaves our infrastructure except to the model provider" is a materially different claim than "no data leaves except to the proxy vendor and then to the model provider."
• Logs are yours. Usage, latency, and error logs land in your Postgres, your ClickHouse, your object storage. Retention is your policy, not a tier limit. No per-log-ingestion fee.
• Switching cost collapses. "Switch providers with a configuration change. No code rewrites, no migration pain" — every self-hosted gateway advertises this, but self-hosted makes it true at the gateway layer too. Moving from OpenRouter to direct provider calls means unwiring a vendor. Moving from LiteLLM gpt-4o to azure/gpt-4o means changing one line in config.yaml.

For teams that cannot send unredacted prompts to a public API at all, the self-hosted pattern extends to provider-side self-hosting — Ollama, vLLM, or a Hetzner GPU box (the RTX PRO 6000 Blackwell line at €889/mo) behind the same gateway endpoint. The gateway abstraction is identical whether the model lives at OpenAI or on your own hardware.

When to Pick Which​

No honest guide ends with "always self-host." The threshold is clear:

Pick managed (OpenRouter, Portkey Free, Cloudflare) when:

• You spend under ~$2K/mo on tokens (the 5.5% fee is under $110 — cheaper than on-call for a gateway).
• You have no DevOps capacity and want zero infra to operate.
• You need the broadest model discovery without curating a provider list.

Pick self-hosted (LiteLLM, Mantis, Bifrost, Portkey OSS) when:

• You spend over ~$10K/mo on tokens or route 100M+ tokens/month (the flat container cost is 3–14x cheaper than the gateway fee).
• Data residency, prompt privacy, or key custody matters — vendor cannot see prompts.
• You already operate a PaaS or container platform where the gateway is just another tenant service.
• You want unified routing across managed providers and your own self-hosted models (vLLM/Ollama) behind one endpoint.

Hybrid is also real. Many teams run LiteLLM or Mantis as the primary gateway for cost and privacy, with OpenRouter as a fallback provider inside the self-hosted routing table. You get the flat container cost for the common path and the managed breadth for the tail. The gateway routes to OpenRouter the same way it routes to OpenAI — it is just another provider line in config.yaml.

The Broader Pattern​

LLM gateways are following the same arc as every middleware category that once looked like a billable service and turned out to be a config file: API gateways, job queues, object storage. The initial managed offering is valuable — it proves the pattern and carries teams that don't want to operate anything. Then the self-hosted implementation gets good enough that the managed tax is the main thing left to justify.

Mantis arriving as a single-command deploy is a signal that the self-hosted side has crossed that threshold for small teams. LiteLLM crossing it for large teams happened earlier. The pattern — model routing, fallback, and cost tracking as a tenant workload beside your apps, on infrastructure you own — is stable regardless of which gateway you pick.

The next time a gateway vendor emails about a new observability tier, check whether you already own the database it would log to.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.