Skip to main content
bex.co
419
Blog

Your MCP Traffic and Your Tenants' LLM Calls Just Got a Real Kubernetes Primitive

9 min readDora NodaDora Noda
Infrastructure
Self-Hosting
PaaS
AI agents
Model Context Protocol
Share
On this page

On March 9, 2026, SIG Network announced a new AI Gateway Working Group whose entire job is to standardize something almost every AI-adjacent platform has already hand-rolled at least once: token-based rate limiting, model-aware routing, and provider failover, all as first-class Gateway API primitives instead of a bespoke proxy layer bolted onto Envoy. If you're running a platform that pushes both tenant app traffic and your own agent-facing MCP calls through the same cluster ingress, that's not a committee footnote — it's a primitive you'd otherwise have to build yourself.

What's already real, and what's still a proposal​

It's worth separating the parts of this story that ship today from the parts that are still GitHub issues, because the two get conflated constantly in AI-infra roundups.

Already GA, already usable: the Gateway API Inference Extension — the project that spent 2025 as an incubating SIG Network effort — reached general availability and shipped two CRDs any cluster operator can install today: InferencePool, which groups a set of model-serving pods the way a Service groups regular pods, and InferenceModel, which lets whoever owns a model declare its routing weight and priority independent of how the pool is scaled. Sitting in front of both is an "endpoint picker" — an Envoy ext_proc extension that reads live signals from each model server (KV-cache utilization, queue depth, active LoRA adapters) and routes each request to whichever replica can actually serve it fastest. In published benchmarks this model-aware routing measurably cuts tail latency once traffic passes roughly 400–500 QPS against GPU-backed backends — the regime where round-robin routing starts sending requests to replicas that are already saturated. Multiple gateways implement it today: Envoy Gateway, kgateway, GKE Gateway, NGINX Gateway Fabric, and Alibaba Cloud ACK all ship it as an extension point, not a fork.

Just concluded, its job handed off: WG Serving, the working group that incubated the Inference Extension, formally wound down on February 26, 2026. Its own announcement is blunt about why: "The goal of this working group was to ensure that Kubernetes is an orchestration platform of choice for inference workloads. This goal has been accomplished." The Inference Extension moved to permanent SIG Network ownership; autoscaling and fast-bootstrap requirements went to SIG Node/Scheduling; multi-node orchestration went to SIG Apps and the device-management working group.

Brand new, pre-CRD: the AI Gateway Working Group is what SIG Network stood up eleven days later to pick up where Inference Extension's scope stops — traffic shaping and access control, not just model-aware routing. Its charter names two active proposals. Payload Processing standardizes inspecting and transforming full LLM request/response bodies for prompt-injection defense, response content filtering, and semantic caching — work that today means shipping a custom ext_proc filter per gateway vendor. Egress Gateways standardizes routing outbound traffic to external model providers — OpenAI, Vertex AI, Bedrock — with managed credential injection, regional compliance routing, and multi-provider failover as declared config, not application code. Neither proposal has a merged CRD yet. The group meets weekly on Thursdays.

The upshot: if your rate-limiting or model-routing need maps onto InferencePool/InferenceModel, you can adopt it now. If it maps onto token-quota enforcement or provider failover, you're watching a working group, not shipping against a stable API.

The part every platform is still hand-rolling: token-based rate limiting​

Model-aware routing (which pod serves the request) is the part Kubernetes just solved. Token-based rate limiting (how much of a given model a tenant is allowed to consume) is the part almost nobody has, and it's worth being concrete about why the DIY version degrades instead of just "not scaling gracefully."

The typical hand-rolled setup looks like this: a piece of application middleware sits in front of the LLM call, reads the provider's token-usage field off the response, and increments a per-tenant counter in Redis with a TTL that resets the window. It's maybe 80 lines of code, and at low volume — one Redis instance, one gateway pod, a handful of tenants making occasional calls — it works fine. The trouble starts as soon as you scale it in either of two directions:

  • More gateway replicas. Every replica issues its own INCR against the same Redis key. Under real concurrency you get a classic check-then-act race: two replicas both read "9 of 10 tokens used" before either writes back, and both let a request through that pushes the tenant to 12. The fix (Lua scripts, WATCH/MULTI, or a rate-limiting-specific store) is well understood, but it's infrastructure you're now maintaining alongside the app.
  • More concurrent requests at a quota boundary. When a tenant's window resets, every queued or retried request that was waiting on the old quota fires at once — a thundering herd against Redis and against the tenant's own downstream call, exactly when you most want smooth backpressure instead of a burst.

None of this is exotic — it's the same class of problem any distributed counter has — but it's work a platform team pays for in on-call pages, not in a line item. Compare that to what Envoy AI Gateway — one of the concrete implementations the WG's proposals are formalizing — already does with its Global Rate Limit API: token usage is extracted from response metadata natively, the counter lives in the gateway's own distributed rate-limit service (built for exactly this consistency problem, not repurposed from a cache), and circuit breaking against a specific model backend is a declared policy, not a try/except block. The honest comparison isn't "hand-rolled is bad and standardized is good" — it's that hand-rolled is the right call for a single tenant behind a single gateway pod, and stops being the right call the moment you add a second gateway replica or a quota-reset burst, which for any platform serving more than a handful of tenants is not a hypothetical.

Two traffic classes, one ingress layer​

This is where it stops being an abstract Kubernetes-ecosystem story and becomes a design question for a platform like Bex — one where AI agents aren't just a feature tenants build, they're a first-class way of operating the platform itself.

A git-push PaaS with agent-facing MCP tooling has, structurally, two categories of LLM-shaped traffic moving through the same cluster:

  1. Tenant app traffic. A tenant's own application calls out to an LLM provider — a chatbot feature, a summarization endpoint, an agent it built on top of the platform. The platform doesn't own this traffic's content, but it does own the network path it takes out of the cluster.
  2. The platform's own agent traffic. When an AI agent deploys or operates an app through Bex's MCP surface — "redeploy this service," "roll back the last release," "scale this up" — that's an inbound MCP request hitting the same Gateway API ingress, and it may itself trigger outbound calls to a model provider to reason about what to do next.

Both classes want the same three things this working group is standardizing: rate limiting so one noisy tenant (or one runaway agent loop) can't starve everyone else's quota, model-aware routing so requests land on a healthy backend, and provider failover so an OpenAI outage doesn't take down every tenant's AI feature at once. Concretely, that third one is what the Egress Gateway proposal is aimed at: today, routing a call to OpenAI first and falling back to Bedrock if it times out or rate-limits is application code — a wrapped SDK call with a try/except and a second client configured for the fallback provider, credentials for both baked into the app's environment. Under the Egress Gateway model, that becomes a declared backend list on a Gateway route: primary OpenAI, fallback Bedrock, credential injection and regional routing handled by the gateway, with the tenant's app code making one call to one endpoint and never knowing a failover happened. That's a meaningful reduction in what every tenant currently has to reimplement — and it's also, not coincidentally, the same primitive a platform would use to make its own agent-facing MCP traffic provider-resilient.

What to actually adopt now, and what to keep watching​

Given the split between what's GA and what's a charter with two open proposals, the concrete call for a self-hosted, Cluster-API-based platform in mid-2026 is:

  • Adopt now: if you're running any GPU-backed model-serving pods behind Kubernetes, install the Gateway API Inference Extension. InferencePool and InferenceModel are stable CRDs with multiple production gateway implementations, and the latency win at 400+ QPS is real, not speculative.
  • Prototype, don't productionize: the Payload Processing and Egress Gateway proposals describe exactly the shape of primitive a platform routing both tenant and agent traffic needs, but neither has a merged CRD as of this writing. Building against Envoy AI Gateway's existing Global Rate Limit API today is reasonable — it's a working implementation the WG's own proposals are drawing from — but treat it as a vendor-specific stopgap you'll likely swap for the standardized version once the working group ships one, not a permanent architecture decision.
  • Keep hand-rolling exactly where it's still cheaper: a single-tenant or low-concurrency rate limit genuinely doesn't need a distributed rate-limiting service yet. The line to watch for is the second gateway replica or the first quota-reset burst — that's the point where the DIY counter's failure mode stops being theoretical.

The bigger signal here isn't any one CRD — it's that "how does a git-push platform meter and route LLM traffic for its tenants and its own agents" just went from "every platform invents its own answer" to "a SIG Network working group is actively formalizing the answer." That's a faster standardization path than most infrastructure primitives get, and it's worth tracking the weekly meeting notes rather than rediscovering the same rate-limiter race condition a second time.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with AI agents as first-class operators through an MCP-native deploy surface. Star the repo on GitHub or deploy your first app today.

Sources​

Related articles

Give your agents a chain backend

Autonomous agents hit RPC endpoints very differently than people do. See what bex router handles on their behalf.

Read the agents guide
Back to all posts