Dora NodaFive vendors that compete on everything just agreed on one thing: how to serve an LLM on Kubernetes. On March 24, 2026, at KubeCon Europe in Amsterdam, IBM Research, Red Hat, and Google Cloud donated llm-d — their Kubernetes-native distributed inference framework — to the CNCF as a Sandbox project, with NVIDIA and CoreWeave as founding collaborators and AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI signed on.
Here is the verdict up front for teams self-hosting AI-agent workloads: llm-d is the emerging open standard for running inference as declarative Kubernetes primitives instead of bespoke GPU-serving glue. Its routing layer is worth piloting now; full prefill/decode disaggregation is worth tracking but not yet worth betting a production critical path on. And Sandbox status is neutral governance, not a production certification — more on exactly what that does and doesn't promise below.
What llm-d actually is
Strip away the launch announcements and llm-d is four techniques composed into one stack, each mapped onto a Kubernetes primitive you already understand.
Inference-aware routing. A plain round-robin load balancer is a bad way to spread LLM traffic: two identical prompts landing on different replicas mean two cold KV caches and twice the prefill compute. llm-d is the reference implementation of the Kubernetes Gateway API Inference Extension (GAIE). Its Endpoint Picker (EPP) scores model-server pods on live signals — queue depth, cached prefix overlap, predicted latency — and routes each request through an Envoy-based proxy via ext-proc to the replica most likely to already hold the prompt's prefix in cache. Since v0.7 the proxy-plus-EPP combination is called the llm-d Router, and the InferencePool custom resource declares which pods it considers.
Disaggregated serving. Prefill (digesting the prompt) is compute-bound; decode (emitting tokens one by one) is memory-bandwidth-bound. Running both phases on the same GPU forces one resource to idle while the other saturates. llm-d splits them into independently scalable pod pools — separate LeaderWorkerSets for prefillers and decoders — and ships the KV cache between them (NIXL for the transfer, LMCache-style handling in the ecosystem). Scale the pool that's actually saturated.
Hierarchical KV-cache management. The KV cache is the working set of inference, and llm-d treats it as a tiered resource: GPU memory for the hot set, spilling across TPU, CPU, and storage tiers rather than forcing a binary fit-or-evict choice. Mistral AI is contributing a DisaggregatedSet operator for LeaderWorkerSet aimed at exactly this class of problem for mixture-of-experts models.
Native orchestration. Multi-node replicas and wide expert parallelism become LeaderWorkerSets and Gateway API routes — reconcilable, GitOps-able objects — instead of shell scripts that SSH into GPU boxes. The vision the founders state is "any model, any accelerator, any cloud," and the project works across vLLM (the default engine), SGLang, and TensorRT-LLM, on GPUs and TPUs.
KServe's own framing captures where llm-d sits: "If KServe is the control plane for models, llm-d is the distributed intelligence scheduling layer." KServe's LLMInferenceService gives you lifecycle, an OpenAI-compatible endpoint, and autoscaling down to zero; llm-d supplies the smart routing, disaggregation, and cache-aware scheduling underneath it.
Why agent sandboxes are llm-d's best-fit tenant
A generic chatbot backend benefits from llm-d. But the workload shape of AI-agent sandboxes — the thing a self-hosted PaaS actually runs — maps onto llm-d's strengths almost suspiciously well. Consider the request path:
agent sandbox → Gateway → EPP → prefill pool / decode pool → response
(llm-d Router) (vLLM model servers)Three properties of agent traffic make this topology pay off disproportionately.
Repeated prefixes. A multi-step agent loop resends a growing conversation — system prompt, tool definitions, tool results, then the next question — on every iteration. That is the ideal input for prefix-cache-aware routing: the EPP keeps landing follow-up calls on the replica already holding the shared prefix, skipping redundant prefill on the longest part of every request. Round-robin would scatter those calls and recompute the same prefix N times.
Bursty, spiky concurrency. Agents don't stream evenly; they think in bursts — ten tool calls in thirty seconds, then silence. llm-d's load- and latency-aware scoring spreads bursts across warm replicas instead of queueing behind one busy pod, and KServe underneath can scale the pool to zero when the fleet goes quiet at 3am. Compare that with a per-token hosted API meter that charges full freight for every call regardless of your fleet's idle capacity.
OpenAI-compatible drop-in. The endpoint speaks /v1/chat/completions with streaming, so pointing an existing agent stack at a self-hosted llm-d service is a base-URL change, not a rewrite. That keeps the migration reversible: proxy to a hosted API today, flip the base URL to your own cluster when the numbers justify it, flip back if they stop.
| Layer | Owner | What it gives the self-hoster |
|---|---|---|
| Model lifecycle, scale-to-zero | KServe (LLMInferenceService) | Declarative deploys, idle costs near zero |
| Smart routing, prefix-cache hits | llm-d Router (EPP + proxy) | Fewer recomputed prefixes under bursty agent loops |
| Prefill/decode at independent scale | llm-d disaggregation | Right-size compute-bound vs bandwidth-bound pools |
| Token generation | vLLM / SGLang / TRT-LLM | PagedAttention-class engine efficiency |
The money question, priced honestly
Self-hosting inference has a floor and a ceiling, and llm-d moves the ceiling more than the floor. Here is the break-even math as a band, not a cherry-picked point.
The floor. A Hetzner GPU box (the GEX44 with an RTX 4000 SFF Ada, 20 GB) lists around €234/month — call it roughly $300/month all-in for one self-hosted inference node once you count the server, power share, and a slice of ops attention. That is the single-pool floor: one GPU node serving everything. Full prefill/decode disaggregation needs at least two GPU pools to disaggregate across, so the honest llm-d-disaggregated floor is roughly double: ~$600/month before serving a single token.
The other side. Hosted API prices as of this month span roughly $1–$5 per million tokens blended for cheap Haiku-class models to around $9 blended for Sonnet-class ($3 input / $15 output since September 1). Break-even monthly volume is simply floor ÷ API price:
- Single-node floor ($300/mo) displacing Sonnet-class API (~$9/M): ~33M tokens/month, or ~1M tokens/day.
- Same floor displacing cheap API (~$1.25/M blended): ~240M tokens/month, or ~8M tokens/day.
- Disaggregated floor ($600/mo): double both — ~2M to ~16M tokens/day.
The sensitivity that actually decides it: utilization. Those thresholds assume the GPU is productively busy. An agent fleet averaging 30% utilization triples the effective cost per served token, pushing the band to ~3M–25M+ tokens/day — and independent analyses put fully-loaded self-hosting at 3–5× the raw GPU price once DevOps time, power, and downtime enter the ledger.
Scale-to-zero (KServe) and bin-packing bursty tenants onto shared pools (llm-d's routing) are precisely the mechanisms that claw utilization back up — but only if your token volume is steady enough that the hardware isn't mostly idle.
The honest read: below ~1M steady tokens/day, the hosted API wins for almost everyone. Above ~10M/day of cache-friendly agent traffic on owned hardware, self-hosting with llm-d's routing starts looking compelling. Between those lines, it depends on your utilization curve and how you value data locality — which no break-even table can decide for you.
How early is Sandbox, really?
CNCF Sandbox is the entry tier: experimental projects, not yet widely tested in production. It buys llm-d neutral Linux Foundation governance — no single vendor can quietly relicense the routing layer your fleet depends on — but it certifies nothing about stability, security audit, or production readiness. Incubating, the next rung, requires demonstrated production users and healthy contributor diversity. llm-d isn't there yet.
The release train, though, is moving fast for a project launched in May 2025:
- v0.7 (May 2026) renamed the core to llm-d Router, took predicted-latency scheduling to GA, made deployment kustomize-first, expanded nightly CI across OpenShift, GKE, and CoreWeave, and marked the batch gateway experimental.
- v0.8 docs are published with Envoy AI Gateway deployment guides and a stabilized artifact set (core EPP image plus optional tokenizer and routing sidecars).
- Sharp edges remain. The v0.7 line requires NVIDIA driver 580+ (CUDA 13.0.2); the batch gateway is experimental; disaggregated P/D is the newest, least-battle-tested surface.
A pragmatic adoption ladder for a self-hosted platform:
- Adopt now: the concepts are free — run vLLM behind KServe's
LLMInferenceServicefor an OpenAI-compatible, scale-to-zero endpoint. That stack is production-grade independent of llm-d. - Pilot now: the llm-d Router (EPP + GAIE
InferencePool) in front of it for prefix-cache-aware routing. This is the highest value-per-risk slice, and it speaks standard Gateway API. - Wait and track: full prefill/decode disaggregation and hierarchical cache tiers on your own critical path until the project matures toward Incubating — while watching neighbors like NVIDIA Dynamo (datacenter-scale disaggregated serving) and AIBrix (ByteDance's vLLM control plane) to see which interfaces converge.
One more signal worth tracking: the same "Kubernetes-native primitives instead of vendor platforms" pattern is playing out one layer up, with Kubernetes' own Agent Sandbox work standardizing how agents execute. The inference layer underneath them standardizing too means a self-hosted PaaS can eventually offer agents-plus-models as one coherent declarative stack.
The bottom line
llm-d's CNCF landing matters less as a news event than as a coordination event: the vendors most likely to fragment inference serving instead agreed on shared plumbing — GAIE routing, LeaderWorkerSet-orchestrated pools, disaggregated prefill/decode. For a team self-hosting AI agents on owned hardware, that turns "run our own models" from a bespoke GPU-ops project into an incremental adoption curve: KServe endpoint today, llm-d routing when prefix-cache wins justify it, disaggregation when the project — and your token volume — earns it.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
