Self-Hosted vLLM on Kubernetes: What Owning Your Inference Layer Actually Costs

A 27B-class open model on your own GPU breaks even against Claude Sonnet 5 at roughly 10 to 15 million tokens per day, and against Haiku at 25 to 45 million. Below that, the GPU sits idle and the engineering time never pays back. That is the verdict worth stating before anything else, because everything about self-hosted inference looks cheap until you price the two things the demo never shows: a GPU that bills by the month whether or not anyone prompts it, and the operations load of keeping a serving stack healthy at 3am.

The concrete starting point for this post is CNCF's July 16, 2026 walkthrough by LINBIT's Matt Kereczman, which documents a self-hosted vLLM stack on Kubernetes backed by replicated LINSTOR storage and reachable through an OpenAI-compatible API. It is a good lab. This post is about the distance between that lab and a production inference layer — what the lab's three-resource pattern gets right, the four things production adds, the breakeven math, and the five cases where self-hosting wins anyway.

Bottom line: the breakeven is not a property of the model or the engine. It is a property of your utilization curve. A GPU is a flat monthly rate spread over however many tokens you actually push through it, so the same box is either the cheapest inference on earth at 80% utilization or the most expensive at 5%. Every section below earns that sentence from a different direction. (Figures from René Zander's September 2026 break-even analysis, cross-checked against SquareOps' 2026 FinOps guidance, which lands on the same 10–15M tokens/day crossover.)

What the CNCF lab actually builds​

The walkthrough's architecture is three Kubernetes resources: a PersistentVolumeClaim for model storage, a Secret holding the Hugging Face token, and a Deployment plus Service running the inference server. That is the whole stack, and its smallness is the point — self-hosted inference on Kubernetes does not require a platform team to start, only to continue.

The one design decision worth stealing is where the model weights live. The Deployment mounts the PVC at /root/.cache/huggingface, which is exactly where the vLLM container caches downloaded weights. The model downloads from Hugging Face once — roughly 2.5 GB for the lab's Llama-3.2-1B-Instruct — and every later pod restart finds the weights already on disk because they persist on replicated storage. Scale the Deployment to zero and the cached weights survive; scale back up and there is no re-download. Model weights are the only state a stateless inference server has, and the lab puts that state in the one place Kubernetes already knows how to keep alive.

Storage choice does real work here. The lab uses LINSTOR through the Piraeus Operator and its CSI driver, with a thin-provisioned LVM class carrying two replicas across the cluster. Two properties matter for weights: thin provisioning, because model files consume tens of gigabytes you do not want to over-allocate per replica; and DRBD-backed replication, because a volume holding a 70 GB weight file cannot be a single point of failure the way a scratch disk can. Any CSI driver with replication would serve the pattern — the pattern is "weights on replicated block storage behind the framework's cache path," not the vendor.

Two more lab details deserve attention precisely because they look like footnotes. First, gated Hugging Face models need an access token stored as a Secret before anything downloads — weight acquisition is a credentialed step in every pipeline, including yours, and it belongs in sealed secrets management rather than a manifest someone pastes.

Second, the lab pins --gpu-memory-utilization to 0.80 after vLLM's aggressive default reservation caused startup failures in a constrained environment. That flag governs how much memory the engine pre-allocates for the KV cache, and it is the first knob every production deployment touches: too high and the engine will not start, too low and concurrent throughput collapses. (For the full story on why that cache is the throughput bottleneck, this blog previously traced vLLM's PagedAttention curve against Ollama — 793 vs 41 tokens per second at high concurrency on identical hardware, converging to near-parity at one concurrent user.)

The lab's limits are stated honestly: a 1B model, CPU-only nodes, a single replica. Production starts where those limits end.

The four things production adds that the lab skips​

Compute is the largest gap and the least interesting technically: GPU nodes need drivers, device plugins, and taints/tolerations so only inference lands there, plus a node-provisioning story for adding the second GPU machine. (The scheduling mechanics — resource fields, node affinity, admission — got the full worked treatment in this blog's earlier vLLM-on-git-push piece; they are necessary and no longer the hard part.) The operational bite is driver and CUDA version drift across the pool, which fails silently as degraded throughput rather than loudly as a crash.

One compute question deserves its own line because it moves the money math: device sharing. The breakeven table assumes one tenant workload per card, which is also the simplest thing to operate — but it strands capacity whenever that workload idles. MIG slices partition supported cards in hardware, HAMi offers fractional-GPU sharing as CNCF-incubating middleware, and Kubernetes 1.36 keeps maturing Dynamic Resource Allocation's claim-based model for requesting slices of specialized hardware through core scheduling primitives. None of these is "turn it on" yet for a small fleet — each is an additional ops surface with its own failure modes — but sharing is the lever that lifts a card from 30% toward the 70%+ utilization where self-hosting wins, so track DRA's graduation the way you'd track any dependency your unit economics rely on.

Weights at production scale change character. A 1B model's 2.5 GB download finishes over coffee; a 70B model's weights are an 80–140 GB artifact that must exist on the node before the pod is useful, or every scale-up event starts with a twenty-minute download. Production weight management means versioned artifacts, pre-warming on GPU nodes, and rollouts sequenced so the new weights are resident before the new engine starts. The lab's cache-the-download pattern scales to this fine — it just needs to be deliberate rather than incidental.

Lifecycle is where "point a pod at a GPU" visibly breaks. vLLM loads weights at startup, so a rolling upgrade is a minutes-long readiness gap per replica, not a seconds-long one; run too few replicas and the rollout is downtime with extra steps. Scale-to-zero still works — the lab's insight that the PVC survives scaling holds — but the cold start on scale-up is a model-load, not a container-start, and somebody's first request of the morning pays it. Liveness probes need to distinguish "engine loading weights" from "engine wedged," because the default probe configuration cannot tell them apart and will happily restart a pod that was ninety seconds from ready.

The request path is the subtlest gap. Each replica carries gigabytes of KV-cache state that a round-robin proxy cannot see, so naive load balancing sends new requests to the replica closest to eviction. At real concurrency this is where throughput goes to die, and it is why the ecosystem is building cache-aware routing — the Gateway API Inference Extension and queue-aware schedulers exist for exactly this problem. You do not need them on day one. You need to know their absence is the ceiling on naive scaling, so the day latency degrades under load you recognize the shape.

The money math, with utilization doing the driving​

Take the table from the intro and add the two costs it omits. First, engineering: budget 10 to 20 percent of a senior engineer for inference operations — driver upgrades, weight rollouts, the 3am page when the engine wedges. Zander's analysis includes this line item and most back-of-napkin math does not, which is most of why back-of-napkin math favors self-hosting. Second, utilization, which cuts harder than any other variable.

A GEX131-class box at ~€1,199/month costs the same at 5% utilization as at 95%. At 10M tokens per day — the low end of the Sonnet breakeven — the box serves ~300M tokens a month, putting effective hardware cost near $4 per million tokens before engineering time, roughly at parity with a Sonnet-class blended rate. Halve the volume and the effective rate doubles past the API with nothing to show for it; double the volume and self-hosting pulls decisively ahead.

The breakeven band of 10–15M tokens/day is really a statement about keeping one card busy: one 27B–70B-class model, served steadily, saturates roughly one large GPU's worth of concurrent throughput. Spiky traffic — ten idle hours and two crush hours — pays the flat monthly rate for the crush capacity and amortizes it over the idle troughs, which is the worst of both worlds and the most common real-world shape.

Hetzner's June 2026 GPU price increase sharpens the point: the 20 GB entry box moved from ~€184 to ~€234/month, and the GEX131 sits at ~€1,199 plus setup. These are still a fraction of hyperscaler GPU rents — an H100 at $2,100–3,100/month is two to three GEX131s — but they reset any breakeven computed against older pricing. Recompute before you commit, and recompute against the API prices that moved the other way: frontier-class hosted rates have generally fallen while owned-hardware rates just rose.

The honest summary fits in one line: self-hosting converts a per-token variable cost into a flat monthly fixed cost plus a permanent ops retainer, and that trade wins exactly when volume is high, steady, and predictable.

When self-hosting wins anyway, and the hybrid middle​

Zander's verdict — APIs win for roughly 95% of production workloads in 2026 — leaves five cases where owning the layer is the right call, and they are worth listing whole because most teams hold exactly one of them and mistake it for three:

• Very high steady volume. Past ~50M tokens/day the math tilts; past ~100M it is clearly ahead, provided an open model meets your quality bar.
• Strict data residency. "Data must never leave our network" is narrower than "data must stay in the EU" — Bedrock and Vertex regional endpoints plus zero-retention arrangements cover the latter at ~10% premium. Only the former forces self-hosting.
• Fine-tuned owned weights. Weights trained on your data, redeployable anywhere, are self-hosting by definition.
• Cost-sensitive bulk at high utilization. Classification, extraction, embeddings, summarization — predictable load that keeps a card above ~70% around the clock.
• Models with no hosted equivalent. Small specialized weights, frozen legacy versions, domain-tuned Hugging Face models nobody serves.

For everyone else there is the hybrid the CNCF piece gestures at: serve high-volume or sensitive workloads locally, route the rest to a managed API, and let a router split by cost, latency, or capability. The open-source llm-d project is building exactly that request router for Kubernetes. This is also the shape a platform team should default to — one self-hosted model for the workload that justifies it, managed APIs for everything else — because it concentrates the ops burden (drivers, weights, pages) onto the single workload whose volume pays for it.

Note what never appears in the win column: latency control as an abstract goal. Self-hosting can beat API latency for co-located workloads, but "we wanted lower P99" without a volume or residency story is how teams end up operating a GPU fleet to serve 200K tokens a day. Measure the requirement first; the P99 of a well-chosen managed endpoint embarrasses a poorly-utilized self-hosted card.

Owning your inference layer, priced honestly, is three lines: the GPU bill (flat, monthly, utilization-sensitive), the weights pipeline (storage, versioning, pre-warming — the lab pattern grown up), and the ops retainer (a slice of a senior engineer, forever). The CNCF lab shows the shape for the cost of an afternoon. The breakeven table shows when the shape pays. If your workload clears 10M steady tokens a day on an open model, or your data cannot leave the building, the lab is the first page of a genuinely good plan. Otherwise the cheapest GPU is the one you never rent — and the best inference architecture is a router pointed mostly at somebody else's fleet.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.