OpenCost 1.121 Brings Inference Cost Tracking to Kubernetes: Metering GPU Token Workloads Like Any Other Tenant

What does a million tokens actually cost when you serve them yourself? Depending on your hardware and — far more importantly — your utilization, the honest answer spans more than an order of magnitude:

Until this summer, that table was something you built once in a spreadsheet, argued about, and never verified. OpenCost — the CNCF incubating project that became the open-source standard for Kubernetes cost allocation — changed that in July 2026, when v1.121.0 shipped "AI Inference Costs v1" (PR #3845). One version-number caveat up front: the feature landed in 1.121.0 on July 20, but a usage-cost-basis bug in the initial release was fixed in v1.121.1 on August 5, 2026 (PR #3901) — so 1.121.1 is the release to actually run.

The significance is bigger than a changelog line. For the first time, the same open-source tool that tells you what the payments namespace spends on CPU and memory can meter what the inference namespace spends per token — on machines you own. This post walks through why token workloads broke the old cost model, decomposes the rent-vs-own math the table above summarizes, and shows how to wire vLLM's token metrics and OpenCost into a multi-tenant showback report.

Why Token Workloads Broke Kubernetes Cost Allocation​

The OpenCost specification allocates workload cost with a simple, defensible rule: a container's cost for each resource is max(request, usage) multiplied by the resource's unit price. GPUs were folded into this model the same way CPU and RAM were — a workload that requests one GPU is billed for one GPU-hour per hour, with node pricing normalized so CPU, RAM, and GPU costs sum to the node's actual price.

That rule works beautifully for web services. It fails in two specific ways for LLM inference:

• GPU allocation is all-or-nothing in time. A vLLM pod requests the whole GPU and holds it 24/7, whether it's serving 2,000 tokens per second at peak or sitting idle at 3 a.m. GPU-hour showback bills the tenant identically in both cases, so it can't tell you whether the workload is worth the hardware.
• Tokens are the unit everyone else reasons in. Your product team compares against API list prices quoted in dollars per million tokens. Your finance team asks how self-hosting compares to the OpenAI or Together invoice. GPU-hours don't answer that question; tokens served per dollar of GPU cost does.

The gap between those two units — GPU-hours allocated versus tokens actually produced — is precisely the utilization sensitivity in the opening table. It's also exactly what OpenCost's inference cost feature exists to measure: v1 follows a usage-based cost basis, tying inference spend to what workloads actually consume rather than only what they reserve. That's why the usage-basis bug fix in 1.121.1 mattered enough to ship two weeks after the feature itself.

This lands at the right moment. Self-hosters have spent 2025 and 2026 moving vLLM and open-weight models onto the same clusters that run their web apps — the whole point of owning a Kubernetes fleet is that a GPU node is just another node. But "just another node" needs to become "just another line in the cost report," and that required the metering to catch up.

The Rent-vs-Own Math, Actually Decomposed​

Let's rebuild the opening table from parts, because the parts are where the arguments live.

Assumptions (stated so you can swap in your own): a 70B-class open-weight model served with vLLM in FP8, sustaining roughly 1,000 aggregate output tokens/second on a 96GB RTX PRO 6000 Blackwell under continuous batching, and roughly 2,000 tokens/second on an H100 (whose ~3.35 TB/s HBM bandwidth roughly doubles decode throughput over GDDR7). Real throughput varies with context length, batch depth, and quantization — treat these as round, defensible midpoints, not benchmarks.

Path 1: Hardware you actually own​

A purchased RTX PRO 6000 Blackwell workstation card runs about $9,000 street in mid-2026. Straight-line over three years of 8,760 hours each:

• Depreciation: $9,000 ÷ 26,280 hours ≈ $0.34/hour
• Power: ~450W system draw under load × $0.15/kWh ≈ $0.07/hour
• Host amortization: a ~$3,000 chassis over the same three years ≈ $0.11/hour

Total: ~$0.52/hour, excluding rack space, networking, and your own ops time — real costs, but ones you're likely already paying for the cluster the GPU joins. At 1,000 tokens/second, an hour at full tilt produces 3.6M tokens: $0.14 per million tokens at 100% utilization, $0.21 at 70%, $0.36 at 40%, $1.44 at 10%.

Path 2: A leased dedicated GPU box​

Hetzner's GEX131 puts the same RTX PRO 6000 Blackwell Max-Q (96GB) in a dedicated server at €889/month. That's €1.22/hour, about $1.41/hour — no capex, no procurement, cancel monthly. Same throughput assumption: $0.39 per million tokens at 100% utilization, $0.56 at 70%, $0.98 at 40%, $3.92 at 10%.

Path 3: Rented H100s​

The H100 rental market in August 2026 spans $1.89/hr (UpCloud) to $6.98/hr (Azure), averaging about $3.64/hr — a spread worth noticing on its own, since the same silicon costs 3.7x more depending on the logo on the invoice. At the $3.64 average and 2,000 tokens/second, an always-on H100 delivers $0.51 per million tokens at 100% utilization, $0.72 at 70%, $1.26 at 40%, $5.06 at 10%. At the $1.99 RunPod-class low end, halve those numbers.

Path 4: Just call an API​

Serverless providers list 70B-class open-weight models at roughly $0.70–$0.90 per million output tokens in mid-2026 (frontier closed models run an order of magnitude higher). The API price has one property no self-hosted path shares: it is flat. Your utilization is the provider's problem.

The crossover is a utilization number, not an opinion​

Read the columns together and the rent-vs-own argument collapses into a single measurable threshold. The Hetzner lease beats the API list price only above roughly 40% sustained utilization. The purchased card beats it above roughly 10%. Below those lines, the API is cheaper and someone else carries the pager.

Which means every self-hosting cost argument — including the ones this blog makes — rests on a number most teams have never measured: what fraction of the GPU-hours you pay for turn into tokens somebody wanted. That's not a spreadsheet input. That's a metering problem.

Wiring It Up: vLLM → OpenCost → a Showback Report​

Here's the concrete path from "vLLM pods on a GPU node" to "inference costs sitting next to CPU and RAM in the monthly tenant report."

1. Expose and scrape token metrics. vLLM's OpenAI-compatible server ships Prometheus counters out of the box — the ones that matter here are vllm:prompt_tokens_total and vllm:generation_tokens_total, alongside latency and cache-usage gauges. Point a ServiceMonitor (or plain scrape config) at the vLLM pods so these land in the same Prometheus instance OpenCost already reads. No sidecar, no exporter to write.

2. Tell OpenCost what your hardware really costs. On-prem and dedicated-server clusters don't have a cloud billing API, so OpenCost falls back to custom pricing you configure. Set the GPU node's price from your actual invoice — the €889/month GEX131 figure, or your depreciation-plus-power schedule for purchased hardware. This step is what makes the output an audit rather than an estimate: garbage node prices in, garbage $/M-token out.

3. Run v1.121.1 or later with inference costs enabled. The feature arrived as "AI Inference Costs v1" in 1.121.0 (PR #3845); the usage-cost-basis fix in 1.121.1 (PR #3901) corrects the calculation methodology, so treat 1.121.1 as the effective minimum version. Note also that recent releases deactivate some endpoints unless an admin token is configured — plan for that in a multi-tenant setup where the cost API itself needs access control.

4. Query the allocation API by namespace. OpenCost's allocation API already aggregates cost by container, pod, deployment, label, and namespace. With inference metering active, a per-namespace query gives each tenant a row with CPU, RAM, GPU, and inference-usage costs side by side — the "inference as just another column" report. Export it to Grafana or CSV exactly as you would the ordinary compute showback.

5. Divide to get the number that settles arguments. Measured tokens per namespace (from the vLLM counters) over allocated GPU cost per namespace (from OpenCost) yields realized dollars per million tokens, per tenant, from production data. If the agents namespace realizes $2.80/M tokens against an API list price of $0.85, its GPU is underutilized and you can prove it — then consolidate models, add batch traffic, or shrink the fleet, and watch the realized number move next month.

One honest caveat: this is a v1, weeks old, and its documentation is thinner than the mature allocation paths. Expect sharp edges, pin versions, and validate the first month's report against raw Prometheus queries before anyone bills a team on it.

Inference as Just Another Tenant​

Step back and the pattern is familiar. Kubernetes absorbed stateful workloads, then batch jobs, then GPUs — each time, the ecosystem's answer was to make the new thing a regular tenant: scheduled by the same scheduler, secured by the same RBAC, and eventually metered by the same cost model. OpenCost 1.121 is that last step landing for inference.

For a self-hosted platform, this matters more than any single feature. A platform that claims "run your models next to your apps and save money" owes its operators the instrumentation to check the claim — per-tenant, from live data, in the same report as everything else. The alternative is what most teams have today: a rent-vs-own spreadsheet from last quarter, built on a utilization guess nobody has validated since.

This is also squarely an agent-era concern. As AI agents become operators that deploy services and provision capacity, machine-readable cost state stops being a nice-to-have — an agent deciding whether to schedule a model on the owned GPU or fall back to an API needs exactly the realized $/M-token feed this pipeline produces. Platforms built for agent operations, Bex.co among them, treat inference as a first-class tenant workload on machines you own for precisely this reason: the rent-vs-own math should be a query, not a belief.

The Metering Becomes the Argument​

The interesting thing about OpenCost 1.121 isn't the feature — it's what the feature makes falsifiable. Every fight about self-hosted inference eventually reduces to the utilization threshold in the table above, and until now both sides argued from assumptions. Open-source, per-token, per-tenant metering turns the threshold into a dashboard panel.

Expect the rest of the stack to follow. Cost visibility standards have a way of becoming procurement requirements — showback for CPU and RAM did — and per-token showback will likely be table stakes for platform tooling within a year. The teams that come out ahead are the ones already measuring: they'll know their crossover number before the invoice, or the vendor negotiation, forces the question.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• OpenCost releases — v1.121.0 (July 20, 2026) and v1.121.1 (August 5, 2026)
• OpenCost v1.121.0 release notes — AI Inference Costs v1 (PR #3845), usage cost basis fix (PR #3901)
• OpenCost specification — allocation model and GPU pricing normalization
• OpenCost — CNCF incubating project overview
• Hetzner dedicated GPU server line — GEX131 (RTX PRO 6000 Blackwell)
• Cloud GPU price comparison — NVIDIA H100 rental market
• vLLM production metrics documentation