OpenCost 1.121 Finally Answers: What Does Each Token Cost on Your Own GPUs?

Your GPU bill is rising. Your models serve billions of tokens. Yet one question has no answer: what does each token actually cost?

That was the opening line of the CNCF's August 5, 2026 OpenCost 1.121.0 announcement, and it lands because every team running LLMs on Kubernetes has lived it. You know what the node costs per hour. You know how many tokens vLLM reports. But the dollar-per-token number — the one your finance team asks for, the one that decides whether you keep renting an API or own the hardware — has been a spreadsheet guess stitched between two systems that never talked to each other.

OpenCost 1.121 is the first release that makes those two systems talk natively.

Built with llm-d, the CNCF sandbox for distributed LLM inference on Kubernetes, the release ships Kubernetes-native inference cost tracking as Prometheus metrics and REST APIs: allocation cost versus usage cost, input versus output tokens, KV-cache-corrected. For the first time, cost-per-token is a first-class platform metric you can alert on — not a quarter-end autopsy.

This post wires up what 1.121 actually does, runs the numbers for a real self-hosted fleet, and shows why transparent per-token metering is the piece that makes "own the hardware" a defensible budget conversation.

The Missing Metric: Why Your GPU Bill Hides the Token Cost​

OpenCost existed before — as the CNCF incubating project's Kubernetes cost-allocation engine. It already knew how to turn node prices into per-namespace, per-pod, per-label costs by multiplying observed CPU, GPU, memory, and storage usage by configurable unit prices. What it never knew was what those GPUs were doing.

Meanwhile vLLM knew exactly what they were doing — how many prompt tokens and generation tokens each replica processed — but had no idea what the GPUs underneath cost.

The gap mattered because the gap was most of the money. Datadog's State of Cloud Costs 2024 found 83% of container costs were associated with idle resources — 54% cluster idle (over-provisioned infrastructure) plus 29% workload idle (requests larger than actual use). Containers already consume 35% of EC2 compute, up from 30% a year earlier. For GPU-backed inference, the idle fraction is worse: a model replica reserves an entire GPU whether it serves 10 requests per minute or 1,000. The difference between "the GPU exists" and "the GPU is producing tokens" is the entire economics of self-hosting — and until 1.121, no single metric expressed it.

The announcement put it plainly: an enterprise's cost for SaaS inference is the price on the API pricing page. An enterprise's cost for self-hosted inference is an hourly GPU rental plus a throughput number someone has to join manually. 1.121 does the join inside the platform.

How It Works: Two Kinds of Cost per Token​

OpenCost 1.121 does not reinvent token counting. It reuses what an llm-d deployment already emits and multiplies it by what OpenCost already knows.

From those four, 1.121 publishes a new family of inference cost metrics to Prometheus and via its REST API, with two complementary perspectives.

Allocation cost: "What does availability cost?"​

Allocation cost divides the full GPU-hour price — idle time included — by tokens served. If a replica with one NVIDIA L4 at $0.60/hr serves 1 million tokens in an hour, allocation cost is $0.60 per million. If it serves 200,000 tokens in the same hour because traffic is light, allocation cost is $3.00 per million. Same hardware, same price, very different efficiency — and that variance is the insight.

Usage cost: "What did active inference actually burn?"​

Usage cost strips out idle reservation and counts only the GPU time consumed during real forward passes, then corrects for KV-cache hits. A request whose prefix was already in the KV cache does not re-encode those tokens, so charging it as if it did overstates cost. 1.121's usage metric accounts for cache hits, giving a lower-bound "marginal token" price that should converge toward the hardware's theoretical best-case.

The gap between the two is the story:

• Allocation − usage ≈ idle tax. A large gap says your GPUs are reserved but not producing tokens — scale down replicas, batch more, or share the node.
• Usage itself ≈ hardware efficiency. If even usage cost is high, the model or quantization is the bottleneck, not scheduling.

The integration also separates input and output token costs using processing-time weighting. Input tokens are typically cheaper per token than output tokens because prefill is parallel and decode is sequential — the metrics reflect that rather than smearing one average.

vLLM users who do not use llm-d can also benefit: the core token-throughput counters come from vLLM directly, so the same cost math applies without the full llm-d control plane.

Wiring It Into Your Cluster in Three Steps​

You need three things: an instrumented inference stack, OpenCost, and a place to query.

1. Deploy llm-d with vLLM​

llm-d is a Kubernetes-native disaggregated serving layer on top of vLLM. Even a single-replica install gives you the standard token counters. The exact Helm shape evolves, but the contract is stable: vLLM exporters expose vllm:prompt_tokens_total and vllm:generation_tokens_total, and llm-d's scheduler exports the processing-time signals OpenCost joins on.

2. Install OpenCost 1.121+ with GPU allocation​

OpenCost's Helm chart already supports custom pricing for on-prem or Hetzner-equivalent hardware. Set the GPU hourly price you actually pay — cloud instance price, bare-metal amortization, or spot — and OpenCost turns it into per-pod allocation.

# opencost-values.yaml — bare-metal L4 amortized to ~$0.60/hr as an example
opencost:
  exporter:
    defaultClusterId: "hetzner-fsn1"
  pricing:
    # Fallback hourly price for GPU nodes labelled accelerator=nvidia-l4
    customPricing:
      enabled: true
      provider: custom
      description: "Amortized L4 on owned hardware"
      CPU: "0.02"        # per vCPU-hour, example
      RAM: "0.002"       # per GiB-hour
      GPU: "0.60"        # per GPU-hour — the only line token math keys off
      storage: "0.0001"

helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm upgrade --install opencost opencost/opencost \
  -n opencost --create-namespace \
  -f opencost-values.yaml --version ">=1.121.0"

For managed GPU nodes, leave GPU unset and let the cloud-pricing integration (AWS/GCP/Azure APIs) populate it. The inference metrics work either way.

3. Query It​

Two new metric families appear in Prometheus (names follow the opencost:* convention from the release; confirm in your scrape config if you vendor-prefix):

# Allocation cost per million input tokens, by model
sum by (model) (
  opencost_inference_allocation_cost
) / sum by (model) (
  vllm:prompt_tokens_total
) * 1e6
 
# Usage cost per million output tokens, KV-cache-corrected
sum by (model) (
  opencost_inference_usage_cost
) / sum by (model) (
  vllm:generation_tokens_total
) * 1e6

Via the REST API, the same data is available without PromQL:

kubectl port-forward -n opencost svc/opencost 9003:9003 &
curl "http://localhost:9003/allocation/compute?window=1h&aggregate=pod" | jq .
# Response includes inferenceCost, inferenceTokens, costPerToken for each pod

Point Grafana at either surface. The release ships a sample dashboard that plots allocation vs. usage per model over time — the visual gap is your idle tax in real time.

Worked Example: Dollars per Million Tokens on Owned Hardware​

Numbers make the abstract concrete. Take a single-replica Llama 3.1 8B deployment on one L4, priced at an amortized $0.60 per GPU-hour — a plausible figure for owned Hetzner-adjacent hardware or a reserved cloud instance after the 2024-2026 H100 spot correction. Assume the replica sustains 2,000 tokens per second at full tilt (mixed prompt + generation — conservative for an 8B quantized model).

How much a million tokens costs depends almost entirely on one variable: what fraction of the hour the GPU is actually generating tokens.

How to read this:

• Allocation cost is what you actually pay per token after idle time is included. At 20% utilization you pay 4× more per token than at 80% — same hardware, same hourly price, wildly different economics. This is the 83% idle phenomenon expressed in token terms.
• Usage cost — $0.08 ($0.04 with cache) — is the floor. It does not move with utilization because it excludes idle time. The closer allocation gets to usage, the better your fleet is packed.
• KV-cache hits cut usage cost roughly in half in this toy model. In a real RAG or agentic loop with repeated system prompts, Datadog-style tracing will show the usage line dropping as hit rate rises — free proof that prompt-caching is not just a latency win but a dollar win.

The complete picture for a given hour is therefore three numbers, not one: allocation per million, usage per million, and the ratio between them.

Now swap the hardware price to bound sensitivity:

At owned-hardware pricing, even poorly utilized (20%) inference undercuts most API flagships on raw token price. At on-demand cloud GPU pricing, you need sustained high utilization to win — exactly the break-even analysis the next section quantifies.

What This Changes versus an Opaque Cloud Bill​

A SaaS API price is simple: OpenAI GPT-4o lists roughly $2.50 input / $10.00 output per million tokens, Anthropic Claude 3.5 Sonnet roughly $3.00 / $15.00 (check current pricing pages — they move). What you see is what you pay, and the vendor's idle GPU fleet is their problem.

A self-hosted bill used to be the opposite: the GPU price is simple, the token price is opaque, and your idle fleet is your problem — but invisible. You could compute a quarterly average after the fact, not a per-model, per-hour signal you can act on.

OpenCost 1.121 closes that asymmetry.

Put the worked example back next to API pricing:

Rule of thumb from practitioner reports: self-hosted inference typically breaks even against API pricing when sustained utilization exceeds 30-40% of GPU capacity, roughly equivalent to 50,000 tokens per minute averaged over the billing period — but until 1.121 that break-even was asserted, not measured. Now you can compute it per model, per namespace, per day:

# Break-even: allocation cost vs. hypothetical API price (example: $5/M blended)
(
  sum(opencost_inference_allocation_cost) / sum(vllm:prompt_tokens_total + vllm:generation_tokens_total) * 1e6
) < 5

When that expression flips from false to true and stays there, you have a data-driven case to own the hardware. When it flips back, you have an honest signal to burst to an API.

Making Build-vs-Buy a Budget Conversation​

The real product of 1.121 is not the metric — it is the conversation the metric enables. Platform teams can finally answer three questions finance actually asks, without opening a spreadsheet:

1. Which model costs what? Allocation-per-token by model label lets you compare Llama 8B vs. 70B vs. a fine-tuned variant on the same hardware. The larger model may be 3× better on quality and 4× worse on cost per token — now you can say so with a graph.

2. Are we idle or inefficient? If allocation is far above usage, you have a scheduling problem (too many replicas, too little batching). If usage itself is high, you have a model problem (quantize, distill, or swap the base model). The two need different fixes, and conflating them is why "just add GPUs" sometimes makes cost per token worse.

3. What is the marginal token worth? Usage cost with KV-cache correction is the price of one more token given the fleet you already run. If you are building an agent loop that makes 20 tool calls per user request, that marginal price — not the average allocation — is what determines whether the feature is economically viable.

A minimal operational loop looks like this:

• Dashboard: allocation vs. usage per model, input vs. output, over 1h and 24h windows.
• Alert: fire when allocation / usage > 3 for more than 30 minutes — the replica is reserved but not producing.
• Action: Cluster Autoscaler or KEDA scales replicas; or llm-d's scheduler packs more requests per GPU.

# Example PrometheusRule — idle tax is too high
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: inference-idle-tax
spec:
  groups:
    - name: inference-cost
      rules:
        - alert: InferenceIdleTaxHigh
          expr: |
            (
              sum by (model) (opencost_inference_allocation_cost)
              /
              sum by (model) (opencost_inference_usage_cost)
            ) > 3
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "Model {{ $labels.model }} allocation is 3x usage — scale down or batch up"

For a Cluster-API-provisioned fleet on Hetzner or equivalent, this closes a loop the PaaS pricing debate has been missing. Every 2026 price hike — Hetzner's third, Vercel's fourth, Fly.io's second batch of new meters — is an argument about whose bill is more predictable. Predictability without metering is just a flat number you hope is low enough. Metering without predictability is a bill that surprises you. What 1.121 offers is predictable metering on hardware you already own: a cost curve that moves with demand and that you can see move.

The Bottom Line​

Before OpenCost 1.121, "cost per token on my own GPUs" was a phrase, not a metric. You could estimate it, debate it, and put it in a slide — but you could not graph it, alert on it, or show finance a per-model, per-hour number backed by the same Prometheus every other platform signal already flows through.

After 1.121, you can. Two numbers — what availability costs and what active inference burns — plus the gap between them, split by input and output and corrected for the KV cache that makes real workloads cheaper than benchmarks suggest. All from counters vLLM already exposes, joined to GPU prices OpenCost already tracks, via an integration llm-d already provides.

That is not a niche FinOps feature. It is the missing piece that turns "run inference on owned hardware" from a hunch about hourly GPU prices into a budget line a team can defend — and the signal that tells them when the API is the cheaper call after all.

Your GPU bill is rising. Your models serve billions of tokens. Now you can finally say what each token costs.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Bring your own GPU nodes or your managed Postgres, and let the platform's Render-compatible API and MCP server give agents the same deploy-and-observe loop your team already uses. Star the repo on GitHub or deploy your first app today.