Native Histograms Hit Kubernetes 1.37: The ~10x Prometheus Shrink Your Self-Hosted Fleet Pays For in Disk

A single apiserver_request_duration_seconds histogram with 40 buckets costs you 42 Prometheus series. One native histogram costs you one.

That 42-to-1 gap is not a rounding error. It is the reason the Kubernetes instrumentation docs promise a ~10x reduction in time series per histogram, why a Grafana demo shows 14x fewer samples per scrape and a 93% index reduction, and why a fleet that runs its own Prometheus on Hetzner local NVMe feels the difference in gigabytes and euros, not just in query graphs.

If you run a self-hosted PaaS on Cluster API and Hetzner, every histogram you scrape lives on a disk you bought. The next two tables tell you, before you touch a scrape config, what switching to native histograms actually saves — and why the savings depend on the two variables most posts hand-wave away: how many buckets your classic histograms have, and how many of them your fleet runs.

*le label index overhead: the Grafana demo measures 44 KB of label index per classic histogram that native histograms avoid entirely by encoding buckets inside a single protobuf sample.

†15s scrape interval, 30-day retention, ~1.5 bytes/sample classic vs ~6 bytes/sample native after TSDB compression, plus index. Native samples are larger per series but you have 10–42x fewer of them — disk and index both shrink. Numbers are order-of-magnitude for budgeting, not a benchmark on your data shape.

The pattern is simple: the more buckets you were paying for, and the more histograms you run, the closer you get to the headline 10x. A fleet heavy on Go prometheus.DefBuckets and kube-apiserver's 40-bucket latency histogram lands at the top of the range; a fleet with a handful of 8-bucket histograms still saves 90% on every histogram it touches. Either way the first dollar saved is on the series you no longer index.

Why Classic Histograms Are Expensive on Owned Hardware​

Classic Prometheus histograms are fixed-bucket histograms. You declare boundaries up front — le="0.1", le="0.2", le="0.5"… — and Prometheus stores each bucket as its own time series plus _sum and _count. An http_request_duration_seconds with 12 buckets is 14 series that share a metric name but diverge on le. Every rate() or histogram_quantile() you run has to fetch all 14, and every scrape has to index them.

That design made sense in 2015. It hurts in 2026 for three reasons that compound on a self-hosted fleet:

1. Bucket count multiplies, not adds. Kubernetes' own apiserver latency histogram ships with 40 buckets — amount of buckets for this histogram was increased to 40(!) in controller-runtime PR #1273, flagged in issue kubernetes/kubernetes#105346 as "enormous amount of time-series." At 40 buckets the single metric apiserver_request_duration_seconds with just 30 label combinations (5 verbs × 3 resources × 2 codes) already costs 30 × 42 = 1,260 series. On a 3-cluster fleet (management + 2 workload) you scrape that three times.

2. Label cardinality fans everything out. Histogram buckets multiply by every other label on the metric. A tenant service that adds handler="/api/users" and status="200" to its request-duration histogram does not add one histogram — it adds cardinality × (B+2) series. That is why the demo's 44 KB of le index overhead per histogram matters: it is 44 KB × cardinality, paid on every scrape.

3. You own the disk. A hyperscaler's managed Prometheus bill hides this as "ingested samples." On Hetzner you see it as prometheus_tsdb_head_series, as Thanos compactor CPU, and as the NVMe you chose for /var/lib/prometheus. The etcd post in this series showed why etcd belongs on local NVMe (p99 wal_fsync under 10 ms, Cloud Volumes at 8–40 ms). Prometheus is less latency-sensitive than etcd but more capacity-sensitive: retention is 30–90 days, not one WAL. A fleet that keeps 90 days of classic histograms on a 240 GB Hetzner Cloud Volume learns quickly that series you can delete at scrape time are cheaper than bytes you compact for three months.

The Grafana k8s-monitoring-helm PR #2586 states the fleet-level effect bluntly: Alloy v1.11's convert_classic_histograms_to_nhcb collapses each histogram from N+2 series into one, "reducing active series by 90%+ for histogram-heavy workloads like Temporal, Istio, or any app with high-cardinality classic histograms." That 90%+ is the same 10x in the other direction.

How Native Histograms Actually Work​

Native histograms — also called sparse or exponential histograms — remove the bucket guessing.

Instead of fixed le boundaries, they use exponential buckets that automatically adjust to the data distribution. Small values get fine-grained buckets; large values get coarser ones; the boundary set scales with the observed range rather than with a declaration you made when you instrumented the code. The result is a single series whose sample encodes spans and buckets as a compact protobuf, not as N distinct label values.

Three properties make them a different primitive, not just a smaller one:

One series, not N+2. A classic histogram's sample is a float. A native histogram's sample is a histogram object — schema, zero threshold, positive and negative spans — carried in one series. Cross-instance aggregation becomes a merge of histogram sketches rather than a fragile alignment of bucket boundaries that only works if every instance declared the same le set.

Higher precision at lower cost. The Grafana sparse-histograms prototype reported precision error dropping from 43% to 4.3% while index size fell 93%. The Temporal feature request that cites Björn Rabenstein's talk quotes the line directly: "you get 10x the resolution at half the price" — because exponential buckets put resolution where the data actually lives instead of wasting buckets where it does not.

Wire and storage efficient. le disappears from labels, which is where the 44 KB per-histogram index saving comes from. On the wire, histogram data is encoded once per scrape instead of once per bucket. On disk, TSDB stores one compressed sample instead of N.

Kubernetes and Prometheus meet in the middle. Kubernetes' instrumentation docs note native histograms need "Kubernetes v1.36 or later with the NativeHistograms feature gate enabled" and "Prometheus 2.40 or later." Prometheus stabilized scraping in the 3.x line: the --enable-feature=native-histograms flag is a no-op as of 3.9, and you enable native histograms per scrape config instead. Grafana Mimir, Alloy, and the OpenTelemetry Collector have followed with scrape_native_histograms and convert_classic_histograms_to_nhcb paths so you can adopt without re-instrumenting every app.

Worked Recompute for a Self-Hosted PaaS on Hetzner​

The tables at the top are the headline. This section is the ledger a platform team can paste into a sizing doc.

The fleet we are sizing​

• 3 Kubernetes clusters (1 management, 2 workload) on Hetzner Cloud, Cluster API with CAPH.
• 20 worker nodes (CPX31/CPX41 mix) plus 3 control-plane nodes.
• 100 tenant services, each exposing http_request_duration_seconds (12 buckets) plus one custom business histogram (10 buckets) — 200 histograms per scrape interval from tenants alone.
• Control-plane histograms: apiserver_request_duration_seconds (40 buckets), etcd_request_duration_seconds, workqueue_queue_duration_seconds, and controller-runtime's controller_runtime_reconcile_time_seconds (10 buckets) — ~40 more histograms with high cardinality.
• Instrumentation mix: Go client_golang v1.17+ (native-capable) for platform services, plus legacy tenant apps that still expose only classic buckets.
• Prometheus: one central Prometheus per cluster, 15s scrape interval, 30-day retention locally then Thanos Ship/Compact to Hetzner Object Storage or a second Hetzner volume. Some fleets run 90-day retention; we show both.

Step 1: Count histograms, not series​

This is the step most sizing posts skip. Count distinct histogram definitions (metric name + label set without le):

• Tenant histograms: 100 services × 2 histograms × ~5 label combinations average = 1,000 defs
• Platform histograms: apiserver (30 combos) + etcd (10) + controllers (40 × 5 avg) ≈ 240 defs
• Total for the 20-node example: ~1,240 histogram defs scraped per interval per cluster, × 3 clusters ≈ 3,720 defs fleet-wide. For the medium-fleet rollup we rounded to 12,500 defs including per-pod cardinality and Istio-style sidecars — a heavier but realistic histogram-heavy shape.

Step 2: Convert to series​

• Classic average B=12 → 14 series per def → 12,500 defs × 14 = 175,000 series
• Native → 1 series per def → 12,500 series
• Saving: 162,500 series, or 92.9% of histogram series.

If your histograms average B=8, the saving is 90%; if you are heavy on apiserver's B=40, a slice of the fleet saves 97.6%. The 10x headline is the average, not the best case.

Step 3: Convert series to disk​

TSDB compression is content-dependent, but two numbers dominate budgeting:

• Classic sample: ~1–2 bytes after XOR/Delta-of-Delta compression. Native sample: larger (spans + buckets) but still a single sample — order of 30–60 bytes uncompressed, ~5–8 bytes compressed.
• Index: classic pays le per bucket; native pays none. At 44 KB per histogram definition in the demo, 12,500 defs × 44 KB ≈ 550 MB of index you stop writing.

For 30-day retention at 15s scrape:

• Samples per series = 30 × 86,400 / 15 = 172,800
• Classic storage ≈ 175,000 × 172,800 × 1.5 bytes ≈ 45 GB for histogram samples alone, plus index.
• Native storage ≈ 12,500 × 172,800 × 6 bytes ≈ 13 GB, plus minimal index.
• Net: ~32 GB saved on histogram samples, ~0.5 GB on index, per month of retention. Triple it for 90-day retention.

The overall Prometheus footprint does not shrink 10x — non-histogram metrics (counters, gauges) are unaffected — but histogram-heavy fleets routinely report 60–70% histogram storage reduction, which translates to 25–40% total TSDB reduction when histograms are 30% of series. For the medium fleet's 86 GB → 56 GB total, that is 30 GB you do not provision, compact, or snapshot. On a Hetzner CPX31 with a 240 GB local NVMe, that is the difference between one volume and two, or between 30-day and 90-day retention on the same disk.

Query latency moves with it. histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)) on classic fetches 14 series per handler and does a quantile over le. The native equivalent — histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m])) by (handler)) or the newer hist_quantile helpers — fetches one series per handler and quantiles over the sketch. Fewer series means less PromQL fan-out, less querier memory, and less Thanos Store Gateway chatter for the same SLO.

Sensitivity: what changes the answer​

The saving is not a single number you memorize. It moves with:

• Bucket count: The driver. A fleet that standardized on 8 buckets saves 10x per histogram; one that inherited 40-bucket apiserver defaults saves 42x on that metric. Audit with count by (__name__) ({__name__=~".+_bucket"}) to find your heaviest.
• Cardinality: More label values → more defs → more multiples of the saving. Tenant services that label by user_id or route with 100+ values are histogram-cardinality bombs even with few buckets.
• Retention: Longer retention multiplies the GB delta linearly. If you keep 90 days locally, the 30-day saving above triples.
• Native sample size: High-resolution histograms (smallest bucket width) produce larger native samples. Even then you win on index and series head overhead — Prometheus holds ~1.5 KB per series in head — so 162,500 series × 1.5 KB ≈ 244 MB of head memory you never allocate.

If you run Istio, Linkerd, or Temporal, treat this as the high end of the range: their histograms are both high-bucket and high-cardinality, exactly the shape Alloy's "90%+ reduction" note targets.

Migrating Without Re-Instrumenting Every Service​

You do not need to recompile tenants to get most of the win. Prometheus can convert classic histograms to native histograms with custom buckets (NHCB) at scrape time.

1. Upgrade Prometheus and enable native scraping​

Prometheus 3.9 is the pivot: the daemon flag disappears and scrape config takes over.

scrape_configs:
  - job_name: "kubernetes-apiservers"
    scrape_protocols: ["PrometheusProto"]
    scrape_native_histograms: true
    convert_classic_histograms_to_nhcb: true
    # Keep classic during migration so old dashboards still work
    always_scrape_classic_histograms: false
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        action: keep
        regex: default;kubernetes

• scrape_native_histograms: true — detect and scrape native histograms over PrometheusProto; ignore the classic exposition for metrics that have native defined.
• convert_classic_histograms_to_nhcb: true — for apps that still expose only classic buckets (most tenant code), synthesize a native histogram with custom buckets at scrape time. This is the "collapse N+2 → 1 without re-instrumenting" switch Alloy v1.11 also exposes as convert_classic_histograms_to_nhcb on prometheus.scrape.
• always_scrape_classic_histograms: true — dual-write both forms during migration. Leave false unless you need a dashboards transition window; it doubles ingestion for the transition.
• scrape_protocols: ["PrometheusProto"] — native histograms have no text exposition; the scrape must negotiate protobuf.

2. Enable the Kubernetes feature gate​

On the components you want native histograms from:

• kube-apiserver --feature-gates=NativeHistograms=true
• kubelet --feature-gates=NativeHistograms=true
• kube-scheduler, kube-controller-manager as needed

With Cluster API this is a KubeadmControlPlane or KubeadmConfigTemplate patch, not a hand-edit:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          feature-gates: NativeHistograms=true

Roll the control plane, then verify apiserver_request_duration_seconds no longer exposes _bucket when scraped with PrometheusProto.

3. Migrate PromQL, dashboards, and alerts​

Classic:

histogram_quantile(0.99,
  sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le, verb)
)

Native (exponential) or NHCB:

histogram_quantile(0.99,
  sum(rate(apiserver_request_duration_seconds[5m])) by (verb)
)
# or the newer helpers where available:
# hist_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])) by (verb))

Checklist:

• Replace *_bucket, *_sum, *_count groupings with the single metric name.
• Drop le from by clauses — it no longer exists.
• Update recording rules that pre-aggregate buckets.
• Keep a short dual-write period if you have many tenant dashboards: set always_scrape_classic_histograms: true for one retention window (e.g., 7 days of 30) while teams migrate, then flip it off and watch prometheus_tsdb_head_series drop.
• The OpenTelemetry Collector's prometheusremotewrite exporter and Grafana Mimir both now handle NHCB dual-write; validate against your Thanos/Mimir version before relying on it for long-term retention.

4. Watch the right signals​

• prometheus_tsdb_head_series — should fall by roughly the histogram series count you calculated above.
• prometheus_tsdb_head_samples_appended_total — sample ingestion rate should drop even as resolution rises.
• prometheus_build_info / prometheus_config_last_reload_successful — confirm the 3.9 config landed.
• apiserver_request_duration_seconds cardinality via count by (__name__) ({__name__="apiserver_request_duration_seconds"}) — should be 1 per label combination after migration, not 42.
• Wallet watch: node_filesystem_avail_bytes on the Prometheus data mount — the GB you modeled above should be visible within one compaction cycle.

Rollback is a config revert: flip scrape_native_histograms and convert_classic_histograms_to_nhcb off, reload, and classic buckets resume. No Tenant redeploy needed because conversion happened at scrape, not at exposition.

When Native Histograms Do Not Help Yet​

Native histograms are stable in Prometheus 3.9's scrape path, but the ecosystem around them is still catching up:

• You must be on Prometheus 3.x. 2.50 can scrape native histograms experimentally, but the stable knobs (scrape_native_histograms, NHCB) and the flag removal land in 3.9. If you are still on prometheus-operator chart 65 with Prometheus 2.x, upgrade first.
• Remote-write and long-term storage must understand them. Thanos, Mimir, and Cortex have native-histogram support in recent releases; older Store Gateways and queriers will not compact or query them correctly. Check your Thanos version against the Prometheus compatibility matrix before enabling NHCB for retention.
• Custom bucket shapes can lose fidelity on conversion. NHCB preserves classic bucket boundaries as custom buckets, so a histogram with pathologically chosen buckets (e.g., all buckets clustered at one end of the distribution) converts faithfully but does not gain exponential resolution. The fix is still to adopt native exposition at the source — Go client_golang with native histograms enabled — not just conversion.
• No text exposition. curl /metrics will not show native histograms as text; you will see them only over PrometheusProto. That breaks naive grep _bucket checks and any exporter test that asserts on text output.
• Dashboard migration is real work. If tenants have not adopted the new PromQL, dual-write (always_scrape_classic_histograms: true) is a bridge, not a destination. Budget the dashboard PRs before you claim the disk.

For a self-hosted PaaS that already runs Prometheus 3.x and controls its kube-apiserver flags through Cluster API, those are surmountable. For a fleet still on Prometheus 2.x or on a managed long-term store you do not control, they are reasons to wait one more minor train rather than force a half-migration.

What This Means for a PaaS That Owns Its Disks​

The self-hosted fleet's cost story is rarely about one clever optimization. It is about a series of line items a hosted platform bundles and you pay explicitly: this post's histograms, the prior post's etcd fsync (1–3 ms on local NVMe vs 8–40 ms on a Cloud Volume), the controller-runtime cache that copies every watched object into head memory, and the egress you either meter per gigabyte or include in a Hetzner flat.

Native histograms move one of those line items from "grows with every bucket you declared" to "one series per distribution." On a 20-node, 100-service fleet that is 30 GB per month of retention you do not provision; on a 50-node fleet it is 70 GB. The mechanism — exponential buckets, single protobuf sample, no le index — also gives you sharper p99s for the same reason it saves disk: resolution follows the data.

Kubernetes 1.37 does not make this automatic. You still enable the feature gate, still negotiate PrometheusProto, still migrate dashboards. But the conversion path that matters for a platform team is the one that requires no tenant action: convert_classic_histograms_to_nhcb at the scrape. Flip it on, watch head series fall, cut the dashboards over on your own schedule, and keep the GB.

A self-hosted PaaS wins not by finding one 10x, but by stacking five of them where a hosted bill would have hidden each one.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.