Skip to main content

Kubernetes 1.36's PSI Metrics Graduate to GA: What Pressure Stall Information Catches About Resource Starvation That CPU% Utilization Misses

8 min readDora NodaDora Noda
Share
On this page

A tenant's dashboard shows 22% CPU utilization. Support tickets are still coming in about a crawling app. Every standard alert stayed green through the entire incident, because CPU utilization was never the thing that broke — the container spent most of that window queued behind a neighbor's disk I/O, and utilization has no way to say so.

That gap is exactly what Pressure Stall Information (PSI) measures, and as of Kubernetes 1.36, it's no longer a beta feature you have to opt into. KEP-4205 graduated the KubeletPSI feature gate to GA and locked it to true — it can't be disabled anymore. The kubelet now collects per-container container_pressure_cpu_*, container_pressure_memory_*, and container_pressure_io_* metrics from the Linux kernel and exposes them at both /metrics/cadvisor (Prometheus format) and /stats/summary, at the node, pod, and container level. Each resource reports some (at least one task stalled) and full (every non-idle task stalled) pressure, each as moving averages over 10, 60, and 300 seconds, plus a cumulative total in stall-microseconds.

The distinction that makes this useful: utilization measures how much of a resource a workload consumed. PSI measures how much time a workload spent waiting for a resource it needed and didn't get. A container can be at 90% CPU utilization and perfectly healthy — it's happily using what it asked for — or at 20% utilization and badly starved, because it keeps getting descheduled before it can do anything with the CPU it's nominally consuming. Utilization can't tell those two states apart. PSI's entire job is telling them apart.


What Actually Shipped in 1.36

Kubernetes 1.34Kubernetes 1.36
Feature gateKubeletPSI, beta, opt-inKubeletPSI, GA, locked to true — cannot be turned off
Exposure/metrics/cadvisor, /stats/summarySame, now a stable API
Metricscontainer_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_totalUnchanged metric names, GA stability guarantee
GranularityNode, pod, containerNode, pod, container
Underlying requirementLinux kernel ≥4.20, CONFIG_PSI=y, cgroup v2Same — GA didn't relax the kernel dependency

Nothing about the metric shape changed between beta and GA — this release is entirely about API stability and the gate being locked on. But locking it on matters operationally: any fleet running 1.36 now has PSI collection running on every node whether or not anyone downstream is scraping it, which is the detail that makes the rest of this post relevant even to teams who haven't built a single PSI-based alert yet.


The Worked Example: Same 60 Seconds, Two Different Stories

Here's the scenario the metric exists for — a noisy neighbor hammering disk I/O on a shared bare-metal node, and a second tenant's container paying for it. Same 60-second window, same node, two dashboards:

SignalWhat it readsWhat it means
Container CPU utilization (standard dashboard)22%Looks idle. No alert fires.
container_pressure_io_some_avg1068%The container's tasks were runnable-but-blocked on I/O for 68% of the last 10 seconds.
Request latency (app-level)p99 up 9xThe actual symptom the customer feels.

The CPU graph and the I/O pressure graph are describing the same container at the same moment, and they disagree completely, because they're measuring different things. The container isn't using the CPU it has because it's stuck behind the kernel's I/O scheduler waiting on a disk a neighboring tenant's batch job is saturating. A CPU- or memory-utilization alert has no way to catch this — the container's utilization numbers are, accurately, low. io.some.avg10 is the only one of the two signals that reflects what the tenant is actually experiencing, because it measures stall time directly instead of inferring health from consumption.

This is the case PSI was built for, and it's also the ordinary case for any platform packing multiple tenants' workloads onto shared hardware — not a contrived edge case, but the default failure mode of noisy-neighbor contention on owned bare metal.


Two Ways PSI Can Mislead You If You Trust It Blindly

PSI is a better signal than utilization, but it has two sharp edges worth knowing before wiring alerts to it.

Zero doesn't always mean zero. Kubernetes issue #136333 documented kubelets on 1.34+ emitting container_pressure_cpu_waiting_seconds_total and its siblings at a flat zero on hosts where the underlying kernel didn't actually have PSI enabled — the code checked whether the KubeletPSI feature gate was on, not whether the kernel underneath it supported PSI at all. The result: a Grafana panel showing "zero pressure, all quiet" that actually meant "not measured, no data," on hosts that had never once collected a real PSI sample. The fix landed in PR #137326, but the underlying lesson generalizes past that one bug — a metric that silently reports zero when it can't collect is indistinguishable, on a dashboard, from a metric reporting zero because there's genuinely no contention.

Self-inflicted pressure looks identical to a noisy neighbor. PSI currently can't distinguish contention caused by a competing tenant from contention caused by the container's own configured limits. A container running a compute-heavy workload under a deliberately tight 20m CPU limit will show close to 100% CPU pressure with zero other tenants anywhere near it — the container is, correctly, stalled waiting for CPU, but the thing it's waiting on is a quota the platform itself imposed, not a neighbor's workload. Reading that as "this node has a noisy-neighbor problem" without checking the container's own limits first is a fast way to chase phantom contention.

Neither edge case is a reason to skip PSI. They're the reason it needs verification, not blind trust, which is the next section.


The Pre-Flight Checklist for a Cluster API / Bare-Metal Fleet

Because a self-hosted platform owns the machine image, not a managed node pool someone else patched for you, "PSI is GA" doesn't mean "PSI is measuring anything on your fleet." Confirm these on the base image before a single alert depends on the signal:

  1. Kernel version ≥ 4.20. PSI support was added in that release; anything older has no PSI accounting at all, gate or no gate.
  2. CONFIG_PSI=y at kernel build time. Not universal — check with zgrep CONFIG_PSI /proc/config.gz or the equivalent for the image's boot config.
  3. cgroup v2 mounted. PSI accounting for containers rides on cgroup v2; a v1 hierarchy gets no per-cgroup PSI data regardless of kernel support.
  4. psi=1 boot parameter, if the kernel ships CONFIG_PSI_DEFAULT_DISABLED. Some distributions compile PSI in but leave it off until a boot flag turns it on — check cat /proc/cmdline for psi=1, or confirm the default isn't set.
  5. Non-zero values under real load — not just that the metric exists. After the above are confirmed, run a deliberate contention test (a stress-ng --io job against another pod on the same node, say) and check /proc/pressure/io directly on the host, then confirm the same spike shows up in container_pressure_io_some_avg10 via /metrics/cadvisor. A metric existing and a metric measuring anything real are different claims — issue #136333 is the proof that step matters.

That last step is the one worth not skipping. Every step before it can pass and the fleet can still be looking at #136333's exact failure mode until someone forces contention and checks for a real number on the other end.


Wiring It Into Autoscaling and Noisy-Neighbor Alerting

Once the checklist passes, the concrete change is a new alert and autoscale input alongside the existing CPU/memory rules — not a replacement for them:

  • Noisy-neighbor alert: fire when a tenant's container_pressure_io_some_avg10 or container_pressure_memory_some_avg10 crosses a threshold (a starting point most operators land on is 50-60% sustained over the avg60 window) while that tenant's own CPU/memory limits aren't already saturated — the second half of that condition is what filters out the self-inflicted-pressure false positive from the section above.
  • Autoscale trigger: add io.some.avg10 and cpu.some.avg10 as a Prometheus Adapter custom metric feeding a tenant's HPA, so a workload queued-but-not-consuming triggers a scale event that a resource-utilization-only HPA would miss entirely, since its utilization numbers stay low right up until the stall clears.
  • Per-node bin-packing input: on a Cluster API fleet scheduling tenant workloads across owned Hetzner nodes, full pressure (not some) on a node is the signal that node is actually saturated — some pressure is common and often fine; sustained full pressure means every non-idle task on that node is stalled simultaneously, which is the point at which the scheduler should stop placing new tenants there.

None of this requires new infrastructure beyond what the checklist already confirmed — it's the same /metrics/cadvisor endpoint most Prometheus setups already scrape, with three more metric families in the query.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, on a Cluster API fleet where noisy-neighbor detection can key off the same PSI signal this post walks through instead of a utilization graph that stays green through the incident. Star the repo on GitHub or deploy your first app today.


Sources

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex