Skip to main content
bex.co
419
Blog

Kubernetes 1.36's CRI List Streaming Doesn't Cut Kubelet Memory — It Removes a Hard Container-Count Wall Bin-Packed Nodes Actually Hit

8 min readDora NodaDora Noda
Kubernetes
Self-Hosting
PaaS
Share
On this page

Every recap of Kubernetes 1.36's alpha features repeats the same line about CRI List Streaming: it "cuts kubelet memory" on high-density nodes. The KEP's own text says the opposite. Quoting KEP-5825 directly: "Streaming does not reduce kubelet memory usage... the full result set is still held in memory. Streaming solely addresses the gRPC per-message size limit, not memory pressure."

So what does it actually fix? A number, not a curve: once a node accumulates roughly 11,000 containers (~16.5 MB) or 14,000 pod sandboxes (~16.8 MB), the kubelet's ListContainers and ListPodSandbox calls to the container runtime blow past gRPC's 16 MiB default message limit and fail outright — not "run slower," fail. That failure has already bricked at least one production node for an hour. This is the real story, and it's a better one than the memory-savings headline: a hard ceiling that a bin-packed, fixed-hardware node can walk straight into, with no autoscaler standing by to bail it out.

What CRI List Streaming actually does​

The Container Runtime Interface's List* RPCs — ListContainers, ListPodSandbox, ListImages, ListContainerStats, ListPodSandboxStats, ListPodSandboxMetrics — are unary today. The kubelet asks, the container runtime (containerd, CRI-O) builds the entire result set, and sends it back as one gRPC message. gRPC caps a message at 16 MiB by default. At roughly 1.5 KB per container and 1.2 KB per pod sandbox in the response encoding, that ceiling arrives at about 11,000 containers or 14,000 pod sandboxes on a single node — numbers KEP-5825 states directly, not estimates this post is making up.

KEP-5825, gated behind CRIListStreaming, adds six streaming equivalents — StreamContainers, StreamPodSandboxes, StreamContainerStats, StreamPodSandboxStats, StreamPodSandboxMetrics, StreamImages — that let the runtime open a gRPC stream and send results incrementally instead of building one giant response. It shipped Alpha in Kubernetes v1.36 (released April 22, 2026, codenamed Haru). Backward compatibility is symmetric: an old kubelet talking to a new runtime still gets unary responses, and a new kubelet talking to an old runtime detects an UNIMPLEMENTED status and falls back to unary automatically. Nobody's forced onto the new path until both sides support it — the KEP's graduation plan calls for Beta once both containerd and CRI-O implement streaming, and GA once every supported runtime does.

What streaming buys is exactly one thing: results no longer have to fit in a single 16 MiB envelope, so the call itself stops failing at scale. What it does not buy — per the KEP's explicit non-goal — is a smaller in-memory footprint. The kubelet's own list-handling code collects every streamed chunk into the same full in-memory list before it does anything with it. The bytes on the wire get chunked; the bytes in the kubelet process do not.

The failure this fixes is already happening today​

This isn't a hypothetical scaling problem someone extrapolated for a blog post. It's an open, already-diagnosed GitHub issue: kubernetes/kubernetes#131407, "kubelet not cleaning up exited containers and eventually failing due to gRPC message size."

The reporter's cluster ran 100+ CronJobs on one-minute schedules, each configured with what looks like defensible cleanup hygiene: concurrencyPolicy: Forbid, ttlSecondsAfterFinished: 30, successfulJobsHistoryLimit set to 1 or 2, failedJobsHistoryLimit: 1. That's not a misconfigured cluster — it's a team that read the CronJob docs and set sane limits. It didn't matter. The kubelet_running_containers{container_state="exited"} metric crossed 10,000 exited containers within eight hours, because kubelet's own garbage collector (MaxPerPodContainer defaults to 1, MaxContainers defaults to -1, meaning unbounded unless an operator sets it) couldn't keep pace with the churn rate.

Once ListPodSandbox's response exceeded gRPC's 16 MiB ceiling, the failure cascaded fast:

  • The kubelet's pod-lifecycle event generator started logging "ListPodSandbox with filter from runtime service failed" on every reconciliation pass.
  • Cilium's IPAM reported "range is full" — the kubelet couldn't get an accurate picture of what was actually running, so IP addresses tied to dead sandboxes never got released, and new pods sat Pending.
  • The kubelet itself stopped responding to commands.
  • The node only recovered after being cordoned and drained — it took about an hour for garbage collection to work through the backlog and bring exited-container count from over 10,000 down to 6.

That's the concrete failure mode CRI List Streaming targets: not "memory pressure," a hard RPC failure that takes the node offline for an hour with pods stuck in limbo, triggered by ordinary CronJob churn, not an edge-case pathological workload.

Why a fixed Hetzner box hits this wall differently than an autoscaling fleet​

On a cloud autoscaling group, a node quietly approaching this ceiling is (mostly) someone else's problem to absorb: the cluster autoscaler notices pressure, provisions a fresh node, and workloads reschedule while the sick node gets cordoned and eventually recycled. The failure is real but it's a blip, not an outage, because there's always another node one API call away.

A Cluster-API-managed fleet running on owned Hetzner hardware doesn't get that safety net. A node going NotReady because ListPodSandbox started throwing ResourceExhausted isn't "capacity moved elsewhere" — it's capacity gone, full stop, until a human (or an automation someone built) cordons, drains, and waits out garbage collection, or replaces the machine. There's no metered API call that conjures a 48-core Hetzner AX162 into existence in ninety seconds the way a cloud autoscaler summons a fresh VM.

And a multi-tenant PaaS node is exactly the shape of workload that reaches 10,000+ exited containers faster than the ceiling's headline number suggests, because the count that matters isn't live pod count — it's container churn, dead ones included. A single Hetzner box bin-packing dozens of tenant apps, each with its own build jobs, deploy jobs, health-check sidecars, and restart-on-crash containers, generates the same kind of relentless short-lived-container turnover that tripped the 100-CronJob cluster in issue #131407 — except it's coming from git-push deploys and CI runs instead of scheduled jobs, and it's continuous rather than occasional. A platform that treats "just add another node" as the default answer to density problems doesn't have that answer available when the nodes are owned, not rented.

What actually changes once it graduates — and what doesn't​

Once CRIListStreaming reaches Beta (both containerd and CRI-O shipping it, gate on by default) and eventually GA, the specific failure in issue #131407 goes away: ListPodSandbox and ListContainers calls stop hard-failing at the 16 MiB wall, because the runtime streams results across as many messages as it needs instead of building one. A node that piles up 15,000 or 50,000 exited containers because garbage collection is falling behind won't watch its CRI calls start throwing ResourceExhausted — the calls will simply take longer and use more memory, which is a degradation you can see coming on a dashboard, not a cliff you fall off without warning.

What doesn't change is everything the KEP explicitly disclaims. The in-memory list the kubelet builds from a streamed response is the same size as the list it used to build from a unary one — streaming moves the bottleneck from "the wire" to "however much memory the kubelet process happens to have," and a runaway garbage-collection backlog still eats real memory even after the gRPC ceiling stops being the thing that kills the node outright. Two moves stay necessary regardless of which stage the feature is at:

  • Tune the GC knobs the #131407 case actually needed — MaxPerPodContainer and, notably, an explicit non-default MaxContainers (it's -1, unbounded, out of the box), plus keeping ttlSecondsAfterFinished and job history limits tight on anything that spawns short-lived containers, tenant build jobs included.
  • Watch kubelet_running_containers{container_state="exited"} as a leading indicator, not kubelet_running_pod_count. The #131407 incident happened with live pod count nowhere near a ceiling — it was dead containers piling up that did it, and that metric is the one that would have shown it coming hours before the RPCs started failing.

For a Cluster-API-managed fleet doing node-sizing math on something like a Hetzner AX162 (48 cores, 128 GB RAM), the right planning number isn't "how many pods fit" — it's "how many containers, live and recently-dead combined, will this node's tenant mix generate between garbage-collection passes," with enough margin below 11,000–14,000 that a GC hiccup doesn't turn into an outage before the alpha feature graduates far enough to remove the hard failure mode entirely.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, built on Cluster API so a fleet's actual container count is something an operator can watch and size for, not a number that quietly builds toward a gRPC wall. Star the repo on GitHub or deploy your first app today.

Sources​

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex
Back to all posts