Dora NodaEvery self-hosted control plane has a moment it dreads: something restarts — the API server, a controller, the box itself — and etcd's memory graph goes vertical. On a cloud-managed control plane that spike disappears into somebody else's headroom. On the 4GB node your own control plane runs on, it can end in an OOM-kill. etcd v3.7.0, released July 8, 2026, ships a real fix for exactly that spike: a streaming range read called RangeStream. Here is the concrete before-and-after for one big LIST, why it matters disproportionately on small boxes, and the checklist to run before you upgrade a Cluster API-managed cluster to the 3.7 line.
One big LIST, before and after: the memory picture
The problem is in etcd's oldest read path. A Range RPC is unary: the server reads every matching key-value pair, serializes the whole set into one protobuf message, and only then sends a single byte back. For a large result — the kind Kubernetes generates every time a controller performs a full LIST of every object of a type — three copies of the data coexist in server memory at once: the raw key-value slice read from the backend, the serialized protobuf, and the outbound gRPC send buffer. Paginated clients do not escape the tax either: each page of a paginated Range recomputes the total result count by walking the full index again, redoing work the previous page already paid for.
RangeStream, defined in KEP-5966, replaces that with a server-streaming RPC. It takes the same RangeRequest and returns the same result set, but etcd splits the results into chunks and streams them. Chunk size adapts to the values being returned, so a collection of large objects is bounded by bytes rather than by key count, and memory is freed as the stream progresses instead of being held until a whole page is assembled. The API server decodes each chunk as it arrives and releases it before pulling the next one — so neither side ever holds the whole collection at once.
Put numbers on it for the box this post is about. Take a modestly sized cluster whose full object listing serializes to roughly 60MB. Under the unary path, etcd can briefly hold on the order of 150–180MB for that single read — the raw slice, the serialized message, the send buffer — plus whatever the API server buffers on its end assembling the response. On a 4GB control-plane node that also runs the API server, the scheduler, and a controller-manager, a 150MB+ transient, landing at the same moment a restart triggers several such reads at once, is the difference between a slow minute and an OOM-killed etcd followed by a quorum flap. Under RangeStream the same read peaks at roughly one chunk — single-digit megabytes — on each side. The total bytes moved are the same; the peak is an order of magnitude lower, and, more importantly for a small box, predictable.
What RangeStream is, mechanically
RangeStream is a server feature in etcd v3.7, but Kubernetes only uses it when both halves are in place. The kube-apiserver integration point is the watch-cache initialization path: when the feature is enabled, the watch cache's initial sync calls the streaming RPC instead of paginated range requests, converts each incoming chunk into synthetic create events, and queues them inline. The fallback paths benefit too — anywhere a list request cannot be served from the cache and the API server must read etcd directly, it streams instead of buffering.
Two facts determine whether your cluster actually gets this. First, the EtcdRangeStream feature gate must be enabled on the kube-apiserver. It is beta and on by default in Kubernetes v1.37, which went generally available on August 26, 2026 — but a 1.37 API server paired with an older etcd simply keeps using the paginated path. The API server probes etcd's capabilities at startup and falls back at runtime if a streaming call returns Unimplemented, so a mixed-version fleet degrades gracefully rather than breaking. Second, etcd itself must be v3.7 or later. If you need to turn streaming off after enabling it, the escape hatch is one flag:
--feature-gates=EtcdRangeStream=falseVerification is a metric, not a log line. The API server records streamed reads under their own operation label on its etcd request metrics — a nonzero count on etcd_request_duration_seconds_count{operation="listStream"} means RangeStream is actually serving reads. Check that after the upgrade, not just the version string: version skew between the API server and etcd members is exactly how a fleet ends up believing it has streaming reads while still paying the buffering tax. Unit benchmarks for the feature show watch-cache initialization running roughly 1.4 times faster, with larger combined gains when paired with the concurrent object decoding that landed in the same release — but on a small box, treat the memory predictability as the prize and any speedup as a bonus.
Why the small box feels it most
None of this machinery cares how big your fleet is. It cares how thin your headroom is, and small self-hosted control planes run thinner than anything else in the ecosystem.
Start with etcd's own guardrails. The default backend quota is 2GiB, with about 8GiB as the practical recommended maximum — and when the database crosses the configured quota, etcd raises a NOSPACE alarm and goes read-only, blocking all API server writes until an operator intervenes. A single etcd request is capped around 1.5MiB by default, which is why oversized ConfigMaps and fat CRD status blobs are an anti-pattern in the first place. These limits bite earlier on a small cluster not because the data is bigger but because the machine is smaller: there is no spare gigabyte anywhere.
Then add co-location. A self-hosted control plane on one 4GB node typically runs etcd, the API server, the scheduler, and the controller-manager on the same kernel, competing for the same page cache and the same OOM killer. A managed control plane spreads those four across isolated capacity with headroom budgeted by someone else; your box does not. The unary-read spike described above lands on etcd's cgroup at the same moment the API server is buffering the same response on its own heap — both contestants for the same 4GB.
Finally, add the trigger pattern. The buffering tax is paid per full read, and full reads come in storms: every controller restart, every API server restart, every dropped watch that forces a relist. Kubernetes controllers are built on informers that LIST once and then WATCH, so steady state is cheap — but the LIST side of that contract fires precisely during the incidents when the box is already under stress. A node reboot that restarts five controllers at once is five full LISTs hitting etcd within seconds of each other, each holding its triple-buffered peak simultaneously. That compounding is what turns a survivable 150MB transient into a dead etcd member, and it is why a feature that only changes how reads are served still changes the survival odds of the whole node.
The part RangeStream does not fix: your controllers' own caches
This is the boundary worth stating plainly, because it decides where your tuning effort goes after the upgrade. RangeStream shrinks the etcd side and the API-server side of a large read. It does nothing to the memory your controllers hold afterward.
A controller-runtime controller keeps a complete local copy of every object type it watches — one informer per type, each holding its objects in memory behind a LIST-then-WATCH connection to the API server. An unfiltered informer caches every object of that type in every namespace, and flooding a cluster with large ConfigMaps or Secrets can OOM-kill the operator itself, a failure mode Red Hat's operator security research has demonstrated concretely. Critically, nothing in the streaming path changes that: the controller still ends up holding the same objects. Common misconceptions make this worse — the informer resync period (10 hours by default in controller-runtime) does not re-list from the server; it re-emits what is already cached — so the fix for controller-side memory is cache scoping, not a newer etcd.
The practical consequence is a two-sided tuning rule. After upgrading to etcd 3.7, etcd-side LIST spikes stop being your binding constraint — but controller-side cache size becomes relatively more important, because it is now the largest remaining memory variable you control. Scope every controller's cache with label or field selectors so it watches only what it reconciles (ByObject cache configuration in controller-runtime), beware unregistered types silently spawning cluster-wide informers on first Get/List, and keep fat objects out of watched types. RangeStream buys you headroom; spend it on correctness, not on watching more than you need.
The pre-upgrade checklist for the 3.7 line
etcd 3.7 is not a drop-in patch release — it removes things. Run this checklist against a Cluster API-managed cluster before the rolling upgrade, not during it.
1. Confirm the floor: v3.6.11 or later first. The supported path is v3.6 → v3.7 from a recent 3.6 patch. If any member lags, bring the whole quorum current on 3.6 first and let it settle before introducing 3.7 binaries.
2. Grep your manifests for --experimental-* flags. Every deprecated experimental flag is removed in 3.7. Migrate each one to its corresponding feature gate or stabilized argument beforehand — a member that fails to start on an unknown flag during a rolling upgrade turns a routine operation into a quorum-risk event. This is the single most likely breakage for clusters whose etcd manifests were written years ago and rarely re-read.
3. Check for v2store dependencies. v3.7 is the first release that is 100 percent on the v3 storage backend: v2 discovery, v2 request handling, and the v2 client are all gone. Anything still speaking the v2 API — ancient sidecars, legacy bootstrap tooling, discovery URLs from another era — breaks silently or loudly at upgrade time. Inventory clients before you start.
4. Upgrade one member at a time, and verify quorum between each. This is standard etcd rolling-upgrade discipline, but it matters more here because the release also carries a protobuf overhaul and a storage-layer refactor. Confirm each member is healthy and the cluster has a stable leader before moving to the next. Snapshot every member before you begin — snapshot and restore compatibility is part of what you are changing, so the backup you take on the old version is the one you can trust for rollback.
5. Audit compaction and defragmentation hygiene. Auto-compaction should already be on (periodic mode with a short retention is the common self-hosted setting), and a defrag pass before the upgrade keeps backend size — and therefore upgrade-time I/O — down. A bloated backend makes every subsequent step slower and riskier.
6. Plan the Kubernetes side to actually get streaming. etcd 3.7 alone changes nothing for the API server until a 1.37+ kube-apiserver with EtcdRangeStream enabled talks to it. Sequence the etcd upgrade first, then the Kubernetes minor bump (or verify your management tooling pairs them), then confirm with the listStream metric — not the version string — that streamed reads are being served. Know the rollback story on both layers: the etcd downgrade path and the single-flag gate disable.
7. Load-test the relist storm, not the steady state. Before the upgrade, capture etcd memory during a controlled controller restart. After the upgrade, repeat it. The metric that should visibly flatten is peak transient memory during watch-cache rebuilds. If it does not flatten, you have version skew (step 6) or your spike was never LIST-driven — both worth knowing before the next real incident.
(For completeness: 3.7 also ships adjacent performance work such as keys-only reads served from the in-memory index and faster lease handling. They help the same small boxes, but they are a separate tuning story from the streaming-read change this post is about.)
What to expect — and what not to
RangeStream will not let a small control plane hold more objects. The steady-state costs — per-object storage, revision churn that compaction must keep pace with, raft consensus on every write — are untouched by a chunked read API, and the documented guidance (2GiB default quota, ~8GiB practical ceiling) still binds. If your etcd database grows monotonically, streaming reads buy you time on the read path while the write path keeps filling the disk; fix the growth separately with compaction, retention, and smaller objects.
What it does is narrower and, for a 4GB box, more valuable: it converts the most violent transient in etcd's read path from an unpredictable multi-copy spike into a flat, byte-bounded stream. Fewer OOM-kills during restarts, fewer quorum flaps cascading from a single dead member, and a memory graph you can provision against instead of padding with guesswork. For a fleet operator sizing control planes for cost rather than worst-case LIST storms, that predictability is the whole point — measure the relist peak before and after, and let the flattened graph be the receipt.
Bex.co runs its tenant fleets on Cluster API against owned hardware, so control-plane memory on small boxes is a capacity-planning input, not an academic question. If you are sizing your own — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
Sources
- Announcing etcd v3.7.0 — Kubernetes blog, July 8, 2026: RangeStream, keys-only optimization, leases, protobuf overhaul, v2store removal,
--experimental-*flag removal. - Kubernetes v1.37: etcd RangeStream Cuts Memory Use on Large List Reads — September 1, 2026:
EtcdRangeStreambeta/default-on, watch-cache init path,listStreammetric,--feature-gates=EtcdRangeStream=falseoff-switch, ~1.4x cache-init benchmark. - KEP-5966: etcd RangeStream — the design proposal behind the streaming RPC and apiserver integration.
- Kubernetes v1.37: Garhwal — release notes: GA August 26, 2026; combined cache-init gains with concurrent decode over 150k pods.
- How the controller-runtime Cache Actually Works — July 29, 2026: informer LIST-then-WATCH, resync semantics, cache scoping.
- Protect your Kubernetes Operator from OOMKill and 5 anti-patterns that cause Kubernetes operator vulnerabilities — Red Hat Developer: unfiltered informer OOM via ConfigMap flooding,
ByObjectselectors. - etcd v3.6 to v3.7 upgrade guide — 3.6.11+ prerequisite, experimental-flag and v2store removals, rolling-upgrade procedure, downgrade/rollback pointer.
