Dora Nodaetcd v3.7.0 shipped a feature the project has wanted for close to a decade: RangeStream, a new RPC that streams a large key range back in chunks instead of building the whole response in memory before sending a single byte. It's a real fix for a real problem — the exact list-watch pattern every Kubernetes controller depends on. It is not, however, the thing that will let a Cluster API management cluster hold more workload clusters. The math that actually sets that ceiling — etcd's own documented soft limits on database size and object count — hasn't moved, and when Cluster API's own maintainers stress-tested 300 workload clusters on one management cluster, the wall they hit wasn't etcd memory pressure at all.
What RangeStream Actually Fixes
The problem RangeStream solves is specific and well understood. etcd's existing Range RPC is unary: the server reads every matching key-value pair, serializes the whole set into one protobuf message, and only then sends it back over the gRPC connection. For a large result — the kind a Kubernetes API server generates every time a controller does a full LIST of, say, every Machine object in a fleet — that means three copies of the data coexisting in server memory at once: the raw key-value slice read from etcd's backend, the serialized protobuf, and the outbound gRPC send buffer. Paginated clients make it worse in a different way: each page of a paginated Range call recomputes the total result count by walking the full B-tree index again, redoing work the previous page already paid for.
RangeStream, defined in KEP-5966, replaces that with a server-streaming RPC that reuses the same RangeRequest but returns chunks incrementally. The integration point that matters for Kubernetes is the kube-apiserver's watch-cache sync path: when the feature is enabled, the watch cache's initial sync calls KV.GetStream instead of a unary Range, converts each incoming chunk into synthetic "created" events, and queues them inline — never holding the full list in memory at once. Direct LIST calls from controllers benefit the same way. etcdctl picked up a matching --stream flag on get for direct inspection.
Two caveats matter more than the feature announcement suggests. First, this isn't live by default: EtcdRangeStream is a Beta-stage, off-by-default feature gate targeted at Kubernetes 1.37, not something a fleet running current kube-apiserver gets automatically the day etcd 3.7 lands. Second, it fixes the buffer-tax on a specific event — the moment a watch cache is built from scratch, which happens on apiserver startup, controller restart, or any fresh watcher establishing its initial state. It does nothing for the steady-state cost of keeping thousands of objects and their watchers alive between those moments, which is where the real ceiling lives.
The Ceiling, in etcd's Own Numbers
etcd's hardware guidance recommends keeping the backing database under roughly 8GiB — a soft ceiling, not an enforced one, with a 2GiB default backend storage quota that operators typically raise for anything beyond a small cluster. GKE, notably, caps its managed etcd at 6GiB rather than the full 8GiB headroom etcd itself allows, treating the documented soft limit as one they don't trust operators to approach in production.
Object count degrades earlier and more gradually than database size does. Independent benchmarking puts the knee in the curve at roughly 30,000–40,000 objects, where API server latency stops scaling linearly — routine reads that ran at a 50ms baseline start landing north of 500ms. Past roughly 80,000 objects, the same data characterizes the risk profile as materially higher failure risk, not just slower responses. The same benchmark found the effect starts showing up even earlier than that headline number suggests: creating 1,000 Secrets directly against a lightly-loaded cluster produced a 10–13x latency spike, peaking around 650ms, well short of the 30k mark. Object size compounds this independently of count — a cluster carrying a modest number of large objects (10–100KB pods, in one write-up's example) can destabilize at a fraction of the node count Kubernetes officially supports at 5,000 nodes, because size, not just count, drives revision and compaction pressure.
None of these numbers move because RangeStream shipped. They're steady-state costs — per-object storage footprint, the revision churn compaction has to keep pace with, raft consensus overhead on every write — that a chunked read API was never going to touch, because it only changes how a large read is served, not how much the cluster has to store or how often that state changes underneath a live watch.
What Changes for a Cluster API Fleet — and What Doesn't
This is where Cluster API's own public scaling data is more useful than any etcd microbenchmark, because it's a real management cluster carrying real CAPI workload objects, not a synthetic Secret-creation loop. In GitHub issue #8052, CAPI maintainers scaled a single management cluster to 300 workload clusters — reaching 100 clusters in about 15 minutes, but 300 clusters in roughly 135 minutes, with a single additional cluster at that point taking more than 8 minutes to provision. That's a steep, non-linear slowdown, exactly the shape the etcd object-count benchmarks predict. But the maintainers' own diagnosis of why is the important part: "the bottleneck seems to be the Kubeadm control plane provider. There is a long pause after the KCP is created before the Machines appear" — a controller reconcile-pacing problem, not an etcd memory-buffering problem. The team also tried sharding the KubeadmControlPlane controller by namespace (10 namespaces × 10 clusters each) as a mitigation, and it made things worse — CPU usage climbed and the whole system slowed down, because splitting a controller doesn't remove work, it adds coordination overhead on top of the same underlying etcd.
That's the honest read on what RangeStream buys a Cluster API fleet: faster, lower-memory watch-cache rebuilds on apiserver or controller restarts, and on any operation that does a genuine full-cluster LIST — a clusterctl inventory pass, a kubectl get machines -A across a large fleet, a backup tool walking every object. It does not touch the KubeadmControlPlane reconcile-pacing bottleneck that actually capped Cluster API's own 300-cluster test, and it does not change the per-object revision churn that CAPI's own controllers generate by writing Machine and KubeadmControlPlane status on every reconcile loop. A fleet that hits the same wall CAPI's maintainers hit will hit it exactly as fast with RangeStream enabled as without it.
Object Count vs. Byte Size: Which Ceiling Binds First
Translating etcd's raw thresholds into "how many workload clusters" requires a rough per-cluster object count, and it's worth being explicit that this is an estimate, not a spec. A small CAPI-managed workload cluster — 3 control-plane machines, 3 worker machines, one MachineDeployment — carries roughly: 1 Cluster, 1 KubeadmControlPlane, 1 infrastructure-provider cluster object (e.g. HetznerCluster), 1 MachineDeployment plus 1 MachineSet, and 6 Machine objects each paired with an infrastructure machine CR and a bootstrap KubeadmConfig (18 objects), plus a handful of generated secrets (CA, kubeconfig, service-account keys) — roughly 28–30 core objects per workload cluster, before counting anything a ClusterResourceSet installs on top.
Run that count against both of etcd's ceilings separately:
- Object-count bound: at ~30 objects/cluster, the 30,000–40,000-object latency knee caps a single management cluster at roughly 1,000–1,300 workload clusters before read latency starts degrading non-linearly; the 80,000-object failure-risk line caps it around 2,600 clusters.
- Byte-size bound: CAPI objects carry substantial status (conditions, provider IDs, node references, addresses), commonly landing in the 3–8KB range serialized; at a representative 4KB average, 30 objects/cluster is only ~120KB per workload cluster. Against the 8GiB soft ceiling, that's roughly 69,000 workload clusters before database size becomes the binding constraint.
The gap between those two numbers is the point: for a typical CAPI object mix, object-count latency degradation binds tens of thousands of clusters before database size ever becomes the problem. A fleet operator watching etcd's disk usage as their early-warning signal is watching the wrong gauge — a management cluster will feel the pain of a slow LIST and a laggy watch cache long before its etcd data directory approaches anything close to 8GiB.
What to Watch Before You Hit the Wall
Given that, the practical monitoring list for a Cluster-API-managed fleet looks different from "watch etcd's disk usage":
- Total object count on the management cluster, tracked against the 30,000-object soft knee — not database size.
- Watch-cache initialization latency on apiserver and controller restarts — the one metric RangeStream, once GA and enabled, should visibly improve. A widening gap between this metric before and after enabling
EtcdRangeStreamis the direct signal the feature is paying for itself. - KubeadmControlPlane reconcile latency, specifically the delay between a KCP object's creation and its Machines appearing — the metric that actually flagged Cluster API's own 300-cluster ceiling, and the one RangeStream does nothing for.
- A documented no to "should we shard the control-plane controller by namespace" as a scaling fix — CAPI's own experiment shows that move can make CPU usage and coordination overhead worse, not better, absent additional tooling on top.
For the harder ceiling — the one where a fleet outgrows a single etcd's ability to serve reads at all, not just the watch-cache-rebuild tax RangeStream addresses — the actual fix under active development is server-side sharded list and watch, targeted for a future Kubernetes release. That's a genuinely different mechanism (splitting the apiserver's read-serving work across shards, rather than making one shard's reads cheaper to stream) and deserves its own treatment rather than a hand-wave here. What RangeStream buys today is real, but it's a latency and memory-predictability improvement on one specific operation — not a higher ceiling on how big a Cluster API fleet's control plane can grow.
Bex.co runs tenant fleets on Cluster API against owned Hetzner hardware, which means questions like "how many workload clusters can one management cluster actually hold" aren't academic — they're capacity-planning inputs. Knowing which metric actually predicts the wall (object count, not disk size; controller reconcile latency, not etcd memory) is the difference between a documented scaling plan and a surprise outage. Star the repo on GitHub or deploy your first app today.
Sources
- KEP-5966: etcd RangeStream — kubernetes/enhancements
- etcd v3.7.0-rc.0 Now Available for Testing — etcd blog
- Announcing etcd v3.7.0-beta.0 — Kubernetes blog
- Hardware recommendations — etcd documentation
- Kubernetes etcd Sharding vs Virtual Clusters: What Scales? — vCluster
- Why etcd breaks at scale in Kubernetes — learnkube
- Slow cluster creation when working with hundreds of clusters — kubernetes-sigs/cluster-api issue #8052