Dora NodaKubernetes v1.36 shipped an alpha feature that sounds like exactly what a large Cluster API fleet needs: KEP-5866, Server-Side Sharded List and Watch, lets a horizontally-scaled controller tell the API server "only send me the slice of objects I own," instead of every replica receiving, deserializing, and discarding the full event stream. It's a real fix for a real cost.
Here's the part the release notes don't say out loud: turning on the feature gate today changes nothing for Cluster API's own machine controller, cluster controller, KubeadmControlPlane controller, MachineHealthCheck controller, or the cluster-autoscaler provider that sits next to them — because none of those run more than one active replica at a time. Sharded watch is a filter for controllers that are already spread across multiple active instances. CAPI's aren't.
What Server-Side Sharded List and Watch Actually Does
The mechanism is a new selector, not a new API. A client — typically each replica of a horizontally-scaled controller — adds selector=shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000') to its ListOptions. The API server computes a deterministic 64-bit FNV-1a hash of the named field for every object and returns only the ones whose hash lands inside [start, end). The same filter applies to both the initial LIST and the subsequent WATCH stream, so a replica's whole reconciliation loop — not just its startup sync — only ever sees its own shard.
Two field paths are supported at launch: object.metadata.uid and object.metadata.namespace. The hash is stable across every API server instance in the cluster, so two replicas asking for adjacent, non-overlapping ranges never double-process or drop an object. A shardInfo field comes back on the response so a client can confirm the server actually applied the filter — if it's absent, the client is expected to assume it got the unfiltered set and handle it accordingly.
That confirmation field matters because the feature is alpha and off by default: it lives behind the ShardedListAndWatch feature gate on the API server, targeted for wider adoption in later releases, with no compatibility guarantee attached yet. Nothing about enabling the gate breaks existing clients — a controller that never sends a shardRange() selector gets the exact same unfiltered stream it always did. The gate only changes behavior for the specific list/watch calls that opt into it.
The Failure Mode This KEP Targets — Cluster API Already Lived It
The problem KEP-5866 solves isn't hypothetical for Cluster API. When CAPI maintainers stress-tested a management cluster against 300 workload clusters, they found the KubeadmControlPlane controller becoming a bottleneck as reconciliation load grew — and one of the mitigations they tried was running one KubeadmControlPlane controller instance per namespace, manually partitioning the load across processes. It made things worse: the write-up describes the approach consuming excessive CPU and slowing everything down, not speeding it up.
That's the KEP-5866 problem in miniature, minus the fix. Splitting a controller into multiple instances without a server-side filter means every instance still has to receive and evaluate the full object stream to figure out which objects are its own — client-side filtering after the fact, paid for on every replica, every event. Namespace-per-instance sharding without a server-side selector doesn't cut that cost; it multiplies the number of processes paying it. KEP-5866 exists specifically to move that filter upstream, into the API server, so a replica's network and CPU bill scales with its shard, not with the whole cluster. CAPI's maintainers hit this exact wall building the thing the KEP now has an answer for.
Why the Feature Gate Changes Nothing for Stock CAPI Controllers Today
Here's the catch: sharded watch only pays off for a controller that's actually splitting reconciliation work across multiple simultaneously active replicas, each responsible for a disjoint slice of objects. Cluster API's controllers aren't built that way. Every core CAPI controller — the machine controller, the cluster controller, KubeadmControlPlane, MachineHealthCheck — runs on the standard controller-runtime high-availability pattern: multiple pods for redundancy, coordinated by leader election so that exactly one replica is ever doing reconciliation work at a time. The standby replicas sit idle, ready to take over if the leader dies; they don't share the load, because leader election exists to prevent two replicas from acting on the same object concurrently, not to divide objects between them.
The cluster-autoscaler's Cluster API provider is built the same way. It's typically deployed with multiple replicas for availability, but only the elected leader does any scaling work at any given moment — running more replicas buys failover, not throughput.
So of the three categories a Cluster-API-managed fleet actually runs at multi-tenant scale — machine health checks, autoscaling, and CAPI's own core reconciliation — all three sit behind the same active-passive gate. A management cluster reconciling Machine and Cluster objects across hundreds of tenant namespaces today pays the full watch-stream tax on its single active MachineHealthCheck reconciler, its single active KubeadmControlPlane reconciler, and its single active autoscaler instance. Enabling ShardedListAndWatch on the API server doesn't touch any of that — there's no second active replica for the filter to route work to.
The Architecture That Would Have to Exist First
Active-active Kubernetes controllers aren't unprecedented — they're just not what CAPI ships. Gardener's kubernetes-controller-sharding project, built by Tim Ebert as part of a master's thesis on the exact problem, is the closest working prior art. It defines a ControllerRing custom resource declaring a virtual ring of controller instances and which API resources get distributed across them. Each shard announces itself by holding a Lease object; a separate sharder component watches those leases, assigns objects to shards, and relabels each object with the owning shard's identity. Every controller instance then runs a controller-runtime cache filtered by a label selector matching its own shard label, so it only ever reconciles objects labeled as its own.
It works, but the sharder-proxy is doing exactly the client-side-adjacent job KEP-5866 makes unnecessary: relabeling every object so a label selector can approximate what a hash-range selector now does natively at the API server. A CAPI-style controller built on this pattern today still pays for that relabeling step on every object write. Once shardRange() is available and stable, that layer collapses — a shard announces the hash range it owns instead of waiting to be relabeled, and the API server does the partitioning the sharder-proxy used to do by hand. That's the real value of KEP-5866 for a project like CAPI: not a drop-in speedup for the controllers that exist now, but the missing primitive an active-active redesign would build on, without needing a Gardener-style sharder component running alongside it.
Where This Actually Helps a Multi-Tenant Fleet Today
The near-term win isn't CAPI's built-in controllers — it's the controllers a platform operator writes themselves. A Cluster-API-managed fleet running a self-hosted PaaS typically has its own tenant-facing operator watching a custom resource per tenant app — a bex.yml-style App CR, one per deployed service, potentially thousands across a shared management cluster once the fleet's tenant count crosses a few hundred namespaces. Unlike CAPI's core controllers, that operator is code the platform team controls end to end, which means it can be designed active-active from day one instead of inheriting leader-election defaults.
Concretely: run N replicas of the App-CR controller, and have each declare selector=shardRange(object.metadata.namespace, start_n, end_n) over a disjoint set of hash ranges covering the full 0x0 to 0xFFFFFFFFFFFFFFFF space — the same metadata.namespace field CAPI itself uses to scope tenants. Each replica's watch cache holds only its shard's tenant namespaces; CPU and memory for that replica scale with shard size, not fleet size, and adding a tenant namespace never adds load to a replica outside its owning shard. That's a genuine structural win, available the moment the feature gate is on and a client library supports the selector — it just requires building the controller to expect it, not turning on a switch for an existing one.
Is Flipping the Feature Gate in Production Worth It
The gate itself is low-risk to enable: it's opt-in per list/watch call, so a production API server with ShardedListAndWatch on behaves identically for every controller that isn't explicitly asking for a shard — which, per the sections above, is every stock CAPI controller a fleet runs today. The real tradeoff is everything downstream of the gate. Alpha means no compatibility guarantee — the shardRange() grammar, the supported field paths, or the feature itself could change or be pulled before graduating to beta, and no ergonomic client-go or controller-runtime helper for it has surfaced yet, so adopting it now means hand-rolling the selector and the shardInfo response check.
For a fleet operator, that argues for a specific, narrow use of the feature gate right now: turn it on in a staging management cluster, and use it to prototype the active-active tenant-operator pattern described above — not to expect any change in how CAPI's existing controllers behave, and not to depend on the exact selector syntax staying stable in a production fleet before it reaches beta. The multi-tenant scale problem the KEP is built to solve is real and CAPI's maintainers have already run into it firsthand; the fix for CAPI's own controllers is still a redesign away, gated on beta stability and on someone actually building the active-active reconciler that would consume it.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, on a Cluster API-managed fleet built for exactly this kind of scaling question. Star the repo on GitHub or deploy your first app today.
