Dora NodaSNCF runs more than 200 Kubernetes clusters on Cluster API, provisioned in about 30 minutes each, reconciled back to declared state on a monthly cadence the CNCF calls a zero-drift posture. The French national railway proved the declarative model holds at national-infrastructure size, with node autoscaling extended into its own datacenters — a capability CAPI made possible outside any public cloud.
Your fleet is probably five Hetzner machines, not two hundred clusters. But the uncomfortable question SNCF's playbook asks of you is not about machine count. It is about whether your controllers have ever seen real contention at all.
Here is the answer for most self-hosted platforms: no. The Machine controller, the autoscaler, the descheduler, and whatever tenant-lifecycle controllers you wrote in-house have only ever operated at today's load. The week your biggest tenant doubles is the week you discover your reconcile-loop budget.
There is a way to find out sooner, on a laptop, in an afternoon: KWOK (Kubernetes WithOut Kubelet) simulates thousands of nodes and pods with no kubelets, no runtimes, and no workloads actually running. The project reliably maintains 1,000 fake nodes and 100,000 fake pods on ordinary laptop hardware, creating about 20 nodes or pods per second.
The trick that makes it a load-testing tool rather than a toy is what it does not fake: the API server, etcd, and your controllers are all real. Every watch, list, cache sync, and reconcile in the loop below runs the genuine code path — only the kubelets are imaginary.
So here is the concrete plan this post delivers: a ramp matrix that takes your fleet's controllers from today's load to 400 fake nodes in four rungs, with a pass/fail threshold at each rung. Run it before tenant growth runs it for you.
The ramp matrix: the whole test on one table
One clarification before the table, because the TODO item behind this post blurred it and a reviewer rightly objected: SNCF's number is 200 clusters; the laptop test below runs hundreds of fake nodes inside a single KWOK cluster. They are not the same unit. SNCF is the proof that CAPI reconciliation holds at fleet scale; the fake nodes are how you find out whether your controllers hold as object counts climb toward the scale your own growth implies. Pick the rung where your fleet is heading, then run one rung past it.
| Rung | Fake nodes | Fake pods (10/node) | Controllers under test | Pass threshold | Fail signal |
|---|---|---|---|---|---|
| 1. Today-plus | 50 | 500 | Machine/MachineDeployment | p99 reconcile flat vs. baseline, queue depth near zero | Reconcile latency climbs with object count instead of staying flat |
| 2. Growth year | 100 | 1,000 | + cluster-autoscaler | Pending pods clear within N autoscaler loops (10s each) | Desired-size oscillation, pending pods never clearing |
| 3. Title rung | 200 | 2,000 | + descheduler | Evictions complete, balance restored in one pass | Eviction QPS throttled, PDB-blocked pods stall the pass |
| 4. Breaking point | 400 | 4,000 | + your tenant controllers | Your SLOs hold, controller memory sublinear | Whichever controller you wrote breaks first — that is the point |
All four rungs sit two orders of magnitude under KWOK's demonstrated ceiling, so when something fails, it is your controller failing — not the harness. At 20 objects per second, standing up the top rung takes minutes, not hours. Total laptop time for the full matrix: an afternoon, most of it watching dashboards.
KWOK in one paragraph, then the setup
KWOK has two pieces: kwok, which simulates the lifecycle of fake nodes, pods, and other API objects through configurable stages (a node appears, then flips Ready; a pod goes Pending, then Running — with no kubelet ever involved), and kwokctl, a CLI that builds you a whole cluster with those simulated nodes in seconds. Its documented uses name this post's audience directly: measuring how well a controller scales with node and pod counts, and simulating node failures or network partitions by flipping node conditions or deleting nodes at random. Install it, create the rehearsal cluster, and point kubectl at it:
kwokctl create cluster
kubectl config use-context kwok-kwok
kubectl apply -f rehearsal/nodes-rung-1.yaml
kubectl get nodesKeep each rung's nodes and pods as checked-in manifests (nodes-rung-1.yaml through nodes-rung-4.yaml) so the ramp is reproducible and diffable — the fleet you load-test in six months should be the same fleet plus growth, not a remembered incantation. Record a baseline first: with zero fake load, capture each controller's p99 reconcile latency, work-queue depth, resident memory, and the API server's list latency. Every rung after that is a comparison against this baseline, not against vibes.
Running the rungs: which controller breaks, and how you will know
Rung 1 — the Machine controller at 50 nodes. Install your Cluster API controllers (or your platform's own machine-lifecycle equivalent) pointed at the KWOK cluster and create Machine and MachineDeployment objects against the fake nodes. What you are measuring is reconcile-loop discipline: does p99 reconcile stay flat as the object count grows, or does it slope upward from the very first rung? An upward slope at 50 nodes is the cheapest bug this whole exercise can find — usually an unscoped watch or a reconcile that lists instead of getting. Fix it here, where the fix is a one-line selector, not an incident.
Rung 2 — the autoscaler at 100 nodes. Add pending pods faster than current capacity can absorb and watch the cluster-autoscaler loop, which evaluates every 10 seconds. The honest caveat: no real machines will appear, because there is no cloud provider behind the fake nodes — what you are testing is the decision path, desired-size computation and oscillation behavior, not the provisioning path. The fail signal is specific: desired size flapping up and down across loops, or pending pods that never resolve into a scale decision at all. Both are logic bugs that are invisible at low pod counts and catastrophic when a real tenant surge arrives, because the autoscaler is the one controller whose failure mode is "do nothing while the queue grows."
Rung 3 — the descheduler at 200 nodes. With 2,000 fake pods spread unevenly, trigger a descheduling pass and watch evictions. This is the rung the title promises, and it tests the controller most teams never think to load-test: eviction QPS against the API server, PDB-blocked pods stalling a pass, pods with local state that can never move. A descheduler that works beautifully at 500 pods and stalls at 2,000 has handed you a slow-motion outage — balance degrades gradually, nobody pages, and by the time latency alerts fire the cluster is thoroughly lopsided.
Rung 4 — your own controllers at 400 nodes. This is the rung that finds your bug. Tenant-domain provisioning, per-tenant quotas, build-queue backpressure — the bespoke reconciliation logic that no upstream project load-tests for you. Point it at 4,000 fake pods and watch resident memory first, reconcile latency second. In our experience of where these break, the cause is almost always the same, and it deserves its own section.
Why the 200-to-400 rung is where the informer cache breaks
By default, a controller-runtime cache starts an informer behind every Get and List and keeps a full in-memory copy of every object of that type the controller is allowed to see. On a small fleet that behavior is harmless. As the object count climbs, it becomes the dominant term in the controller's memory footprint — cache memory is the constraint teams hit before API-server load, and a cluster-wide Pod watch is a very different proposition from watching your own CRD. The cache also has to fully list and sync before the controller serves anything, so WaitForCacheSync gets slower exactly when you most want fast restarts.
This is the failure the 200-to-400 rung exists to expose, and the fix is a checklist, not an architecture:
- Narrow every watch with label or field selectors before touching anything else.
- Scope watches to namespaces where cluster-wide visibility is not genuinely needed.
- Strip managed fields from cached objects; the savings are significant on Pod-heavy watches.
- Never read a high-cardinality type (Pods, Events) from the cache unless the reconcile truly needs it — Get the single object instead.
Two metrics tell you whether the checklist worked: controller resident memory per watch (flat across rungs means the cache is bounded; linear growth means something is caching the world) and API-server list latency during a cold cache sync. If rung 4 doubles memory over rung 3, you do not have a scaling problem — you have a selector missing, and KWOK just found it for the price of a laptop afternoon.
What KWOK cannot tell you
A load test is only honest if it names its blind spots, so here are this one's. KWOK fakes kubelets, which means everything below the API is untested: image-pull latency, CPU throttling and OOM kills, volume attach and CSI behavior, CNI and network-policy enforcement. A node that goes Ready in the simulation never had to initialize a GPU driver or a storage plugin — the exact half-ready-node failure mode that per-node-group readiness gating exists to catch.
And while the API server and etcd under the simulation are real, their I/O profile under thousands of light fake objects is kinder than production's heavier, churn-ier writes. KWOK answers "do my controllers hold as object counts grow" with high fidelity. It does not answer "do my nodes behave." That second question still needs a staging cluster with real machines — ideally the same declarative CAPI flow as production, just smaller.
Run it on a Friday afternoon
The full matrix — baseline, four rungs, cache checklist, honest notes on what still needs staging — fits in one focused afternoon. Do it this week, while the fleet is healthy and the results are allowed to be embarrassing. The alternative is the schedule the TODO item warned about: discovering your reconcile-loop budget the same week your biggest tenant doubles, when every fix ships under incident pressure and every missing selector costs real latency.
Concretely: check in the four rung manifests, record the baseline numbers somewhere your future self will find them, and re-run the matrix every time you add a controller or double the fleet. SNCF's discipline was monthly reconciliation of 200 clusters toward declared state. Yours can start smaller — four rungs of fake nodes, re-run on a cadence, zero drift between what you believe your control plane can handle and what it actually can.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
