Your Controller Makes 11 API Calls, Then Goes Quiet: What the controller-runtime Cache Costs a Cluster API Fleet at Scale

A well-behaved Kubernetes controller makes about eleven API requests in its entire life — all of them in the first second — and then never reads from the API server again. That is not a boast about a particularly efficient controller. It is the normal, measured steady state of anything built on controller-runtime: one snapshot per watched type at startup, one watch stream per type after that, and every Get and List inside Reconcile served from process memory. An independent request-counting experiment confirmed the shape — the API cost of a controller is almost entirely a startup cost, and cache warm-up (WaitForCacheSync) is where it shows.

Here is the second half of the promise, up front, because this post is written for teams running their own Cluster API fleet: the defaults behind that quiet are fine below roughly one hundred Machines, and they need tuning past it. The reason fits in one sentence — every watched type means a full in-memory copy of every object of that type, multiplied by every controller manager that restarts at once — and the rest of this post proves it: how the cache actually works, the three mistakes everyone makes against it, and a sizing table that says exactly which knob to turn at 50, 200, and 500 Machines.

The occasion is the Kubernetes blog's July 29, 2026 deep dive, "How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server", which finally wrote down the internals most operator authors treat as folklore. What follows is a close read of that post, aimed at one specific reader: the platform engineer whose management cluster reconciles Cluster, MachineDeployment, and Machine objects on machines they own.

One snapshot, one watch, then silence​

The cache in controller-runtime is not an optimization layered over direct API reads. It is the operating model, built from the same three primitives that power Kubernetes itself: a Reflector, a delta queue, and an Indexer, fused into a SharedIndexInformer.

The Reflector is the only component that ever talks to the API server, and it has exactly two jobs: fetch the initial snapshot at startup, then hold a watch open from the snapshot's resourceVersion. There is no gap between the two because the watch resumes exactly where the snapshot ended. Modern versions even collapse them into one request — the Reflector opens the watch with sendInitialEvents=true and the API server begins the stream with synthetic ADDED events for current state before switching to live changes, falling back to a plain list only if the server cannot stream.

If the connection drops, the Reflector reconnects from the last known version; only a 410 Gone ("you are too far behind") triggers a full re-snapshot, called a relist. Relists happen on failure, never on a schedule.

Between the Reflector and the store sits the delta queue — historically DeltaFIFO, now the simpler, strictly ordered RealFIFO, which client-go 1.36 made the only option. And the store itself is an Indexer: a map of objects plus inverted indexes, guarded by a single sync.RWMutex that readers share and the informer needs exclusively to write. That lock is worth remembering — a List holds the read lock while it walks every object of a kind, and the next store write waits behind the walk.

It was a real bottleneck in kube-controller-manager at scale (kubernetes#130767), and recent client-go releases shorten the write-lock hold time. The contention never fully goes away; it just moves.

The keyword in SharedIndexInformer is shared. The manager creates one informer per group-version-kind, and every controller, webhook, and event source in the process subscribes to it. Ten reconcilers watching Pods still cost the API server exactly one snapshot and one watch.

Startup runs in a fixed order: mgr.Start brings up every informer, each Reflector takes its snapshot, indexes rebuild, the informer marks itself synced — and only then do workers start draining workqueues into Reconcile. "The reconciler is running but the cache is empty" is not an observable state, with one exception: a Get for a type nobody registered a watch for starts a new informer on the spot and blocks until it warms. If you need to read before mgr.Start, use mgr.GetAPIReader(); the regular client fails fast with ErrCacheNotStarted.

So the very first r.Get inside your reconciler is already a map lookup plus a DeepCopy. No HTTP, no TLS, no protobuf, no etcd. No "first time slow, then fast" — the warm-up happened before the first Reconcile ever ran.

Reads are memory, writes are the network​

client.Client is a composite with a deliberate split personality: reads (Get, List) go through the cache, writes (Create, Update, Patch, Delete) go straight to the API server. Reads are frequent and should be cheap; writes are rare and should be exact. Writing through the cache would invite split-brain — the local copy believing a change landed that the API server already rejected.

"Exact" is enforced by optimistic concurrency. The cached object you read carries the resourceVersion the Reflector last observed. Your Update sends that version back, and the API server compares it against etcd: match, and the write lands; newer in etcd, and you get a 409 Conflict because somebody beat you to it. No locks are taken; losers re-read and retry. A 409 is not a bug — it is the protection working.

The corollary is the write-visibility window. Your Update returns, and for a few milliseconds — with no guaranteed upper bound — the cache still holds the old version, because the new state only arrives via the watch stream. A Get immediately after an Update can hand you back the object you just overwrote. Every classic cache mistake grows out of that window:

• Read-after-write expecting immediacy. Reconciling, updating a status, then reading the object back and acting on the stale copy. If the next decision depends on your own write, chain it from the write response, not from a fresh read.
• Mutating shared memory on the event path. The read path (Get, List) deep-copies by default, and has since the earliest releases — you can opt out with UnsafeDisableDeepCopy, named that way on purpose. But the event path is not shielded: there is no copy anywhere between the informer and your Predicate or EventHandler. Anything arriving in UpdateFunc or EnqueueRequestsFromMapFunc is shared with every other subscriber to that type. Call DeepCopy() before mutating, or you silently corrupt the cache for the controller next door. In review, any SetLabels or status assignment inside a predicate or map function without a preceding copy is a stop-the-line finding.
• Believing resync is relist. The informer's resync period (cache.Options.SyncPeriod, ten hours by default) does not rebuild anything from the API server. It re-emits everything already in the indexer back through the queue as synthetic OnUpdate(old, old) calls — zero API traffic. It exists for controllers whose state lives partly outside the API (a cloud resource changed out-of-band produces no watch event). And because both sides of the synthetic update are the same object, predicates that compare old and new, like GenerationChangedPredicate, quietly drop it. Do not lean on resync as a drift safety net; it cannot see what the watch never told it.

The cache is a query engine — if you index it​

The Indexer half of the informer is an underused superpower. Register a field index at manager setup, and the cache becomes a near-complete in-memory query engine:

if err := mgr.GetFieldIndexer().IndexField(ctx, &corev1.Pod{}, "spec.nodeName",
    func(obj client.Object) []string {
        pod := obj.(*corev1.Pod)
        if pod.Spec.NodeName == "" {
            return nil
        }
        return []string{pod.Spec.NodeName}
    }); err != nil {
    return err
}

Then List with client.MatchingFields{"spec.nodeName": "node-1"} resolves through the inverted index instead of walking every Pod. Three facts from the deep dive that most code gets wrong: the index name is an arbitrary string key, not parsed JSONPath — it only has to match the MatchingFields key exactly. The indexed value is computed by your function, not read from a field — you can lowercase, join fields into composite keys, or bucket timestamps. And every MatchingFields query needs its registered IndexField, or the call fails outright rather than degrading. (Get never consults a field index; it is a direct store-key lookup.)

Contrast that with MatchingLabels, which is fine to use but buys no traversal savings — labels filter during the walk, they do not shrink the candidate set. To shrink what the informer holds at all, filter at cache-population time with cache.ByObject, which pushes the selector down into the watch itself, optionally combined with a Transform that strips noise like managedFields on the way into the store:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{
        ByObject: map[client.Object]cache.ByObject{
            &corev1.Secret{}: {
                Namespaces: map[string]cache.Config{"my-controller": {}},
                Label: labels.SelectorFromSet(labels.Set{
                    "app.kubernetes.io/managed-by": "my-controller",
                }),
            },
            &corev1.Pod{}: {
                Transform: func(obj any) (any, error) {
                    pod := obj.(*corev1.Pod)
                    pod.ManagedFields = nil
                    return pod, nil
                },
            },
        },
    },
})

The ordering trap here is real and has bitten production controllers: Transform runs after the objects are listed and decoded. Contour was OOMKilled on startup because its Secret informer performed an unfiltered cluster-wide list that fully decoded tens of thousands of accumulated helm.sh/release.v1 Secrets — multi-megabyte blobs each — before the existing Transform could strip them, spiking past the container memory limit before the GC could react (contour#7660). The same unbounded-cache shape OOMed the cass-operator in cluster-scoped mode (cass-operator#947). Transform shrinks the steady-state store; only selectors shrink the startup snapshot. If your controller touches Secret, ConfigMap, Pod, or Event in a large cluster, that distinction is the difference between a lean cache and a multi-gigabyte surprise delivered by the first list.

What this costs a Cluster API fleet at 50, 200, and 500 Machines​

A Cluster API management cluster is, from the cache's point of view, an unusually watch-heavy control plane: the CAPI controllers, the infrastructure provider (CAPH on Hetzner, for example), the bootstrap provider, plus your platform's own tenant controllers, all holding informers against Cluster, Machine, MachineSet, MachineDeployment, Node, and Secret objects on the same API server. At dozens of Machines the default configuration is genuinely fine — one snapshot plus one watch per type is a handful of small lists. The table below shows where that stops being true. The numbers are order-of-magnitude Fermi estimates, not benchmarks, built on stated assumptions: a Machine-class object averages ~2KB on the wire and ~5KB decoded and indexed in-store; a Node runs ~4KB wire / ~8KB in-store with status churn; a Secret averages ~5KB once helm-release blobs are excluded by selector — and an order of magnitude more when they are not.

Three conclusions fall out of the table. First, memory is the cliff you hit before API load. Steady-state reconcile traffic stays near zero at every row; what grows is the in-store copy and the startup snapshot that builds it. Budget informer memory per watched type before you tune anything else — narrow the cache with label and field selectors, namespace scoping, and managedFields stripping, in that order. Second, the event to plan for is not "controllers running" but "many controllers starting at once": after a control-plane upgrade, a node drain, or a rollout of your own operator fleet, every manager relists simultaneously and the burst concentrates in WaitForCacheSync. Stagger rollouts of controller managers the way you would stagger anything else with a cold-start cost. Third, resync will not save your infrastructure state. A ten-hour synthetic re-emit catches nothing about a Hetzner server that changed out-of-band; infra drift needs its own recheck loop with real reads (APIReader), not a shorter SyncPeriod.

The tuning checklist​

The deep dive's closing line is the right one to carry back to the fleet: the cache is not an optimization, it is the operating model — Reflector plus queue plus Indexer, the same primitives Kubernetes itself runs on. A self-hosted PaaS inherits that model wholesale the moment it builds on Cluster API, which means its scaling story is the cache's scaling story: quiet at rest, expensive to warm, and tunable exactly where the table says. Defaults carry you to your first hundred Machines. Past that, every informer is a line item in capacity planning — measure the snapshot, scope the watch, index the query, and the controllers go back to making their eleven calls and saying nothing more.

Sources​

• "How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server" — Kubernetes blog, July 29, 2026 — kubernetes.io/blog/2026/07/29/controller-runtime-cache-explained
• Independent request-counting experiment: a trivial controller's fixed ~11-request startup profile; cache warm-up as the cost, WaitForCacheSync where it shows — dev.to/kirponik
• Contour control-plane OOMKilled on startup by an unfiltered cluster-wide Secret list decoding accumulated helm release Secrets before Transform runs — projectcontour/contour#7660
• cass-operator OOMs in cluster-scoped mode: unbounded informer cache scaling with cluster size — k8ssandra/cass-operator#947
• Single-RWMutex store contention at scale in kube-controller-manager — kubernetes/kubernetes#130767
• Label-based resource discovery narrowing the ECK operator cache via ByObject selectors — elastic/cloud-on-k8s#9359

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. A Cluster-API fleet with quiet controllers is exactly what that push lands on. Star the repo on GitHub or deploy your first app today.