Kubernetes 1.36 Teaches Controllers to Distrust Their Own Cache: Why a Skipped Sync Beats a Wrong Reconcile

Every Kubernetes controller works from a copy of the world, not the world itself. It keeps a local informer cache, reads from it because reading is fast, and reconciles the difference between desired and actual state. That copy is usually milliseconds behind reality. Usually, milliseconds don't matter. But when a controller writes to the API server and then reads back a cache that hasn't caught up to its own write, it acts on a world that no longer exists — creating pods that are already there, fighting updates it just made, or deciding a rollout is healthy based on yesterday's pod count.

Kubernetes 1.36, released April 22, 2026, teaches four of the busiest controllers to distrust their own cache. If the cache is behind the controller's last write, the controller skips the sync instead of reconciling from stale data. A skipped reconcile reconverges seconds later. A wrong one can corrupt state. That is the whole thesis: a skipped reconcile beats a wrong one, and on a small self-hosted fleet the wrong one costs more than it does on a hyperscaler's control plane. The table below is the complete shipment; the rest of the post substantiates every row.

What "stale" concretely means​

An informer cache is populated by a list from the API server followed by a watch stream of changes. Every object carries a monotonically increasing resourceVersion. The controller's cache therefore always has a "latest version seen" — and the controller also knows the resourceVersion of everything it has written itself.

Staleness is the gap between those two numbers pointing the wrong way. Walk through it: the ReplicaSet controller creates a pod and the API server accepts it at resourceVersion 1042. The controller's watch hasn't delivered that event yet, so its cache still ends at 1039. The next sync reads the cache, counts fewer pods than desired, and creates another pod that was never needed. The old code had partial guards against exactly this — the ReplicaSet controller's long-standing expectations counter refuses to create pods until expected create events arrive or a timeout fires — but each guard was hand-rolled per controller, and a timeout expiring still meant acting blind.

The 1.36 mechanism generalizes the guard. Before acting, the controller asks one question: is my cache at least as new as my last write? Concretely, the new ConsistencyStore records every write with WroteAt, answers the question with EnsureReady, and cleans up deleted objects with Clear:

// Simplified from the client-go ConsistencyStore contract
store.WroteAt(rs, rs.UID, rsGroupResource, "1042")
if !store.EnsureReady(namespacedName) {
    // cache hasn't caught up to our own write: skip, requeue, try later
    return reconcile.Result{Requeue: true}, nil
}
// cache is at least as new as our last write: safe to act

The ReplicaSet implementation tracks two things, not one: the ReplicaSet's own resourceVersion and the latest written version of the pods it owns. Either one ahead of the cache means the view is stale and the sync is skipped. Note the failure mode this does not fix: a cache that is behind because nobody wrote anything recently is just normal watch latency, and the controller proceeds — the check is specifically "behind my own writes," which is the case where acting is provably working from self-contradictory data.

Underneath, 1.36 also hardens the queue feeding the cache. The AtomicFIFO processing change makes batch arrivals — notably the initial list that populates an informer — apply atomically, so out-of-order delivery can't leave the cache in a state that never existed on the server. And LastStoreSyncResourceVersion() gives any client-go consumer a direct readout of how new its cache is, which is the foundation the controller-manager features are built on.

The failure it prevents​

The upstream post names three harm classes, and each maps to a concrete fleet event:

Wrong action. The controller does something the current state doesn't warrant: extra pods from the example above, a scale-down that deletes pods a newer write already replaced, two writers conflicting until one overwrites the other. The release announcement calls out conflicting updates and data corruption explicitly — this is the class the feature exists for.

Missed action. The controller should act and doesn't, because its stale view says everything is fine. A node that already recovered still shows pressure; a failed pod still shows running. Nothing breaks loudly; the system just sits wrong until the cache catches up.

Slow action. The controller eventually does the right thing but only after fighting itself — create, conflict-retry, re-list, recreate — turning a one-write operation into a minutes-long flap. These are the incidents that show up as "the controller took too long" in postmortems where every individual log line looks correct, because the logic was correct and the input data wasn't.

All three share the property that makes staleness incidents expensive to debug: the controller logs show correct logic executed against incorrect inputs. The 1.36 observability half of the shipment addresses exactly that. stale_sync_skips_total counts every skipped sync per controller, so a spike after an upgrade is visible rather than inferred. store_resource_version exposes each informer's latest seen version with Group, Version, and Resource labels, so an operator can compare an informer's position against the API server's and see which cache is lagging instead of guessing from symptoms.

Why a small CAPI fleet hurts more​

A hyperscaler absorbs an occasional stale reconcile with headroom a small fleet lacks. That decomposes into four concrete asymmetries:

The third row is the one that converts an abstract correctness feature into money. On rented-by-the-hour bare metal, "create then quietly reap" is not free the way it is inside a hyperscaler's spare pool. A MachineDeployment controller that double-provisions because its cache missed its own scale-up write has bought a server, not a pod.

The Cluster API project agrees with this reading. CAPI v1.14 (alpha.0 and beta.0 changelogs) ships PR #13720: stale-controller mitigation in the generic reconcile wrapper and the MachineDeployment controller, plus the store and FIFO informer metrics enabled. That is the exact generalization path upstream sketched — kube-controller-manager's four controllers first, then the wider controller-runtime ecosystem via controller-runtime PR #3473, which aims to give every controller-runtime controller the same read-your-own-writes semantics without hand-rolling the logic. CAPI controllers are controller-runtime controllers, so the wrapper-level adoption covers the whole project at once rather than one controller at a time.

Operator checklist for the 1.36 upgrade​

1. Leave the gates on. The four StaleControllerConsistency* gates default to enabled. You need a measured reason, not a precautionary instinct, to turn one off — the skip-and-requeue behavior is strictly safer than the act-on-stale behavior it replaces.
2. Know the off switch anyway. Per-controller granularity means you can disable one controller's gate (StaleControllerConsistencyDaemonSet=false) while keeping the rest, which is the right blast radius if a specific workload behaves oddly after upgrade.
3. Alert on stale_sync_skips_total. A low background rate is healthy — it means skips are absorbing would-be wrong reconciles. A sustained spike in one controller after an upgrade means that controller's cache chronically lags its writes: investigate watch latency and API-server load, not the gate.
4. Compare store_resource_version against the API server. The per-informer version metric turns "is the cache stale?" from a hypothesis into a subtraction. Dashboard it per controller before you need it during an incident.
5. Track CAPI v1.14 and controller-runtime #3473. The kube-controller-manager coverage protects pods; your machines are protected when your CAPI version includes the reconcile-wrapper adoption. Custom operators you run get the same protection when the controller-runtime work lands — until then, ConsistencyStore is available to informer authors directly, with the ReplicaSet informer PR as the reference implementation.

The direction of travel is clear: read-your-own-writes is becoming default controller semantics rather than a per-controller folk remedy like the old expectations counter. Small fleets should want this more than anyone — they have the fewest redundant views, the slowest reconvergence under load, and hardware where every wrong reconcile has an invoice attached.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.