Your Operator's /readyz Is Lying: The controller-runtime Cache-Sync Trap Behind Green Dashboards and Stalled Reconciles

Every kubebuilder-scaffolded operator ships with this line in main.go: mgr.AddReadyzCheck("readyz", healthz.Ping). It returns 200 the moment the probe server listens — before the informer cache has synced, before leader election completes, before the webhook server serves. Your pod is Ready, your rollout is green, and your controller may not be reconciling anything at all. Here is the fix up front; the rest of this post is why each line exists:

// Ready means: cache synced, this pod leads, webhooks serve. Not Ping.
_ = mgr.AddReadyzCheck("cache-sync", func(_ *http.Request) error {
    ctx, cancel := context.WithCancel(context.Background())
    cancel() // already-cancelled: report current sync state, don't block
    if !mgr.GetCache().WaitForCacheSync(ctx) {
        return errors.New("informer cache not synced")
    }
    return nil
})
_ = mgr.AddReadyzCheck("leader", func(_ *http.Request) error {
    select {
    case <-mgr.Elected():
        return nil
    default:
        return errors.New("not elected leader")
    }
})
_ = mgr.AddReadyzCheck("webhook", mgr.GetWebhookServer().StartedChecker())

Plus one manager option so a cache miss fails loudly instead of stalling silently: Cache: cache.Options{ReaderFailOnMissingInformer: true}.

The trap in 60 seconds​

The scaffold gives you an HTTP health server for free, and free reads as done. But healthz.Ping answers exactly one question — "is this process listening on the probe port?" — while Kubernetes treats /readyz as the answer to three more: should this pod take traffic, should the rollout proceed, should the old ReplicaSet scale down. Everything Ping cannot see becomes a state where the signal is green and the controller is useless:

The official Kubernetes blog's July 2026 deep dive on how the controller-runtime cache actually works documents the mechanics precisely: at startup the manager Lists every registered GVK into informer stores, marks each synced, and only then lets controllers drain the workqueue. Registered informers are safe — workers cannot outrun them. The trap lives everywhere else: webhook handlers that do not wait for sync, Get calls for types nothing registered, and pods that never lead.

How a green rollout hides a dead controller​

The sharpest form of the trap is an RBAC regression riding inside an otherwise perfect release. Walk it through:

1. Your reconciler calls r.Get(ctx, key, &policy) for a NetworkPolicy — a type no controller watches, so no informer was registered at startup.
2. The cached client lazily starts a NetworkPolicy informer on the spot and blocks the Get until the first List completes.
3. The new release's ClusterRole forgot list/watch on networkpolicies. The List is forbidden; the informer retries with backoff forever; the Get never returns.
4. The worker thread is stuck. The queue backs up. No error is logged in the steady state — a blocked call is not a failed call.
5. Meanwhile /readyz answers Ping with 200. The rollout completes, the old ReplicaSet scales down, the dashboard is green. Every reconcile in the fleet is now parked behind a Get that will never return.

This is not hypothetical. The victoria-metrics-operator Helm chart 0.67.0 shipped a ClusterRole without networkpolicies RBAC, and the operator's cached NetworkPolicy read did exactly the above: lazily started an informer that could never sync, and the operator silently stopped reconciling everything.

The invisibility cuts both ways. A 2026 hardening audit of the spark-operator flagged its worst finding as the reverse image of the same mechanism: a cached Get for a type missing from ByObject silently creates a cluster-wide informer on first invocation, with no visible trace in the cache configuration that the informer exists at all. One team gets an informer that never syncs; another gets an informer nobody configured. Both discover it in production, because nothing at startup complains.

A sibling failure mode hit Contour, whose unfiltered Secret informer Lists every Secret in the cluster on startup — including megabyte-sized Helm release blobs — before its Transform strips them: proof that the initial sync is real, expensive work, not an instant handshake, and that a pod marked Ready during it is making promises its cache cannot keep.

Note what makes this a readiness bug and not just an RBAC bug. RBAC mistakes happen; what turns one into a fleet-wide silent stall is a readiness signal with no connection to controller function. A /readyz that actually gated on sync state would have held the rollout at 0/1, paged on the stuck ReplicaSet, and kept the old pods serving. Ping waved it through.

The fix, line by line​

The snippet at the top replaces one meaningless check with three meaningful ones. Each earns its place:

Cache sync, polled without blocking. WaitForCacheSync on an already-cancelled context returns the current sync state immediately instead of waiting, which makes it safe inside a readiness handler that kubelet calls every few seconds. During the initial List window the pod reports NotReady; endpoints and rollout progression wait for the truth. Once every registered informer is synced, the check passes permanently — it costs one boolean poll per probe.

Leadership, from the manager's own channel. mgr.Elected() closes when this pod wins leader election. Gating readiness on it means standby pods stop claiming a readiness they cannot use: with two replicas and leader election, "1/2 Ready" is the honest signal, and a fleet where zero pods are Ready tells you no leader exists instead of showing a comforting 2/2. Use this check only when leader election is actually enabled — without it there is no election to wait for.

Webhook serving, from the server itself. StartedChecker() reports whether the webhook server is up. This closes the cold-cache admission window from the serving side: no traffic until the server that answers it is running, combined with the cache check so its reads land on warm state.

Fail fast on the lazy informer. The three checks cover informers the manager knows about. The victoria-metrics stall came from one it did not — started implicitly by a Get. ReaderFailOnMissingInformer: true in cache.Options turns that implicit start into an explicit error at the call site: loud in staging, greppable in code review, instead of a blocked goroutine in production. Red Hat's operator cache configuration guide recommends exactly this default for new projects, for exactly this reason.

The gotchas in the fix​

No readiness scheme is free, and this one has three sharp edges worth naming before you paste it:

First, leader-gated readiness concentrates webhook traffic. Standby pods run webhook servers that could answer admission requests, but NotReady removes them from Service endpoints, so the leader takes all webhook load. For most platform controllers that is correct — one leader's webhooks are plenty — but if your admission path is hot, gate only the cache and server checks on every pod and keep leadership out of the serving path.

Second, the cache check is point-in-time and registration-scoped. It verifies the informers registered when the manager started; it cannot see an informer started lazily five minutes later, and it cannot predict an RBAC revocation next Tuesday. That is why the fail-fast flag is not optional garnish — the checker and the flag cover disjoint halves of the problem.

Third, rollout strategy must match the new honesty. An HA Deployment with leader-gated readiness will never show N/N Ready; set maxUnavailable: 0 with a surge replica so rollouts replace pods one at a time instead of stalling on an unreachable Ready count. And keep alerting on workqueue depth and reconcile latency — readiness tells kubelet the truth, but only controller metrics tell you the controller is actually progressing.

Audit your controllers this week​

This trap ships invisibly because every step looks like the default. Four greps find it:

• AddReadyzCheck.*Ping in every main.go — each hit is a readiness endpoint asserting nothing.
• Every Get/List call on a GVK with no corresponding Watches(), Owns(), or ByObject entry — each is a lazy informer waiting for an RBAC gap.
• Every ClusterRole's list/watch verbs diffed against every GVK the controller reads — the exact diff that sank the victoria-metrics release.
• Every Deployment with replicas > 1 plus leader election — decide what Ready means for standbys before your dashboard decides for you.

Pair the fixes with the memory half of the story — a companion post here walked through how the shared-informer cache OOMKills control planes at fleet scale and the selectors that shrink it. Readiness and memory are the two ways the same cache bites: this post covers the silent stall, that one the loud crash. A platform whose control plane is a set of controllers — deploy controllers, TLS controllers, Cluster API machine controllers — needs both halves closed before the fleet grows.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Its control plane is controllers all the way down, which is why its readiness endpoints check sync state instead of Ping. Star the repo on GitHub or deploy your first app today.