The Informer Tax: What Every controller-runtime Watch Costs on a 4GB Hetzner Box

Your controller isn't slow. It's holding the whole cluster in RAM and pretending it isn't.

The July 29, 2026 Kubernetes blog post — How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server — makes one idea non-negotiable before any other detail: r.Get() and r.List() inside a reconciler do not hit the API server. They read from a local in-memory copy that the manager warmed with a single list and keeps current through a watch. Every other property of the system — cheap reads, stale reads after writes, invisible memory pressure, silent O(n) scans — follows from that fact.

On a laptop it doesn't matter. On the CX22 (2 vCPU, 4 GB RAM) or CPX21 that teams run a Cluster API management cluster on, it is the capacity plan. One extra GVK watch can move the node from fine to OOMKilled without a single new tenant.

If you only remember one table from this post, remember this one:

The rule: one GVK, one informer, one full copy in memory for every matching object — forever. A manager with five watched types on a 4 GB box handles a few thousand light objects per type, but one unscoped Secret watch against 10k Helm releases blows the budget alone. The rest of this post is how to stay on the right side.

The cache is not an optimization. It is the operating model​

Most Go developers assume r.Get(ctx, key, &pod) fires an HTTP GET. At hundreds of reconciles per second across a dozen controllers, that would drown the API server and etcd in minutes.

Kubernetes avoids polling. The client takes one snapshot and subscribes to a change stream — list + watch. There is no poll loop.

In controller-runtime three client-go primitives do the work so you don't wire them yourself:

• Reflector — the only piece that talks to the API server. It fetches the initial snapshot (now a streaming list with sendInitialEvents=true, falling back to a plain list) and then holds a watch from the returned resourceVersion. If the watch drops it reconnects from the last version; on 410 Gone it re-lists from scratch.
• Delta queue — between reflector and informer. Since client-go 1.36 this is RealFIFO: a flat, strictly ordered slice of deltas, one Pop() per delta, no deduplication, global ordering. The old DeltaFIFO map keyed by namespace/name is gone. Order is preserved globally; nothing is collapsed — deduplication now lives only in the controller's workqueue.
• Indexer (Store) — the local copy. A map[string]interface{} keyed by namespace/name, a single sync.RWMutex, and a dictionary of inverted indexes you register. An uncontended r.Get is a map lookup plus a DeepCopy.

SharedIndexInformer fuses them: read from the indexer, or subscribe a handler (OnAdd/OnUpdate/OnDelete). The manager creates one informer per GVK shared by every controller in the process — one snapshot, one watch per type regardless of how many reconcilers use it.

At mgr.Start(ctx) every informer warms before any Reconcile runs. Snapshot loaded, indexes rebuilt, marked synced — only then workers drain the queue. The one exception: a Get for an unwatched type starts a new informer and blocks until warm.

The composite client.Client splits the path: reads (Get, List) → memory, writes (Create, Update, Patch, Delete) → API server, with feedback via the watch. Between Update and cache convergence there is a window — usually milliseconds, no guaranteed bound — where r.Get returns the previous resourceVersion. That is eventual consistency after optimistic concurrency (409 Conflict if you raced). Reconcile must be idempotent and never rely on read-after-write.

Bill of materials: what actually lives in memory​

An entry is not a pointer to the API server. It's a decoded Go struct, deep-copied into the store for the process lifetime. Per watched GVK:

• One Reflector goroutine + HTTP/2 watch stream
• One RealFIFO / DeltaFIFO queue of pending deltas
• One ThreadSafeStore map with every matching object plus one sync.RWMutex guarding the store and all indexes together
• One set of inverted indexes (namespace + every IndexField you registered)
• Deep copies handed out on every Get/List (unless you opt into UnsafeDisableDeepCopy)

Unit cost is per-object × count. Query cost is dominated by that single RWMutex: a List holds the read lock while walking every object; informer writes need the exclusive lock. The two contend. This bottlenecked kube-controller-manager at scale (kubernetes#130767) and recent client-go narrowed the write-lock window — but an unindexed List still walks every object under the lock, O(n) per reconcile.

Predicates don't cut memory — they filter whether a key is enqueued; the informer still caches the object. Transforms do — they run before the object enters the store.

Where a cheap Hetzner box actually OOMs​

A CX22 has 4 GB total, roughly 2.5–3 GB usable after kubelet, etcd, and the API server. A CAPH management cluster adds Cluster API, CAPH, and custom git-push controllers — all sharing the manager's cache.

Measured field data from July 2026:

• Managed fields are 20–25% of cached bytes. Argo Workflows measured them as a fifth to a quarter of every object; KEDA reports ~90% per-Pod saving when stripping managedFields plus unused Pod fields (keeping only Status.Phase, Status.Conditions, Spec.NodeName). TransformStripManagedFields() is loss-free — the API server ignores nil managedFields on updates.
• Helm release Secrets are the classic killer. Each helm.sh/release.v1 blob is 50–150 KB. Contour #7660 documents a transient OOM at startup: an unfiltered cluster-wide List spikes past 512 Mi before any Transform runs on tens of thousands of accumulated releases. Fix: don't cache Secrets cluster-wide.
• ConfigMaps can be worse. The Spark Operator audit found 700 ConfigMaps at 900 KB each = 630 MB of operator working set. At 10k it is 9 GB before any other GVK.
• Nodes carry status.images — every image ever pulled, tens of KB per node. Trivial for 5 nodes, noticeable if you watch Nodes cluster-wide.

At 5k tenants × 2 objects per tenant, tens of thousands of objects is already the working set. At 20 KB average that is ~200 MB; at 80 KB with Helm releases it is ~800 MB. Add indexes and deep-copy churn and the CX22 has no headroom left for compaction, a relist after 410 Gone, or an image pull.

Five levers that actually cut the Informer Tax​

All of this lives in cache.Options at manager construction. It is manager-scoped — every controller in the binary sees the same filtered world. Tighten for one and a sibling that needed the same GVK cluster-wide stops seeing objects. Audit readers before you narrow.

1. Scope by namespace​

If a controller only manages its own namespace, don't cache the rest:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
  Cache: cache.Options{
    ByObject: map[client.Object]cache.ByObject{
      &corev1.Secret{}: {
        Namespaces: map[string]cache.Config{
          "my-paas": {},
        },
      },
    },
  },
})

2. Push selectors into the watch​

Label and Field selectors become parameters of the watch itself. The API server only sends matching objects, saving both network and memory. This is different from MatchingLabels on a List, which is evaluated locally and still walks every cached object.

&corev1.Secret{}: {
  Label: labels.SelectorFromSet(labels.Set{
    "app.kubernetes.io/managed-by": "bex",
  }),
},

Set DefaultLabelSelector / DefaultNamespaces when every type needs the same restriction.

3. Strip what you don't read​

Transform runs before the object enters the store. Always strip managedFields; also drop spec/data you never read:

Cache: cache.Options{
  DefaultTransform: cache.TransformStripManagedFields(),
  ByObject: map[client.Object]cache.ByObject{
    &corev1.Pod{}: {
      Transform: func(obj any) (any, error) {
        pod := obj.(*corev1.Pod)
        pod.ManagedFields = nil
        // KEDA pattern: keep only what the reconciler reads
        // Containers, volumes, env, annotations are cleared if unused
        return pod, nil
      },
    },
  },
},

Cluster API's ClusterCache per-workload-cluster caches had no DefaultTransform until cluster-api#13779 — verify your CAPH version or pay the tax once per workload cluster.

4. Don't cache the object at all​

When you only need existence or labels, cache metadata:

var list metav1.PartialObjectMetadataList
list.SetGroupVersionKind(schema.GroupVersionKind{
  Group: "", Version: "v1", Kind: "Secret",
})
_ = r.List(ctx, &list, client.InNamespace("my-ns"))

The store keeps only ObjectMeta — no Spec/Data/Status. For Secrets that is an order of magnitude. Caveat: you can't filter on spec fields through a metadata watch.

When you rarely read a fat type and never need events, disable its cache entirely:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
  Client: client.Options{
    Cache: &client.CacheOptions{
      DisableFor: []client.Object{&corev1.Secret{}},
    },
  },
})

Reads go to the API server, no informer starts, no startup list, no permanent memory. Also no events — pair with a metadata watch if you need triggers. mgr.GetAPIReader() bypasses the cache ad-hoc.

5. Avoid the deep-copy tax (carefully)​

Get/List deep-copy by default so you can mutate freely. Objects in Predicate/EventHandler are not copied — they are the shared store objects. Mutating without DeepCopy() corrupts the cache for every other controller on that GVK.

If profiling shows deepcopy is hot and you can prove you never mutate the returned object:

Cache: cache.Options{
  ByObject: map[client.Object]cache.ByObject{
    &corev1.Pod{}: {UnsafeDisableDeepCopy: true},
  },
},

The name is the warning — it trades safety for allocation pressure. Use per-GVK, not globally, and only after pprof says copying is hot.

The hidden O(n): MatchingLabels is not an index​

Most controllers eventually write:

var pods corev1.PodList
_ = r.List(ctx, &pods, client.MatchingLabels{"app": "bex"})
for _, p := range pods.Items {
  if p.Spec.NodeName == "node-1" { /* ... */ }
}

That filtered List still walks every cached Pod under the read lock. The selector is checked before the copy — 50k cheap comparisons plus 10 copies beats 50k copies — but the walk is still O(n) and still blocks informer writes.

To make the lookup sub-linear, build an inverted index yourself:

if err := mgr.GetFieldIndexer().IndexField(ctx,
  &corev1.Pod{}, "spec.nodeName",
  func(obj client.Object) []string {
    pod := obj.(*corev1.Pod)
    if pod.Spec.NodeName == "" { return nil }
    return []string{pod.Spec.NodeName}
  }); err != nil {
  return err
}
 
// now this is an index lookup, not a scan:
var pods corev1.PodList
_ = r.List(ctx, &pods,
  client.MatchingFields{"spec.nodeName": "node-1"})

The index name is an arbitrary string — controller-runtime never parses it as JSONPath. The only rule is that IndexField and MatchingFields use the same string. Populate it during the snapshot, so the first Reconcile benefits.

• No index → error, not slow fallback. Equality only — no range/LIKE/sort. For windows, bucket by Truncate(5*time.Minute).

How to measure what each watch costs before prod​

Measure on a staging cluster that has the tenant count you plan to support:

• Heap: go tool pprof -top http://localhost:6060/debug/pprof/heap — sort by inuse_space. Compare before/after TransformStripManagedFields.
• Container memory: container_memory_working_set_bytes — each GVK's initial List is a step of count × avgSize.
• Object counts: kubectl get <kind> -A --no-headers | wc -l plus jq to spot bloat (managedFields, status.images, Helm data).
• Cache size logging: log type → count → bytes after WaitForCacheSync. If the RealFIFO queue grows, the problem isn't query speed.

Audit ladder for a bex fleet:

1. DefaultTransform: cache.TransformStripManagedFields() everywhere — expect 20–25% off.
2. Restrict Secrets/ConfigMaps to needed namespaces; label-select if possible.
3. Replace hot MatchingLabels lists with a MatchingFields index — re-measure p95 Reconcile latency.
4. For counts/existence, switch to PartialObjectMetadata.

Checklist for a Cluster API fleet on Hetzner​

A CX22/CPX21 management cluster is a small computer watching a lot: Cluster, Machine, MachineSet, MachineDeployment, HetznerMachine, plus every native type your platform controllers subscribe to. Each is a full informer.

• Audit cache.Options.ByObject for every controller — including ClusterCache per-workload-cluster caches. Keep server-only registries behind server functions so they never enter the client bundle.
• Set ReaderFailOnMissingInformer: true in development — accidental wide watches fail loudly.
• Predicates don't save memory. Only ByObject constraints, watch-level selectors, Transform, and PartialObjectMetadata do.
• Use RequeueAfter for deferred work, not time.Sleep — it reuses workqueue deduplication without holding a worker.
• If you DisableFor a type, add a metadata watch if the controller still needs events.

The one sentence to remember​

r.Get inside a reconciler reads from memory, not from the API server — not even the first time. The exceptions are the ones you opt into: APIReader, Cache.DisableFor, and PartialObjectMetadata.

Once that is reflex, common review questions answer themselves: why the API server stays quiet under hundreds of reconciles per second, why a stale read after Update is expected, why a Helm-heavy cluster OOMs a fresh operator, and why a filtered List still walks every object until you index it.

The Informer Tax isn't negotiable — you pay it for the watch model that keeps the API server alive. What is negotiable is how much you cache, what you strip, and whether you know the price before tenant count tells you on a Saturday.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.