How the controller-runtime Cache Actually Works: What PaaS Builders Should Know Before Their First OOMKill

Here is the fact that surprises almost everyone who writes their first Kubernetes operator: when your reconciler calls r.Get(ctx, key, &configMap) to read one ConfigMap, that call never touches the API server. Instead, the first read of any Kind silently starts an informer that does a full List of every ConfigMap in the cluster, opens a long-lived Watch stream, and keeps the entire object set in your operator's memory for the life of the process. You asked for one object; you are now holding all of them.

The Kubernetes project spelled this out in a July 29, 2026 deep dive on the official blog — "How the controller-runtime Cache Actually Works" — and its warning is blunt: a controller that "just reads" can quietly consume gigabytes of memory, perform hidden O(n) scans, and trip over stale reads. For anyone building a platform whose control plane is a set of controllers — deploy controllers, TLS controllers, custom-domain controllers, Cluster API machine controllers — this is not trivia. The default cache behavior is the difference between a control plane that fits in a few hundred megabytes on a modest self-hosted node and one that gets OOMKilled the first time the fleet grows. The good news: every part of it is configurable, and this post walks through exactly how.

Get() Never Hits the API Server​

controller-runtime's manager wires your reconciler's client to a cache-backed reader built on the same shared-informer machinery that powers Kubernetes itself. The lifecycle looks like this:

1. The first time your code reads a Kind — any Get or List — the cache lazily starts an informer for that GroupVersionKind.
2. The informer performs an initial List of all objects of that Kind (cluster-wide, unless you scope it) to warm the cache.
3. It then holds a Watch stream open, applying incremental updates so the in-memory store stays current.
4. Every subsequent Get/List for that Kind is served from local memory, indexed by namespace and name — effectively free, no network round trip.

Writes are the opposite: Create, Update, Patch, and Delete always go straight to the API server. The cache is read-only and updated asynchronously by the watch stream. Annotated, a typical reconciler actually does this:

func (r *AppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var app platformv1.App
    // Cache read. First call ever for this Kind? Informer starts,
    // lists EVERY App in the cluster, and caches them all.
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
 
    var deploys appsv1.DeploymentList
    // Also a cache read — and it just made your operator cache
    // every Deployment in the cluster, not only this app's.
    if err := r.List(ctx, &deploys, client.InNamespace(req.Namespace)); err != nil {
        return ctrl.Result{}, err
    }
 
    // Direct API-server write. The cache does NOT update synchronously;
    // an immediate re-Get may still return the old object.
    return ctrl.Result{}, r.Status().Update(ctx, &app)
}

Two consequences follow immediately. First, eventual consistency: a read right after your own write can return stale data, because the cache only updates when the watch event arrives. Reconcilers must be written to converge, not to assume read-your-writes. Second, and the subject of the rest of this post: the size of your cache is decided by what types you touch, not by what objects you need.

This design is why Kubernetes scales at all — thousands of controllers polling the API server directly would melt it. The list-watch pattern trades API-server load for controller memory. The trade is a good one; the problem is that most operator authors don't know they made it.

Where the Memory Actually Goes — and What the OOMKill Looks Like​

Do the arithmetic for a realistic multi-tenant platform cluster and the danger gets concrete. Serialized Kubernetes objects are not small: a Pod with a few containers, env vars, volumes, and status typically weighs in around 8–15 KB of JSON, and managedFields — the server-side-apply bookkeeping almost no controller reads — routinely accounts for a third or more of that. A rough model for an unscoped control plane on a busy cluster:

That's the serialized size. In memory you pay more: decoded Go structs, informer index maps (by namespace, by name, plus every field index you add), and a deep copy of each object handed to your code on every read so you can't corrupt the cache. Multiply by the number of Kinds your controllers touch — a platform control plane easily watches ten or more — and an "it just reads things" operator lands at 1–2 GB RSS without a single leak.

The failure mode has a very specific shape. You set a sensible-looking memory limit — say 512Mi — on your operator's Deployment, because it's "just a controller." It runs fine for months. Then the fleet crosses a threshold: a tenant scales to hundreds of Pods, a CI system starts churning Secrets, or you add one innocent r.List of a new Kind. On the next restart the informers begin their initial sync, RSS climbs past the limit during cache warm-up, the kubelet OOMKills the container, and it restarts into the exact same sync — CrashLoopBackOff, with the control plane down at precisely the moment the cluster got busy. Nothing in your code changed. The cluster grew into your cache.

There's a CPU-shaped cousin of the same problem: an unindexed List with a filter is an O(n) scan over the cached set. r.List(ctx, &pods, client.InNamespace("x")) looks like a targeted query but can iterate tens of thousands of cached objects on every reconcile unless the field is indexed. At fleet scale, that's your reconcile latency budget gone.

The Trimming Toolkit​

Everything above is the default, not the destiny. controller-runtime exposes precise levers for scoping what gets cached, and they compose. All of them hang off cache.Options (passed to the manager) and client.Options.

1. Label and field selectors — cache only what's yours. The single highest-leverage move for a platform: label every object your controllers create, and select on it. The informer's list-and-watch then only ever sees matching objects; everything else never enters memory.

managed, _ := labels.Parse("app.kubernetes.io/managed-by=myplatform")
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{
        DefaultLabelSelector: managed,          // applies to every Kind
        ByObject: map[client.Object]cache.ByObject{
            &corev1.Node{}: {Label: labels.Everything()}, // per-Kind override
        },
    },
})

DefaultFieldSelector and per-Kind ByObject.Field work the same way for field-based scoping (e.g., spec.nodeName on Pods).

2. Transform functions — shrink what you do cache. Transforms run before an object is committed to the store. The built-in cache.TransformStripManagedFields() deletes the managedFields block and is close to free memory savings — often 30%+ per object — for any controller that doesn't read server-side-apply metadata (almost all of them):

Cache: cache.Options{
    DefaultTransform: cache.TransformStripManagedFields(),
},

Custom transforms can go further: strip huge annotations (kubectl.kubernetes.io/last-applied-configuration is a classic offender), drop status fields you never read.

3. Namespace scoping. DefaultNamespaces and per-Kind ByObject.Namespaces restrict informers to the namespaces you actually operate on — the right tool when your platform confines tenant workloads to known namespaces.

4. DisableFor — don't cache the scary Kinds at all. Secrets are the canonical case: high count, potentially large, sensitive to hold in every controller's memory, and often read once per reconcile anyway. The client can be told to always do a live lookup:

Client: client.Options{
    Cache: &client.CacheOptions{
        DisableFor: []client.Object{&corev1.Secret{}},
    },
},

Reads of listed types hit the API server directly; no informer ever starts for them.

5. Metadata-only watches. When you only need names, labels, and ownership — say, garbage-collecting children — watch metav1.PartialObjectMetadata instead of the full Kind. You keep the watch semantics and pay ~1 KB per object instead of 12.

6. mgr.GetAPIReader() — the escape hatch for consistency. For the rare read that must be strongly consistent (checking state immediately before an irreversible action), the API reader bypasses the cache entirely for a one-off live call — without starting an informer.

7. Field indexes — fix the O(n) List. Register an index for any field you filter on, and cached Lists become map lookups:

mgr.GetFieldIndexer().IndexField(ctx, &corev1.Pod{}, "spec.nodeName",
    func(o client.Object) []string {
        return []string{o.(*corev1.Pod).Spec.NodeName}
    })

Scoping a PaaS Control Plane​

Now apply this to the concrete case the title promised: a self-hosted platform whose control plane is a set of cooperating controllers — a deploy controller turning git pushes into Deployments and Services, a TLS controller managing certificates, a custom-domains controller reconciling Ingress objects, and Cluster API controllers managing the machines underneath. (Bex.co, the open-source Render alternative, is exactly this shape: its control plane is controllers reconciling App resources into running workloads on machines you own.) The whole point of such a platform is running on modest owned hardware — which makes an unscoped multi-gigabyte control plane a self-inflicted wound.

A scoping policy for that topology:

• Label everything the platform creates (app.kubernetes.io/managed-by: <platform>) and set DefaultLabelSelector to match. The deploy controller now caches the Deployments it made — not the monitoring stack's, not the tenants' hand-rolled ones. This is the step that converts cache size from "proportional to the cluster" to "proportional to the platform's own footprint."
• Strip managedFields globally with DefaultTransform. No platform controller reads them; it's a one-line ~30% haircut on every cached object.
• Metadata-only watches for high-cardinality tenant objects. The TLS controller that only needs to know which Ingresses exist and their annotations can watch PartialObjectMetadata and drop an order of magnitude of memory.
• DisableFor Secrets. Certificate private keys and tenant env-var Secrets are read at reconcile time, live. No controller holds every tenant's credentials in RAM as a side effect of the cache.
• Index the hot Lists — "Deployments owned by this App," "certificates for this domain" — so reconciles stay O(1) as the fleet grows.
• Then set the memory limit deliberately. Post-scoping, measure RSS after a full informer sync at current fleet size, model growth per tenant (it's now roughly linear in platform-managed objects, at kilobytes each), and set requests/limits from the model — instead of guessing 512Mi and finding out at 2 a.m.

The before/after is stark: the same controllers that idle at 1–2 GB unscoped on a 10,000-Pod cluster typically fit in 150–300 MB once they cache only what they own — comfortable margin on a small Hetzner node running the entire control plane.

Cache Scoping Is Capacity Planning​

The controller-runtime cache is one of the best default trades in infrastructure software — near-free reads and no API-server meltdown, paid for in controller memory. But it's a trade made silently on your behalf, per Kind, cluster-wide, the moment you call Get. The Kubernetes blog's July 2026 deep dive deserves credit for saying plainly what operator authors used to learn from their first production OOMKill: your controller's memory footprint is not a property of your code; it's a property of your cluster, unless you scope it.

For platform builders the lesson generalizes. A control plane that will run on other people's modest hardware — the whole premise of self-hosted PaaS — has to treat cache configuration as part of its architecture, not a tuning afterthought. Label what you own, select on it, strip what you don't read, refuse to cache what you shouldn't hold, and index what you filter. Do that, and "operator memory" becomes a number you compute before the fleet grows — not one the kubelet computes for you.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, orchestrated by a control plane built to stay small. Star the repo on GitHub or deploy your first app today.