Dora NodaOne client per secret worked until it didn't. That is the short version of Infisical's June 2026 postmortem, "Our Kubernetes Operator Didn't Scale, So We Rebuilt It": their v1 Kubernetes operator gave every InfisicalSecret resource its own authenticated API client, which was fine at ten secrets and a cascading failure at hundreds. The operator pod's memory ballooned, it OOM'd, and every restart fired a simultaneous re-authentication burst from every resource at once — straight into rate limits.
The fix was not a bigger memory limit, more jitter, or cleverer event handlers. Infisical evaluated all of those and rejected them. The fix was a new architecture: three small CRDs instead of one monolithic one, and one cached client per machine identity instead of one per resource. If you run controllers that watch hundreds of objects on a Cluster API management cluster — provider controllers reconciling Machines, or your own platform controllers — this postmortem is worth reading as a mirror. The same latent bugs are sitting in many controller stacks right now, waiting for fleet growth to turn them into API-server load.
This post walks through what broke, what fixed it, and the concrete audit those lessons imply for your own controllers.
What broke: the v1 monolith
In v1, users wrote a single monolithic InfisicalSecret resource on the v1alpha1 API. Each one carried everything the sync needed: the address of the Infisical instance, the authentication credentials, the scope to pull from, and the managed Kubernetes secret to write to. One object, four jobs — connect, authenticate, find secrets, reconcile them into native Secret objects.
That factoring works on persistent infrastructure. A VPS or VM is one identity that reauthenticates on restart or config change. A Kubernetes cluster is dozens or hundreds of these resources that redeploy and restart constantly. Because each resource carried its own auth and connection, each held its own independent client in memory. Infisical names three concrete consequences:
Memory footprint scaled with resource count, then OOM'd. Every resource's client lived in the operator pod's memory. As resources proliferated, the fix was raising Helm memory limits — until that stopped being a fix. And each OOM made the next failure worse: when the pod restarted, every resource reconciled and reauthenticated at the same time.
Restarts produced an auth burst that hit rate limits. Hundreds of simultaneous authentication calls tripped the API's rate limits. Backoffs and retries eventually got the cluster to steady state, but slowly — every restart meant a latency penalty before secrets were fresh.
Rotation meant editing every resource. Changing the Infisical host or rotating a machine identity required editing the authentication block on every single InfisicalSecret. That is toil that scales linearly with adoption, the worst kind.
The sentence that matters most in the whole postmortem is this one: the fundamental issue was the overloaded CRD architecture, not missing logic. Event handlers and jitter might have helped at the margin, but they would have added complexity to an already overloaded CRD. So Infisical rebuilt instead of patching.
What fixed it: the v2 reference split
The v2 design, shipped on the v1beta1 API, separates connection, authentication, and sync into three CRDs, explicitly modeled on External Secrets Operator's resource split:
InfisicalConnectiondefines the address of an Infisical instance plus optional TLS settings.InfisicalAuthdefines the authentication details for one machine identity and references a connection.InfisicalStaticSecretdefines a sync — sources to pull from, targets to write to — and references an auth resource. It replacesInfisicalSecret.
Connection and auth objects are defined once. Secret resources point at them. The controller resolves each secret's auth reference to a client cached by identity, so there is one client per identity, not one per resource. The cache is lazy and invalidates on config changes or when a call comes back 401 or 403, at which point the operator drops the cached client and reauthenticates.
The payoff maps one-to-one onto the v1 failure modes. A restart produces one authentication call per identity instead of one per resource. Rotating a host or identity means editing one object. And clients are no longer replicated across every resource, which shrinks the memory footprint back down.
Two details are worth noting. First, Infisical is explicit that they could have fixed the memory and auth problems by deduplicating clients internally by identity, without touching the API. They split the CRD anyway, for developer experience: the new shape mirrors a pattern operators already know from ESO, and it kills the mass-editing problem that internal dedup would have left in place. Second, they explain why they built their own operator rather than depending on ESO: ESO had previously paused development, and Infisical wants a native UX with fewer moving parts plus future dynamic secrets and push secrets that ESO doesn't support. Both v1alpha1 and v1beta1 are maintained today, with v1alpha1 slated for eventual deprecation and a migration guide in the docs.
The v2 extras are small but telling: a static secret can now pull from more than one source path, write to more than one target (a Secret, a ConfigMap, or both — handy for sidecars that only accept ConfigMap mounts), and every resource reports a readiness status so kubectl get shows health directly. Per-object status conditions are cheap and every operator should have them.
Which failure modes generalize to CAPI-scale controllers
Infisical's operator syncs secrets, not machines. But every failure mode above is really a controller-runtime failure mode wearing a secrets costume. Here is the mapping to a Cluster API management cluster, where provider controllers watch hundreds of Machine, MachineDeployment, and infrastructure-machine objects:
| Infisical v1 failure | Generalized controller failure | CAPI-fleet analogue |
|---|---|---|
| One API client per resource | Per-object external clients (cloud SDK sessions, token caches) that scale with watched objects | A provider controller minting one Hetzner/cloud client per Machine instead of sharing one per credential |
| Restart re-auth burst vs rate limits | Thundering herd on controller restart: every object reconciles at once against a rate-limited API | Management-cluster controller restart hammering the cloud API or etcd-backed API server simultaneously |
| OOM → mass reconcile → OOM loop | Memory proportional to object count with no ceiling, where the crash amplifies the load that caused it | Operator memory growing with Machine count until the OOM crash itself becomes the load spike |
| Mass-edit rotation toil | Identity/endpoint config embedded in every object instead of referenced from one | Credentials duplicated across infra-machine templates instead of one referenced secret |
| (Implied by the fix) unindexed lookups | List-and-filter in the reconcile path instead of field indexes, turning each reconcile into a full cache scan | Resolving "which Machines belong to this deployment" by listing all Machines on every event |
| (Implied by the fix) cache bypass | Reads that miss the informer cache and degrade into live API-server reads under load | Direct (non-cached) client reads in a hot path, multiplying API-server QPS with object count |
| Hot-loop requeues | Reconcile errors or status churn that requeue immediately, spinning the workqueue | A provider reconcile that errors before backoff on every Machine, every few seconds, forever |
The controller-runtime facts underneath this table are worth stating plainly. The client your controller gets is a delegating client: reads are served from a shared informer-backed cache, writes go straight to the API server. That cache is what stands between your fleet and etcd — but only if your reads actually use it, and only if your lookups hit indexes instead of scanning. An unindexed list call inside a reconcile function is a full-cache scan that runs once per object per event. A read that bypasses the cache is a live API call that runs once per object per event. Multiply either by hundreds of Machines and a restart that requeues all of them, and you have Infisical's auth burst with a different victim.
The concrete audit for your own controller stack
Run these checks before fleet growth runs them for you. Each one is a question with a measurable answer:
- Count clients per identity, not per object. In every controller you own or vendor, find where external API clients are constructed. If a client (cloud SDK session, token, connection pool) is created per reconciled object rather than cached per credential/identity, you have Infisical's v1 bug. The fix is a lazily-populated cache keyed by identity with invalidation on config change and on 401/403 — exactly the shape Infisical shipped.
- Measure your restart burst. Restart each controller and watch external-API QPS and API-server request latency in the first 60 seconds. If every watched object reconciles simultaneously and your p99 latency craters or you trip rate limits, you need staggered resyncs, rate-limited workqueues with real backoff, or fewer objects per controller — not a bigger rate limit.
- Verify every lookup path hits an index. Grep your reconcile paths for list calls followed by in-memory filtering. Each one needs a field index (
mgr.GetFieldIndexer().IndexField) or a rethink. "List all Machines, keep three" is fine at ten Machines and a full-cache scan per event at a thousand. - Confirm hot-path reads use the cached client. Any read through a direct (non-cached) client in a reconcile loop is a live API-server call per object per event. Cached reads should be the default; direct reads need a written reason (read-after-write consistency is the usual legitimate one).
- Bound your requeue rates. Check what happens when reconcile fails: exponential backoff with a cap, or immediate requeue? A controller that requeues immediately on a persistent error is a hot loop — one object's failure becomes constant API-server load, and N objects' failures become N times that. Alert on workqueue depth and reconcile error rate, not just on crashes.
- Plot operator memory against object count. If memory grows linearly with watched objects with no plateau, find the per-object retention (clients, caches, informers without scoping) before the OOM writes the incident report for you. Infisical's lesson: raising the memory limit is a deferral, not a fix.
- Factor identity out of per-object specs. If rotating a credential or endpoint requires editing N objects, your CRD is overloaded the way
v1alpha1was. Split connection/auth/config into referenced objects once, the way ESO'sSecretStore/ClusterSecretStoreand Infisical's v2 do. Future-you, rotating credentials at 2am, will be grateful.
Items 1, 6, and 7 are the ones Infisical actually hit. Items 2 through 5 are the same bug class wearing API-server clothes — the generalized modes this post set out to map — and they are the ones that bite CAPI management clusters first, because the management cluster's API server is the shared dependency every provider controller pounds at once.
Split the CRD or dedupe internally?
Infisical's most reusable decision is the one they almost didn't have to make. Internal client dedup would have fixed memory and auth bursts without an API migration. They split the CRD anyway, because the monolith was also a UX bug: mass-editing hundreds of objects to rotate one identity is operational toil no internal cache can fix.
That is the rule to take home. When a controller doesn't scale, ask whether the scaling wall is in the code or in the API shape. Code problems — missing indexes, cache bypasses, hot requeues — get code fixes. But when every object carries config that is really fleet-level identity, no amount of jitter will save you. Split the CRD, reference the shared thing, and cache by identity. Your OOMs, your rate limits, and your 2am rotation runbook will all thank you.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
