Controllers Are Cheap to Write Now. Running One Still Isn't

"Writing your own controller for Kubernetes is now a matter of a few hours."

That is the opening line of the July 29, 2026 Kubernetes blog deep dive by Ænix's Andrei Kvapil and Timofei Larkin, and it is the most consequential sentence published about platform engineering this year. The standard path — Go, kubebuilder on top of controller-runtime — hands you a project scaffold, typed APIs, and a working reconciler before lunch. Anything a platform team used to defer as "too expensive to build properly" is suddenly back on the table.

Here is the verdict up front, because the rest of this post is just the receipt: promote a job from scripts to a real controller when it needs continuous convergence over shared state; keep it a script when it is a one-shot ordered procedure. Authoring cost collapsed. Operating cost did not — leader election, informer cache sizing, and reconcile-loop idempotency bill you exactly what they always billed. The teams that win are the ones that spend the saved authoring budget on the operating half instead of on writing twice as many controllers.

The promotion table: six tenant-lifecycle jobs, six verdicts​

Every row below names three things: the state being reconciled, what a small fleet typically does today, and the verdict. If your platform runs on Cluster API, the "state" column is the point — a controller earns its keep only when there is a declared object worth converging toward.

Custom domains go first because they are the purest convergence problem on the list. A tenant adds app.example.com, and three systems must agree forever after: DNS must resolve it, the gateway must route it, and a certificate must cover it. A script handles the happy path fine and then rots on every edge — the tenant deletes the domain but the DNS record lingers, the cert renews against a route that no longer exists, two tenants claim the same hostname. A controller watches one DomainClaim-shaped object and re-converges all three on every change, including the deletions scripts forget. This is the kind of job the Ænix post's "few hours" claim is really about: the scaffold gives you the watch, the queue, and the retry loop, and your code is just the three-way diff.

Per-tenant quotas go second for a duller reason: drift. A script that applies ResourceQuota at signup runs once; everything after that — plan upgrades, downgrades, a human editing the quota by hand at 2 a.m. — silently diverges from the tier the tenant pays for. A controller that derives quota objects from the tier object makes the tier the single source of truth again. Unsexy, high-value, and exactly the kind of automation that never survived a "three weeks of operator work" cost-benefit review but sails through a "one afternoon" one.

Build-queue backpressure gets a split verdict. Keep the actual build execution in your CI runner — it already handles logs, timeouts, and artifact storage. But the admission decision (which builds run now, which wait, which tenant is hogging the queue) is shared state that every concurrent build mutates, and that is where scripts start double-booking capacity. A small controller owning queue order, with the runner as its dumb executor, is the shape that scales.

The TLS row is the most important one in the table precisely because its verdict is "don't." Certificate renewal is continuous convergence over shared state — theoretically a perfect controller candidate — and writing one would be a satisfying afternoon. It would also be malpractice: cert-manager is a mature, audited controller that already owns this problem, including the ACME edge cases you have never heard of. Cheaper authoring does not change the build-vs-adopt math; if anything, it makes "adopt" the default and reserves "build" for the logic that is genuinely yours.

Idle-app reaping stays a script until the policy earns a controller. "Scale down anything idle for 30 minutes" is a cronjob with a timestamp comparison — no shared state, no convergence, nothing to reconcile. The day the policy becomes "idle 30 minutes, except preview environments on business days, except tenants on annual plans, and never while a build is queued," you have reinvented a state machine in bash, and that is the afternoon to promote it.

Deploy-status reporting never promotes. Events flow one way, from build to dashboard, and nothing ever needs to converge: a missed event is a gap in a log, not drift in the world. A curl in a post-step is the correct architecture. Not everything wants to be a control loop, and a team that just learned controllers are cheap needs that sentence more than any other.

What "a few hours" actually buys you​

The claim holds up because the scaffold is genuinely complete. kubebuilder init plus kubebuilder create api generates the custom type definitions, the CRD manifests with validation markers, the RBAC rules scoped to exactly those types, the reconciler skeleton wired into a manager, a Makefile with build/test/deploy targets, and an envtest harness that spins up a real API server for integration tests without a cluster. The controller-runtime manager underneath contributes the parts nobody should hand-roll: the shared-informer cache, the rate-limited workqueue that collapses ten rapid updates into one reconcile, and leader-election wiring.

What you write yourself, on that first afternoon, is the diff: given this desired object and that actual cluster state, what changes. For a domain controller that is under two hundred lines of Go. The Ænix post's real subject is what happens after that afternoon — "as soon as load grows or the controller starts behaving in ways you did not expect, a whole class of edge cases shows up," all tracing back to a fuzzy mental model of the runtime. Which brings us to the bill.

The three costs the scaffold cannot pay for you​

1. Leader election and HA. A controller that mutates the world cannot run twice. The manager's leader election (a Lease object, renew-loop, failover on the order of seconds) solves this, but it converts every controller from "a process" into "a replicated service with a failover story": two replicas minimum, liveness semantics for the lease holder, and a freeze window during every election where nothing reconciles. Scripts have no equivalent cost because a cronjob that double-fires is a bug report, while a domain controller that double-fires is two tenants' DNS fighting. Budget one extra replica and a failover drill per controller, or run it single-replica and admit the control plane has a single point of failure.

2. Informer cache sizing. This is the expensive surprise the Ænix post centers on, and this blog has already covered it twice — what the cache actually is and what it costs on a small Hetzner control-plane node — so here is the one-paragraph version: every Kind your reconciler touches is fully listed into your operator's memory and kept there by a watch. A lean Pod runs 8–15 KB, but objects with managedFields and status bloat run 20–35 KB, and one unscoped watch against 10,000 Helm-release Secrets is over a gigabyte — an OOMKill on the same 4 GB box your Cluster API management cluster lives on. The fix (field selectors, TransformStripManagedFields, DisableFor, PartialObjectMetadata) is a configuration afternoon per controller, and it is not in the scaffold defaults. Multiply it by every controller the "few hours" pitch talked you into writing.

3. Reconcile-loop idempotency. The workqueue redelivers: every error, every resync period, every leader failover re-runs your reconcile against possibly-half-applied state. The classic first-controller bug looks like this — a reconciler that appends a finalizer or an owner entry on every pass without checking whether it is already there:

func (r *DomainReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var domain platformv1.DomainClaim
    if err := r.Get(ctx, req.NamespacedName, &domain); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    // BUG: runs on every reconcile, duplicates the entry on the second pass.
    domain.Finalizers = append(domain.Finalizers, "platform.bex.co/cleanup")
    return ctrl.Result{}, r.Update(ctx, &domain)
}

The fix is one if — check-then-act, or better, compute the full desired object and use server-side apply so re-running is a no-op. But the discipline has to hold across every write path in every controller, forever, because the runtime guarantees at-least-once delivery and nothing stronger. Scripts fail loudly and stop; controllers fail, retry, and corrupt idempotently-unprepared state in a loop. Code review for a new controller should spend more time on "what happens when this runs twice" than on anything the scaffold generated.

What must stay a script​

Ordered, run-once procedures are the anti-controller. Database migrations, one-shot backfills, tenant data exports: these need exactly-once execution in a defined sequence, with a human-readable log and a hard stop on failure. A control loop's superpowers — retry forever, converge from any state — are actively harmful here; "retry the migration forever" is how you get a half-applied schema behind a passing health check. Kubernetes already has the right primitive and it is not a controller: a Job with backoffLimit, launched by CI or a runbook, that either completes or visibly fails. When someone proposes a "migration controller," what they usually want is a Job with better notifications.

The new rule of thumb​

Before 2026, "should this be a controller?" was really a budget question, and the honest answer was usually no — three weeks of Go for quota enforcement nobody would fund. The Ænix post moves the authoring line from weeks to an afternoon, which flips the default for exactly the jobs in the table's top half: continuous convergence over tenant state that scripts can start but cannot sustain. What it does not move is the operating line: each promotion still costs a replicated deployment, a sized cache, and idempotent-everything review. So promote deliberately — domains first, quotas second, queue admission third — adopt where someone already paid those costs for you, and leave the scripts alone where there is nothing to converge toward.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.