Fake Client vs envtest vs Testcontainers K3s: The Real Cost of Testing a Cluster API Controller

Delete a Cluster object in a real Kubernetes cluster and every Machine it owns disappears with it — that's the garbage collector doing its job. Write a test against envtest, controller-runtime's go-to integration harness, and the exact same assertion silently lies to you: the owner reference gets set correctly, the test passes, and the Machine object just... stays there forever. There's no controller-manager running to collect it.

That's not a timing bug or a flaky test. It's a structural gap in the second of three test tiers every Cluster API controller author has to choose between: a fully mocked client-go fake, controller-runtime's envtest, and a real (if disposable) cluster — increasingly, via Testcontainers' K3s module. Here's what each tier actually catches, what it doesn't, and where the honest trade-off sits for a controller that manages real bare-metal machines.

The three tiers, side by side​

The middle column is the one worth sitting with: envtest's own docs describe it as starting "an instance of etcd and the Kubernetes API server, without kubelet, controller-manager or other components." That's a real apiserver binary — the same one your production cluster runs — but with the actual controller-manager omitted, which is exactly why garbage collection doesn't happen. Everything else in this post is a consequence of that one sentence.

What the Testcontainers K3s module actually does​

Before getting to the gaps, it's worth being concrete about what this option costs to adopt, because it's genuinely simple. In Go:

import (
    "context"
    "github.com/testcontainers/testcontainers-go/modules/k3s"
)
 
func TestMachineReconcilerAgainstRealCluster(t *testing.T) {
    ctx := context.Background()
    k3sContainer, err := k3s.Run(ctx, "rancher/k3s:v1.27.1-k3s1")
    if err != nil {
        t.Fatal(err)
    }
    defer k3sContainer.Terminate(ctx)
 
    kubeConfigYaml, err := k3sContainer.GetKubeConfig(ctx)
    // build a client from kubeConfigYaml, apply CRDs, run your reconciler
}

k3s.Run boots the rancher/k3s image as a real (if throwaway) Kubernetes control plane, exposing the API server on 6443/tcp. GetKubeConfig() hands back a working kubeconfig. WithManifest() applies CRDs or fixtures at boot, and LoadImages() pushes a locally built controller image straight into the cluster without a registry — pair it with imagePullPolicy: Never. The same module ships for Java (org.testcontainers:k3s), Python, .NET, Node, and Rust, all wrapping the identical rancher/k3s image. It's an official, documented module, not a community hack — Docker's own engineering blog walked through the exact pattern for a Quarkus-based operator in 2023.

One adoption wrinkle worth flagging up front: K3sContainer runs privileged, so it won't start under rootless Docker or some Docker-in-Docker CI setups. Check your runner before you build a test suite around it.

The gap that actually matters: garbage collection​

This is the headline finding, and it's documented, not inferred. controller-runtime issue #626 is a maintainer confirming exactly this: a contributor tried forcing GC on in envtest with --enable-garbage-collector, and "it doesn't seem to have any effect... there is no garbage collection happening (possibly because controller-manager isn't present)." The accepted workaround in the Kubernetes controller community is to stop asserting cascade-deletion under envtest entirely — check that the owner reference got set, and leave verifying that the owned object actually disappears to a real cluster.

For a Cluster API-style reconciler, that's not a cosmetic limitation — cascade deletion is the behavior under test. Picture a Cluster controller that sets Machine.OwnerReferences pointing back at the Cluster, then a test that deletes the Cluster and asserts the Machine is gone. Under a fake client, that assertion never made sense to write in the first place — nothing enforces owner references there either.

Under envtest, the test can look reasonable and still be dead wrong: the Machine object never gets removed because nothing is watching for orphaned objects, so a reconciler with a genuinely broken finalizer or owner-ref bug would pass the same test that a correct one passes. Only a real control plane — Testcontainers K3s, a shared kind cluster, or a full e2e run — actually exercises the collector and can tell the two apart.

Where the brief's other two claims land differently​

CRD admission and webhook ordering are the other two things a real cluster supposedly catches that a mock can't — but both turn out to be apiserver-native behavior, which means envtest already has them, not gaps that only a real cluster closes.

CRD admission — schema validation, defaulting, and pruning of unknown fields — happens inside the API server process itself, driven by the CRD's OpenAPI structural schema. envtest runs the real kube-apiserver binary, so a CRD applied under envtest gets exactly the same admission behavior a production cluster would give it: send a Machine object with a field your CRD doesn't declare, and both envtest and a real cluster prune it identically.

The tier that actually misses this is the fake client, which just stores whatever Go struct you hand it — no schema, no defaulting, no pruning, because there's no apiserver in the loop at all. So the real three-way split for CRD admission is fake client (nothing) versus envtest-and-real-cluster (identical), not envtest versus real cluster.

Webhook ordering is the same story. envtest.WebhookInstallOptions lets you register real mutating and validating webhooks against the test API server, and Cluster API's own internal/envtest helpers do exactly that — pre-installing CAPI's actual webhooks so integration tests, in the project's own words, "mimic how the system behaves in real clusters." Kubernetes itself doesn't guarantee execution order across multiple webhooks matching the same request regardless of which cluster you're running on, so ordering bugs between your own webhooks reproduce under envtest just as they would in production.

The tier that can't see webhook behavior at all is, again, the fake client — it doesn't invoke admission webhooks, full stop.

Net: of the brief's three promised gaps, one (garbage collection) is real and envtest-vs-real-cluster. The other two (CRD admission, webhook ordering) are real gaps, but they belong to the fake-client-vs-everything-else boundary, not the envtest-vs-real-cluster one. A test suite that never goes past a fake client is missing all three; a suite already on envtest has closed two of them and is missing exactly one — GC.

The boot-cost bill​

The honest answer here is that no one has published a clean three-way benchmark comparing fake-client setup, envtest bring-up, and Testcontainers K3s boot time for the same controller suite — despite a fair amount of searching, that number doesn't exist yet. What does exist are two documented ceilings: envtest's ControlPlaneStartTimeout defaults to 20 seconds, and the Testcontainers K3s module's default container-ready wait is 60 seconds. Treat both as upper bounds the maintainers consider a failure past, not as typical measured times — but the 3x gap between them is directionally real, and it compounds fast if you boot a fresh cluster per test case instead of per suite.

Cluster API's own project tooling already treats this as a real cost worth amortizing, not a rounding error: hack/setup-envtest-with-kind.sh exists specifically to let contributors reuse one kind cluster across a test run instead of paying envtest's control-plane startup on every invocation, and a CAPI_DISABLE_TEST_ENV flag lets a run skip envtest setup entirely when a test only needs the fake client. The practical rule that follows: scope a real cluster (K3s container or kind) to the test session, not the test case, the same way you'd reuse a Postgres Testcontainer across a suite rather than restarting it per assertion.

Which tier for which test​

What this means for a fleet controller in practice​

Bex builds on Cluster API for its own machine and cluster reconcilers, so this exact ladder is the one its own controller test suite has to climb. A reconciler that only ever asserted "owner reference is set" against envtest would ship with unverified cascade-delete behavior — the kind of bug that shows up months later as orphaned bare-metal Machine records nobody's controller ever cleans up. Adding a Testcontainers K3s-backed test tier for exactly the delete paths closes that gap without giving up envtest's speed for the schema-validation and webhook tests that don't need a real controller-manager at all.

Bex.co is the open-source, AI-native Render alternative, built on Cluster API — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Testcontainers K3s module — Go
• Testcontainers K3s module — landing page
• Testcontainers K3s module — Java
• Testcontainers: The Simplest Way to Test Kubernetes Operators — Docker Blog
• envtest reference — The Kubebuilder Book
• envtest package documentation — pkg.go.dev
• Garbage collection in envtest.Environment — controller-runtime issue #626
• Cluster API developer testing guide
• Cluster API envtest test helpers — pkg.go.dev
• Kubernetes admission webhook good practices — ordering guarantees

All facts and quotes cited above are drawn directly from the linked sources.