CAREN Meets Flux: Composable Runtime Hooks for Day-2 Cluster API Without Custom Controllers

The cluster upgraded. The CNI didn't.

You bumped the Kubernetes version in your Cluster API Cluster spec, the control plane rolled forward cleanly, and then new worker nodes joined with a different CNI configuration than the old ones — because the shell script that installs Cilium runs from a ClusterResourceSet that has no idea an upgrade is happening, and the custom controller you wrote to sequence CCM upgrades last quarter doesn't know it's supposed to wait for the CSI driver to settle first. Now you have two CNI versions serving two halves of the same fleet, and a network policy that works on one topology but silently drops traffic on the other.

This is the day-2 gap Cluster API's Runtime SDK was built to close: not provisioning your first cluster, but keeping every cluster and every addon in sync through every upgrade, remediation, and deletion that follows. And the CNCF's 2026 on-demand session on CAPI + CAREN + Flux GitOps argues you shouldn't be closing it with more bespoke controllers.

CAREN — Cluster API Runtime Extensions - Nutanix, nutanix-cloud-native/cluster-api-runtime-extensions-nutanix — is a library of pre-built Runtime SDK hook implementations that turns day-2 lifecycle events into composable extensions. Combined with Flux GitOps, it makes the entire cluster and application stack continuously reconciled from a single Git source. Here is what that actually means, how to wire it into a ClusterClass that provisions on Hetzner via the generic provider path, and what it replaces.

What CAREN actually is (and isn't)​

CAREN is not a fork of Cluster API and not an infrastructure provider. It is a Go binary that speaks the Runtime SDK's extension protocol — an HTTPS server that registers handlers for specific lifecycle hooks, advertised to the management cluster through an ExtensionConfig object.

The core insight: Cluster API's ClusterClass already gives you a single Cluster object to mutate to orchestrate upgrades. Variables declared on Cluster.spec.topology.variables flow through patches into provider and addon templates. The Runtime SDK adds a missing verb to that noun: when lifecycle events happen, an extension can intercept, validate, mutate, and block progress. CAREN implements those verbs for common concerns, with fast-feedback unit tests instead of long-running e2e jobs proving each one.

Concretely, CAREN ships handlers that map to these hooks:

That last row exists because of a very specific, expensive bug: if you delete a workload cluster while a tenant still holds a LoadBalancer Service, the cloud provider's external load balancer is orphaned — you're billed for infrastructure no cluster owns anymore. CAREN's BeforeClusterDelete hook blocks deletion, deletes the Services, and waits for the cloud provider to clean up. Without it, that sequencing is a shell script someone has to remember to run.

The extensions are provider-aware — CAREN supports AWS, Nutanix AHV, vSphere, Docker (for dev), and a generic provider that maps cleanly to CAPH-on-Hetzner and other providers that don't have first-class CAREN support. Variables under clusterConfig and workerConfig carry slightly different schemas per provider (instance types, region, placement, machine details), but the hook lifecycle is shared.

The Runtime SDK contract: ExtensionConfig and blocking hooks​

Understanding the wiring requires one level of indirection deeper than the handlers. Runtime Extensions are not controllers watching Cluster objects. They are HTTPS servers that the CAPI runtime calls at synchronous extension points.

Registration is an ExtensionConfig resource on the management cluster:

apiVersion: runtime.cluster.x-k8s.io/v1alpha1
kind: ExtensionConfig
metadata:
  name: caren
spec:
  clientConfig:
    service:
      name: caren-extension
      namespace: caren-system
      path: /hooks
    caBundle: <PEM-encoded CA for the webhook server>
  namespaceSelector:
    matchLabels:
      caren.nutanix.com/enabled: "true"
  settings:
    # CAREN uses settings to tune handler behavior per fleet
    experimental:
      hooks:
        - name: AfterControlPlaneInitialized
        - name: BeforeClusterUpgrade
        - name: BeforeClusterDelete

When a lifecycle event fires — say, a user updates spec.topology.version on a Cluster from v1.31.0 to v1.32.0 — CAPI:

1. Discovers extensions matching the namespace selector.
2. Calls each extension's BeforeClusterUpgrade handler in priority order.
3. Waits for all handlers to respond with status: Success or status: Failure — the hook is blocking. A failure response halts the upgrade.
4. Proceeds to roll the control plane only after every extension approved.

After the control plane is initialized for a new cluster, AfterControlPlaneInitialized fires — this is where addons (CNI, CCM, CSI) are installed into the workload cluster, not on the management cluster, via the addon-provider pattern (Helm or ClusterResourceSet internally).

At the code level, a handler looks like this (from CAREN's own CAPI book example):

catalog := runtimecatalog.New()
_ = runtimehooksv1.AddToCatalog(catalog)
 
webhookServer, _ := server.New(server.Options{
    Catalog:  catalog,
    Port:     9443,
    CertDir:  "/tmp/k8s-webhook-server/serving-certs/",
})
 
webhookServer.AddExtensionHandler(server.ExtensionHandler{
    Hook:        runtimehooksv1.BeforeClusterUpgrade,
    Name:        "before-cluster-upgrade",
    HandlerFunc: DoBeforeClusterUpgrade,
})
webhookServer.AddExtensionHandler(server.ExtensionHandler{
    Hook:        runtimehooksv1.AfterControlPlaneInitialized,
    Name:        "after-control-plane-initialized",
    HandlerFunc: DoAfterControlPlaneInitialized,
})

The critical property: multiple handlers can serve the same hook. CAREN's single extension server composes its CNI, CCM, CSI, NFD, and autoscaler handlers behind one ExtensionConfig and one webhook endpoint. Each handler's DoBeforeClusterUpgrade is independent — the CNI handler upgrades Cilium, the CCM handler swaps the cloud-controller image, and they don't know about each other except through ordering guarantees the runtime provides.

This composability is what "without hand-rolled controllers" means concretely. The alternative — one custom operator per concern, each watching Cluster and reconciling its own addon — gives you N codebases, N leader-election configurations, N sets of RBAC, and an ordering problem you solve with sleeps or label-driven sequencing. CAREN's approach is one binary, one RBAC set, one upgrade, and an explicit hook ordering.

A word of caution from the CAPI book that is easy to skip past: Runtime SDK is an advanced feature. A failing Runtime Extension can severely impact the Cluster API runtime. Vetting ExtensionConfig registration is treated with the same seriousness as admission webhooks — because structurally, that's what the lifecycle hooks are.

Wiring CAREN into a ClusterClass (the hands-on core)​

Here is the minimal path from empty management cluster to CAREN-managed workload clusters, with the generic provider path highlighted for a CAPH fleet that provisions Hetzner machines.

1. Deploy CAREN​

Via clusterctl (the shortest path):

# clusterctl.yaml — add CAREN as a provider
providers:
  - name: caren
    url: https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/releases/latest/caren-components.yaml
    type: RuntimeExtension

Or via Helm, for a GitOps-managed management cluster — which is where Flux enters and where most teams land:

helm upgrade --install caren oci://ghcr.io/nutanix-cloud-native/caren-chart \
  --namespace caren-system --create-namespace \
  --set extensionConfig.enabled=true

Verify the extension is discovered:

kubectl get extensionconfig caren -o yaml
kubectl get extensionconfig caren -o jsonpath='{.status.handlers[0].name}'
 
# Check CAREN's webhook is responding
kubectl get pods -n caren-system
kubectl logs -n caren-system deploy/caren --tail=20 | grep -i hook

2. Patch your ClusterClass to advertise CAREN's variables​

CAREN needs ClusterClass.spec.patches entries that call out to its external variable discovery hooks. Pick the set matching your provider — for a Hetzner fleet without a dedicated CAPH integration, use generic:

apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: hetzner-quickstart
spec:
  controlPlane:
    local:
      ref:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: KubeadmControlPlaneTemplate
        name: hetzner-control-plane
  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: HetznerClusterTemplate
      name: hetzner-cluster
  workers:
    machineDeployments:
      - class: default-worker
        template:
          bootstrap:
            ref:
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              kind: KubeadmConfigTemplate
              name: hetzner-worker-bootstrap
          infrastructure:
            ref:
              apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
              kind: HetznerMachineTemplate
              name: hetzner-worker
  patches:
    # Generic provider — works with CAPH where no provider-specific
    # variable set exists. Also available: aws, nutanix, docker.
    - name: cluster-config
      external:
        discoverVariablesExtension: genericclusterconfigvars.cluster-api-runtime-extensions-nutanix
        generateExtension: genericclusterv7configpatch.cluster-api-runtime-extensions-nutanix
    # Worker customizations have no generic equivalent (control-plane-only
    # for generic); use docker/worker or omit if your worker template
    # needs no CAREN-managed overrides.

For teams already on Docker for dev clusters and planning a CAPH migration, the docker patch set is the faithful dev-to-prod analog:

  patches:
    - name: cluster-config
      external:
        discoverVariablesExtension: dockerclusterconfigvars.cluster-api-runtime-extensions-nutanix
        generateExtension: dockerclusterv7configpatch.cluster-api-runtime-extensions-nutanix
    - name: worker-config
      external:
        discoverVariablesExtension: dockerworkerconfigvars.cluster-api-runtime-extensions-nutanix
        generateExtension: dockerworkerv7configpatch.cluster-api-runtime-extensions-nutanix

If your provider is AWS, Nutanix, or vSphere, use the provider-specific patch names — they expose additional fields (instance types, placement groups, machine details) that the generic set omits.

3. Declare cluster variables​

With the patches in place, Cluster.spec.topology.variables gains CAREN's schema. A realistic Hetzner workload cluster:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: tenant-eu
  namespace: tenants
  labels:
    caren.nutanix.com/enabled: "true"  # must match ExtensionConfig namespaceSelector
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  topology:
    class: hetzner-quickstart
    version: v1.31.2
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: default-worker
          name: workers
          replicas: 3
    variables:
      - name: clusterConfig
        value:
          # Addons — each becomes an AfterControlPlaneInitialized install
          cni:
            provider: cilium
            cilium:
              version: v1.16.5
          # CCM / CSI would use provider-specific handlers when available;
          # on generic, these flow through the same patch with fewer fields.
          # Optional: opt out of the LoadBalancer GC hook per cluster
          # loadBalancerGC: false  # default true; set false to skip BeforeClusterDelete GC
          # Generic globals available on any provider:
          ntp:
            enabled: true
            servers: ["0.pool.ntp.org", "1.pool.ntp.org"]
          httpProxy:
            enabled: false

What happens from here, without any additional automation: CAPI creates the HetznerCluster, KubeadmControlPlane, and MachineDeployments. Once the control plane is initialized, the AfterControlPlaneInitialized hook fires — CAREN installs Cilium into the workload cluster. When someone later edits topology.version to v1.32.0, BeforeClusterUpgrade fires — CAREN sequences the CNI upgrade before CAPI rolls the machines, so at no point do old and new nodes run mismatched network plugins.

To observe it:

# Watch hooks fire during an upgrade
kubectl get cluster tenant-eu -n tenants -o yaml | yq '.status.conditions[] | select(.type | contains("TopologyReconciled"))'
 
# CAREN surfaces status through cluster conditions and its own logs
kubectl logs -n caren-system deploy/caren | grep -E "BeforeClusterUpgrade|AfterControlPlaneInitialized"

Composability vs hand-rolled: what "without custom controllers" replaces​

The naive alternative is straightforward to describe because most teams have built it at least once: a shell script that runs kubectl apply -f cilium.yaml after cluster creation, a second script for CSI, a third for CCM, and a GitOps Kustomization that re-applies them continuously — plus a bespoke controller someone wrote to handle upgrades because the scripts don't know when to run.

Here is what that actually costs, item by item:

None of this makes CAREN free. Its handlers are opinionated — the CNI handler supports Calico and Cilium, not Flannel; the CSI handler knows the CSI drivers CAREN's maintainers test, not every out-of-tree CSI. If your fleet needs a niche addon CAREN doesn't cover, you'll still write a Runtime Extension handler yourself — but now it's one handler behind the same ExtensionConfig, not a standalone controller with its own RBAC and lifecycle. The composability pays off in that direction: adding one handler is cheaper than adding one operator.

The failure-mode tradeoff is symmetric and worth internalizing: because hooks are blocking, a misbehaving handler halts cluster lifecycle transitions cluster-wide. That's the right default for CNI — you want the upgrade to stop if the CNI can't be upgraded safely — but it means a flaky NFD handler can block unrelated upgrades. CAREN's handlers are tested against this, but the underlying Runtime SDK property still applies: vet the extension before you register it, the same way you vet an admission webhook.

Closing the loop with Flux GitOps​

This is where the CNCF session's title earns the "and GitOps" part. The management cluster already runs Flux (GitRepository + Kustomization reconciling from a Git repo). The workload clusters' definitions — Cluster, ClusterClass, ExtensionConfig, and the addon configuration embedded in clusterConfig — are just Kubernetes manifests in that repo. Flux reconciles all of them continuously, so the entire stack — infrastructure, cluster topology, runtime hooks, and applications — is self-managing from a single source of truth.

A minimal repository layout:

clusters/
  management/
    flux-system/          # bootstrapped once via flux bootstrap
      gotk-sync.yaml
    infrastructure/
      caren/               # HelmRelease for CAREN
        release.yaml
        kustomization.yaml
      caph/                # CAPH provider components
      flux-controllers/
  tenants/
    production/
      tenant-eu.yaml       # Cluster + variables (the manifest from Section 3)
      tenant-us.yaml
    kustomization.yaml     # aggregates all tenant Clusters behind one Flux Kustomization

And the Kustomization that binds it together, with ordering expressed through dependsOn:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: fleet
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/your-org/fleet
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: fleet
  path: ./clusters/management/infrastructure
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-clusters
  namespace: flux-system
spec:
  interval: 2m
  sourceRef:
    kind: GitRepository
    name: fleet
  path: ./clusters/tenants
  prune: true
  dependsOn:
    - name: infrastructure   # tenants reconcile only after CAREN + CAPH are healthy
  healthChecks:
    - apiVersion: runtime.cluster.x-k8s.io/v1alpha1
      kind: ExtensionConfig
      name: caren
      namespace: default
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: tenant-eu
      namespace: tenants

The workflow for a day-2 event — upgrading Cilium across every tenant cluster — collapses to a single Git change:

1. Edit tenant-eu.yaml: cni.cilium.version: v1.17.0.
2. Commit and push to main.
3. Flux pulls the commit, applies the Cluster update. CAPI sees a variable change, fires BeforeClusterUpgrade to CAREN's CNI handler. The handler upgrades Cilium in the workload cluster, returns Success. CAPI proceeds to roll machines. No operator was paged, no script was invoked by hand.

Rolling the Kubernetes version itself is identical: edit topology.version, push, watch the hooks sequence correctly without a separate upgrade playbook. And when Flux's drift detection notices someone ran kubectl edit inside a workload cluster, it reverts the drift to match Git on the next reconciliation — which is the point of the "self-managing" claim that would otherwise be marketing.

For a bex-managed fleet, this is the same shape bex's own reconciliation already aims for: declarative desired state, controllers that converge toward it, and an open Runtime SDK extension point where a platform team can plug in custom lifecycle behavior without forking the provider. The difference is that the CAREN library exists today with pre-tested handlers, so the platform doesn't need to invent the CCM or CNI hook from scratch to get correctness on upgrades.

What this doesn't solve (honest limits)​

• CAREN is Nutanix-maintained. That scopes its provider testing and its release cadence. AWS, Nutanix, vSphere, and Docker have first-class clusterConfig schemas; generic — the path a CAPH fleet actually uses — is deliberately thinner. If you're on bare metal or an unsupported provider, you carry more provider-specific wiring in your ClusterClass patches than the talk makes it sound.

• The x-k8s.io confusion is real but separate. Recent Gateway API changes introduced gateway.networking.x-k8s.io for a new experimental XBackend resource, but GRPCRoute, BackendTLSPolicy, and every other route type a typical PaaS uses remain gateway.networking.k8s.io/v1 Standard in Gateway API v1.6. If you adopted Gateway API early, your migration burden for day-2 is not a mass X-prefix rename — it's the ordinary move from v1alpha2 experimental to v1 Standard for resources that have graduated, plus an upgrade-order trap that can drop traffic if the CRDs and controller versions cross in the wrong direction.

• A blocking hook that fails blocks everything. That's why CNI sequencing is correct under CAREN, but it's also why a buggy custom handler registered for BeforeClusterUpgrade can stall every cluster upgrade in the fleet behind a cluster condition that says "extension failed." The mitigation is input validation on ExtensionConfig and handler-level retries, not removing the blocking behavior.

• Flux is an additional control plane. The management cluster now runs Flux controllers reconciling continuously — that's operational surface, not free. The payoff is that the management cluster's own desired state is also in Git and also reconciled (Flux bootstrapped onto itself), but the initial bootstrap, GitRepository credentials, and Kustomization health checks are setup work bex's own managed control plane abstracts away.

• CAREN doesn't replace cluster autoscaling policy. The Cluster Autoscaler handler deploys the autoscaler and wires it to CAPI's MachineDeployment, but it doesn't decide the scaling policy — minSize, maxSize, and scale-down utilization thresholds remain fleet-operator decisions, and autoscaling on bare metal (fixed machine inventory) behaves differently than on cloud elastic capacity.

If you run a Cluster API fleet on Hetzner and your day-2 automation today is a shell script and a ClusterResourceSet that nobody wants to touch during upgrades, the CAREN + Flux path is worth a spike: one extension server, one Git repo as source of truth, and hook implementations that have been tested against the exact lifecycle sequencing you currently hope works. It won't remove the need for a platform team to understand the lifecycle model, but it does remove the need for that team to reimplement it per addon.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Cluster API, Hetzner-backed, with a Render-compatible API and an MCP server any agent can already call. Star the repo on GitHub or deploy your first app today.