Who Backs Up the Management Cluster? Rebuilding the CAPI Brain When It Dies

Your workload clusters will not page you when the management cluster dies.

That is the trap. At 3 a.m. the Hetzner box that runs your Cluster API management cluster goes dark — disk failure, bad kernel upgrade, someone deletes the wrong HetznerCluster. Your apps keep serving. Grafana stays green for the workloads. Then at 9 a.m. a node fails, a Machine needs remediation, or you push a new MachineDeployment image — and nothing happens. No new Machines. No healing. No upgrades. The fleet is frozen alive.

Cluster API concentrates every decision about your fleet's future into one place: Cluster, Machine, MachineDeployment, KubeadmControlPlane, HetznerMachineTemplate, provider secrets, and CA bundles — all stored as etcd state on the management cluster. The Cluster API book puts it plainly: if the management cluster fails, workload clusters continue to operate, but you can no longer manage them. If you have a backup of all the CRDs, you can create a new management cluster and regain control. That "if" is the entire disaster-recovery plan most teams never write.

This post is that plan. Three recovery paths, when each one works, why etcd snapshot restore has a stale-state footgun that Velero issue #5957 already hit, and a quarterly drill you can run this week to prove the brain is recoverable — not just the limbs.

Pick your path in 30 seconds: the recovery matrix​

You do not need to read all three paths to choose. Use this table first, then jump to the section that matches your situation.

The tradeoff is not just speed. Path 1 can reconcile against stale reality. Path 2 cannot work if the source is dead. Path 3 is the slowest honest path, but it sidesteps stale state entirely because Git — not a point-in-time etcd dump — is the source of truth.

If you run Fleet on owned Hetzner hardware with CAPH, keep a standby management cluster warm or keep Git authoritative. Do not rely on snapshots alone.

What lives in the brain (and what you must back up)​

Before you can recover the brain, you need to know what fits inside it. For a CAPH fleet on Hetzner, the management cluster's etcd holds:

• CAPI core: Cluster, Machine, MachineDeployment, MachineHealthCheck, KubeadmControlPlane, KubeadmConfigTemplate
• Infrastructure provider: HetznerCluster, HetznerMachineTemplate, HCloudMachine (CAPA equivalent for AWS)
• Bootstrap & control plane: KubeadmConfig, provider credentials Secrets, Cluster CA Secrets and ConfigMaps
• Provider manifests themselves: CAPI controllers, CAPH, kubeadm bootstrap/control-plane providers, cert-manager

Miss any of those and the recovered brain disagrees with reality. The classic failure is backing up Machine but not the HetznerMachineTemplate it references — the controller creates a Machine it can never provision because the template is gone.

For an etcd-off-box backup, this is the label-selector set that must be included, not sampled:

# What "everything that matters" actually means for a CAPH fleet
kubectl get clusters,machines,machinedeployments,kubeadmcontrolplanes \
  -A --show-labels
 
# The namespaces that must survive a backup (adjust to your layout)
# - capi-system, caph-system, capi-kubeadm-bootstrap-system,
# - capi-kubeadm-control-plane-system, cert-manager
# Plus every namespace that holds Cluster/Machine objects (often "default" or per-team)
 
# Velero users: last Velero restore cleaned OwnerReferences — see issue #5957.
# Always pause clusters before backup (next section), and verify templates survive.

If you use Velero instead of etcd snapshots, add cluster.x-k8s.io/backup labeling and test that restores preserve OwnerReference chains. The upstream issue is still open: Velero's restore strips owner references during resetMetadataAndStatus, so a naïve backup/restore can orphan Machines from their Clusters. The fix is procedural — pause before you back up — not just tooling.

Path 1: etcd snapshot restore — fastest, with a stale-state trap​

When the management cluster's etcd data directory is intact or you have a recent off-box snapshot, etcdctl snapshot restore is the fewest-steps path. It rebuilds a single-member etcd from a point-in-time keyspace. Done correctly, it takes minutes. Done naively, it hands your controllers a parallel history that silently diverges from the real world.

Take and verify the snapshot​

# On the management control-plane node (or via etcd pod exec)
export ETCDCTL_API=3
export ENDPOINT="https://127.0.0.1:2379"
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
 
# Save
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date -u +%Y%m%dT%H%M%SZ).db
 
# Verify — a truncated file is worse than no backup because it gives false confidence
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-*.db -w table
 
# Ship off-box immediately (S3, Restic, whatever you actually restore from)
aws s3 cp /backup/etcd-snapshot-*.db s3://your-capi-backups/mgmt/etcd/

Schedule this at least hourly. A 24-hour-old snapshot means rebuilding every Machine event from the last day against live Hetzner state — which is where the trap waits.

Restore with a revision bump​

The etcd recovery docs call this out specifically for Kubernetes: restoring takes the cluster back in time, and every watcher (kubelet, controller-manager, CAPI controllers, cloud controllers) sees a conflicting parallel history. The fix is a revision bump that invalidates every watch after the snapshot.

# Stop kube-apiserver and etcd on the target host first
# Then restore to a clean data dir
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260821T120000Z.db \
  --data-dir=/var/lib/etcd-restored \
  --name mgmt-etcd-0 \
  --initial-cluster mgmt-etcd-0=https://10.0.0.10:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-advertise-peer-urls https://10.0.0.10:2380
 
# Critical for Kubernetes consumers: bump revision and mark compacted
# Without --mark-compacted, watches resume against revisions that no longer exist.
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260821T120000Z.db \
  --data-dir=/var/lib/etcd-restored \
  --bump-revisions 1000000000 \
  --mark-compacted
 
mv /var/lib/etcd /var/lib/etcd.bak
mv /var/lib/etcd-restored /var/lib/etcd
systemctl restart etcd kube-apiserver

If you skip --bump-revisions and --mark-compacted, CAPI and cloud controllers may reconcile against revisions etcd no longer serves, producing watch errors that look like transient flakes but mask real missed events.

Why stale state is the scary part​

Between your snapshot time and the failure, Hetzner reality moved: Machines were created or deleted, IPs were assigned, HCloudMachine provider IDs settled. Your restored etcd does not know. When CAPI controllers resume, they compare desired state (stale Machine.spec) against actual infra and try to close the gap — sometimes by recreating Machines that already exist or deleting ones a human already pivoted by hand.

Red Hat's etcd restore docs state it bluntly: restore should only be used as a last resort when the API is unavailable, because "restoring etcd effectively takes a cluster back in time and all clients will experience a conflicting, parallel history." For CAPI specifically, that conflicting history is a Machine inventory that disagrees with the Hetzner console.

Mitigate before you trust a restored brain:

• Pause reconciliation first once the API is up:

kubectl annotate cluster <cluster-name> cluster.x-k8s.io/paused=true --all -A
kubectl annotate machinedeployment <md-name> cluster.x-k8s.io/paused=true -A

• Diff desired vs actual: list Machines in etcd vs hcloud server list (or CAPH provider view) before you unpause. Reconcile server IDs, not just names.
• Unpause one Cluster at a time, watching controller logs for reconcile on Machines that already have provider IDs.

Path 1 wins on speed and works when the source is gone. It loses on correctness when the snapshot is old. If your snapshot interval is daily and your fleet turns over Machines weekly, you are restoring stale on purpose.

Path 2: clusterctl move to a standby — the live-migration escape hatch​

If the management cluster's API still answers, clusterctl move is the cleanest path. It enumerates every CAPI object and its dependencies (Clusters, Machines, Templates, Secrets, CA bundles) from the source and recreates them on the target, honoring finalizer ordering and object-set conservation so no in-flight Machine provisioning is torn mid-creation. The Cluster API book describes it as clusterctl move --to-kubeconfig from bootstrap to target; the same primitive works for brain failover to a warm standby.

The sequence that does not deadlock​

# 1. Standby must be a real management cluster first (and it needs a worker)
#    Single-control-plane with NoSchedule taint is not enough — controllers need scheduling.
clusterctl init --kubeconfig=/path/to/standby.kubeconfig \
  --infrastructure hetzner \
  --bootstrap kubeadm --control-plane kubeadm
 
# 2. Pause workload Clusters on the source so no reconciler races the move
kubectl --kubeconfig=/path/to/source.kubeconfig \
  annotate cluster --all cluster.x-k8s.io/paused=true -A
 
# 3. Move — live (API to API) or via directory for air-gapped / audited moves
#    Live:
clusterctl move --to-kubeconfig=/path/to/standby.kubeconfig \
  --kubeconfig=/path/to/source.kubeconfig
 
#    Directory (auditable, also the path that replaces deprecated clusterctl restore):
clusterctl move --to-directory=/tmp/capi-move \
  --kubeconfig=/path/to/source.kubeconfig
#  inspect /tmp/capi-move, then on standby:
clusterctl move --from-directory=/tmp/capi-move \
  --kubeconfig=/path/to/standby.kubeconfig
 
# 4. Verify the object-set conserved (no silent drops)
kubectl --kubeconfig=/path/to/standby.kubeconfig get clusters,machines -A
 
# 5. Unpause on the standby
kubectl --kubeconfig=/path/to/standby.kubeconfig \
  annotate cluster --all cluster.x-k8s.io/paused-
 
# 6. Decommission source only after standby controllers report Ready
kubectl --kubeconfig=/path/to/standby.kubeconfig get pods -n capi-system -n caph-system

When move works and when it does not​

Move is a migration, not a resurrection. It needs the source API to list objects, so a dead etcd, a quorum-lost management cluster, or a mangled etcd data directory is exactly the failure where move is unavailable. It also surfaces subtle invariants the Cluster API formal-methods work calls out: finalizer ordering on the source, controller-pause coordination, and partial-pivot recovery if the move itself crashes mid-copy. If your source has hundreds of CRDs (Azure Service Operator is a common trigger), the client-side rate limiter can exhaust during proxy.ListResources — bump it or narrow the move namespace.

Dry-run it before you need it:

clusterctl move --to-kubeconfig=/path/to/standby.kubeconfig \
  --kubeconfig=/path/to/source.kubeconfig --dry-run

If you run a standby at all, this is the path that gives zero data loss with the least human interpretation. Keep the standby's provider versions pinned to the source so clusterctl init does not silently upgrade CAPH during the emergency.

Path 3: GitOps re-bootstrap — rebuilding the brain from Git​

The third path trades time for truth. Instead of replaying old etcd state, you re-bootstrap a fresh management cluster from Git and re-adopt the existing workload clusters. Workload clusters keep serving; the new brain just re-learns where they are.

This is the pattern the kind → Flux → CAPI/Proxmox/Talos homelab setups converge on: a just bootstrap command on a laptop seeds a kind cluster, Flux reconciles cert-manager, cluster-api-operator, CAPH, and Talos providers, and then CAPH provisions workload clusters. Pivoting that model to recovery looks like:

# 1. Fresh host — kind is the new bootstrap brain
kind create cluster --name recovery
export KUBECONFIG=$(kind get kubeconfig --name recovery)
 
# 2. GitOps takes over — Flux reconciles provider stack from Git
#    (your repo already holds the desired version of capi, caph, bootstrap/control-plane)
flux bootstrap github --owner=your-org --repository=fleet-config \
  --branch=main --path=clusters/management
 
# Wait for provider health, not just pod Running
kubectl wait --for=condition=Available deployment -l cluster.x-k8s.io/provider -A --timeout=300s
 
# 3. Re-initialize CAPI if not already via Flux's cluster-api-operator
clusterctl init --infrastructure hetzner --bootstrap kubeadm --control-plane kubeadm
 
# 4. Re-apply fleet desired state from Git
#    Every Cluster, MachineDeployment, HetznerMachineTemplate the fleet should own lives here
kubectl apply -k clusters/workloads/
 
# 5. Re-adoption: CAPI controllers see existing Hetzner servers + kubeconfigs
#    and adopt rather than recreate — verify by provider ID, not name
kubectl get clusters,machines -A
hcloud server list  # or hetzner console — compare IDs
kubectl get kubeadmcontrolplanes -A -o wide
 
# 6. Once adopted, the kind recovery cluster can itself be moved off kind
#    onto a permanent management MachineDeployment via another clusterctl move

Why this sidesteps the stale-state trap: Git holds desired state, not a timestamped snapshot of observed state. The controllers reconcile desired vs actual infra directly after adoption, with no "what happened between snapshot and restore" window to hand-wave. The cost is operational: Flux/Argo must be the source of truth before the failure, provider versions must be pinned in Git, and secrets need sealed-secrets / external-secrets / 1Password-operator replication — otherwise the new brain boots with controllers but no credentials to talk to Hetzner.

Time it end-to-end before you quote it to your team. For a three-workload fleet on Hetzner with Flux and CAPH, 15–30 minutes is realistic on a warm laptop; two hours is the budget if image caches are cold and you hit CNI bring-up.

The quarterly drill: prove it before you need it​

A brain you have never restored is a brain you do not have. Run this drill quarterly. It takes an afternoon the first time and an hour after that. Put it on the calendar as a real incident — not a background task.

Setup (once)

• A cold standby context (standby.kubeconfig) or a kind-capable laptop with flux, clusterctl, and etcdctl installed.
• Off-box snapshot destination that is not the same Hetzner volume as the management cluster.
• Git as source of truth for clusters/ — every Cluster/MachineDeployment/Template is kubectl apply'd from Git, not hand-edited.

Drill steps

1. Snapshot freshness gate — Assert latest etcd snapshot is < 24 hours old. Fail the drill if it is older.

ETCDCTL_API=3 etcdctl snapshot status s3://your-capi-backups/mgmt/etcd/*.db -w table
# or local: ls -lt /backup/etcd-snapshot-*.db | head

2. Directory move dry run — Assert clusterctl move --dry-run against the standby succeeds with zero warnings.

3. Git re-bootstrap in staging — On an isolated kind cluster, run the Path 3 sequence against a staging Hetzner project. Assert kubectl get clusters,machines -A matches hcloud server list by provider ID for every Machine.

4. Adoption without recreation — After re-bootstrap, assert no Machine was recreated (check creationTimestamp and provider ID). A recreation where an adoption was expected is a fail — it means templates or CA bundles were missing.

5. Workload continuity gate — During the entire drill, assert workload clusters serve traffic (kubectl --kubeconfig=workload.kubeconfig get pods -A stays Ready, app health checks pass). The point of CAPI is that limbs keep walking while the brain is out.

Pass criteria: snapshot < 24h, move dry-run clean, Git re-bootstrap adopts without recreating Machines, workloads never flap. Log the duration. If it took two hours this quarter, that is next quarter's SLA, not a surprise.

And the bex angle​

A hosted PaaS hides this entire problem behind a vendor's undisclosed remediation queue. When their control plane has a bad week, you queue behind every other tenant — there is no clusterctl move to your own standby because the brain is not yours.

A self-hosted PaaS on owned Hetzner hardware inverts that. The management cluster is yours — which means its disaster recovery is yours too, not someone else's incident ticket. The three paths above map directly onto how a bex fleet recovers:

• Path 1 for when the management volume survives and you want minutes.
• Path 2 for planned handoffs and zero-loss live migration (keep a warm standby or a staged kind pivot).
• Path 3 for when the only honest recovery is re-bootstrap from Git — the path that makes git push the fleet's real backup, not a snapshot cron job.

Wire hourly etcd snapshots, keep clusters/ declarative, and run the drill. The next time the brain goes dark at 3 a.m., the page should say "recovered" before your users ever noticed it was frozen.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.