Preventing etcd Zombies on Upgrade to 3.6: A Cluster API Operator's Pre-Flight Checklist

If your Cluster API management cluster is still running Kubernetes 1.31, 1.32, or 1.33, kubeadm has it pinned to etcd v3.5.24-0 — one patch release below the v3.5.26 floor etcd's own maintainers say you need before touching v3.6. Bump KubeadmControlPlane.spec.version straight to 1.34, and kubeadm hands your new control-plane machines etcd v3.6.5-0 directly. There's no intermediate step where a "safe" binary runs and repairs anything. That gap — not a vague warning about upgrade risk — is the actual pre-flight problem this post walks through.

What a "zombie member" actually is​

The Kubernetes and etcd projects published matching posts in December 2025 titled "Avoiding Zombie Cluster Members When Upgrading to etcd v3.6". The name is evocative, and it's easy to read it as "a process that doesn't fully die during the upgrade." That's not what's happening. A zombie member is a node that was removed from the cluster's membership list some time ago — potentially months or years earlier — reappearing and rejoining Raft consensus as if it were still there.

The mechanism is a storage-layer handoff. Through etcd v3.5, membership data is authoritative in v2store; v3store is a secondary copy. Starting in v3.6, that flips — v3store becomes the source of truth. If your cluster's v2store and v3store ever drifted out of sync before you upgrade, the v3.6 binary reads the stale v3store copy on startup and resurrects whatever membership state it finds there, including members you removed long ago.

Three known histories cause that drift:

• etcdctl snapshot restore on etcd v3.4 or earlier — a bug meant restore didn't fully clear existing members before writing new ones.
• --force-new-cluster recovery on etcd v3.5.21 or earlier — forcibly rebuilding a single-member cluster from one node's data could leave old peers behind; fixed in v3.5.22.
• --unsafe-no-sync enabled — a membership change can commit to v3store but crash before the write-ahead log persists it, in rare cases.

If your management cluster has never used --unsafe-no-sync, never gone through a forced single-member recovery, and was never restored from a pre-3.5 snapshot, this specific bug doesn't apply to you. If any of those happened even once in the cluster's history — including a disaster-recovery drill nobody wrote down — the drift may already be sitting in v3store waiting for a v3.6 binary to read it.

A second, unrelated bug with the same symptom​

Buried in the same upgrade window is a completely separate failure that etcd's maintainers had to explain in a follow-up post after operators kept hitting it: an upgrade that aborts outright with membership: too many learner member in cluster.

This one isn't about resurrected members — it's about members etcd still thinks are learners (non-voting) when they were actually promoted to full voting members years ago. The bug: promoting a learner writes the promotion to v2store, but a defect introduced in etcd v3.5.1 (PR #13348) failed to also commit it to v3store. Under etcd v3.5.x, that was invisible — v2store was still authoritative, so the promotion held. The moment a v3.6 binary boots and reads v3store instead, it sees the member as a stale learner. Etcd allows only one active learner at a time; if two or three "phantom learners" show up simultaneously, the upgrade won't bootstrap at all.

That bug is fixed in etcd v3.5.20 (PR #19563), with a further edge case for clusters sitting between v3.5.20 and v3.5.23 patched in v3.5.24 (issue #20797). Put together, etcd's maintainers converged on one recommendation: don't jump to v3.6.0 from anything earlier than v3.5.26. Land on v3.5.26 or later, confirm every member reports healthy, then move to v3.6.

Why that recommendation doesn't map cleanly onto Cluster API​

That guidance assumes an in-place upgrade path — the kind bare kubeadm does when you run kubeadm upgrade node and it bumps the etcd static pod's image tag on the same VM, same data directory, same member ID. You genuinely can stage through v3.5.26 first, because it's the same node running a newer binary against the same on-disk state.

KubeadmControlPlane doesn't upgrade that way. Per the kubeadm-based control plane proposal and the Cluster API Book, bumping spec.version triggers a rolling replacement: a brand-new Machine boots, runs kubeadm join --control-plane, and joins etcd as a fresh member — running whatever etcd version kubeadm has pinned to the target Kubernetes minor, not the source one. Only once that new member is healthy does KCP's pre-terminate hook remove the outgoing Machine's etcd member and delete it. Repeat per control-plane node, quorum preserved throughout.

That's a good design for availability. It's a gap for this specific upgrade. There is no window where a v3.5.26 binary runs against your cluster's existing membership state and triggers the automatic v2store→v3store repair etcd v3.5.26 ships. The first binary to read that state as authoritative is whatever kubeadm pinned for the target minor — and per kubeadm's own constants.go, that's v3.6.5-0 the instant you move to Kubernetes 1.34.

Kubernetes only supports single-minor version skips, so a fleet on 1.31 doesn't jump straight to 1.34 — it bumps spec.version through 1.32, then 1.33, then 1.34, one KCP rollout per hop. That doesn't buy you a safe landing spot: kubeadm pins etcd to v3.5.24-0 for 1.31, 1.32, and 1.33 alike, so every control-plane machine you replace on the way to 1.34 rejoins etcd still one patch below the v3.5.26 floor. The only hop that changes the etcd version at all is the last one, 1.33 → 1.34 — and it changes it straight to v3.6.5-0. A CAPI-managed fleet making that final hop crosses the exact boundary the etcd team warned about, in one Machine-replacement cycle, with no intermediate node in the entire multi-hop path ever having run the version that would have caught a latent v2store/v3store mismatch.

The pre-flight checklist​

Run this before bumping KubeadmControlPlane.spec.version across the etcd 3.5→3.6 boundary (currently: any jump from a 1.31–1.33-pinned management cluster to 1.34 or later).

1. Confirm which side of the boundary you're actually crossing. Check the etcd version live on each control-plane node (etcdctl endpoint status -w table) against the version kubeadm pins for your target Kubernetes minor. If both are already ≥ v3.5.26 or both are already on v3.6.x, this checklist doesn't apply — you're not crossing the risk window.

2. Audit the cluster's history for the three trigger patterns, not just its current state: has this management cluster ever run etcdctl snapshot restore from an etcd ≤3.4 snapshot, gone through a --force-new-cluster recovery, or run with --unsafe-no-sync? Check runbooks, incident retros, and any manual DR you performed by hand — this is exactly the kind of one-off intervention that doesn't show up in Git history.

3. Diff live membership against live Machines before touching spec.version. Run etcdctl member list -w table and cross-reference every listed member's ID and peer URL against kubectl get machines -n <namespace> for the workload — or management — cluster. Any member with no corresponding running Machine is a zombie candidate today, independent of the upgrade. Remove it with etcdctl member remove <id> before you proceed; don't let the version bump inherit a mess that already exists.

4. If step 2 turned up a real trigger history, don't let KCP's rolling replacement be the first thing to read v3store as authoritative. Take an etcd snapshot, spin up a scratch etcd v3.5.26+ instance against a copy of the data directory offline, and let its startup sync run to completion before you trust the production membership data. This is the manual substitute for the in-place "step through 3.5.26" path KCP's machine-replacement model skips.

5. Roll one control-plane node at a time and re-run the membership diff after each replacement, not just once at the end. KCP's pre-terminate hook removes the outgoing member automatically, but it can fail silently — a hook timeout, an infra provider force-deleting the VM before the hook runs, or a webhook misconfiguration all bypass it. Watch each Machine's EtcdMemberHealthy condition and the aggregate EtcdClusterHealthy condition on the KubeadmControlPlane object as your live signal; don't proceed to the next node until both report healthy.

6. Re-run the membership-vs-Machines diff once more after the full rollout completes. A stale member that slipped through mid-rollout is far cheaper to catch immediately than to discover the next time you're debugging a quorum-math discrepancy months from now.

None of this is exotic tooling — it's etcdctl, kubectl, and two conditions the CAPI API already exposes. The work is doing it deliberately, before the version bump, instead of assuming a rolling replacement is inherently as safe as an in-place upgrade.

Owning the boundary you're crossing​

A managed control plane hides this entirely — a vendor's SRE team decides when etcd crosses 3.6 and absorbs whatever went wrong in the membership history. Running your own Cluster API fleet means that boundary is yours to cross deliberately, checklist and all, in exchange for owning the machines underneath it.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with a Cluster API fleet underneath that you can actually inspect. Star the repo on GitHub or deploy your first app today.