etcd 3.7 Boots From the v3 Store: What Changes About Backups and Recovery on a Self-Hosted Fleet

The worst time to discover your backup procedure is stale is during a restore. etcd v3.7.0, released July 8, 2026, changes the store your control plane boots from: the server now starts entirely from the v3 store, cutting a bootstrap dependency on the legacy v2 store that has lingered since the v3.4 era. If you run stacked etcd on your own machines — a kubeadm control plane, a Cluster API management cluster plus its workload clusters — this is the release that touches the one component you can least afford to misunderstand.

The good news first: your snapshot files did not change format, your restore commands still work, and nothing about the 3.7 store upgrade forces you to rewrite your disaster-recovery runbook. The bad news: "nothing changed" is exactly the assumption that burned operators on the last store migration, when clusters jumping from etcd 3.5 to 3.6 hit zombie-member failures and lost quorum. This post gives you the concrete before-and-after, the three verdicts that matter (snapshot/restore, compaction, corruption recovery), and a fleet-wide backup drill to run before your next Kubernetes minor bump.

TL;DR: the 3.7 store change in one table​

In short: backups work the way they always did, the upgrade path deserves your attention, and v3.8 — not v3.7 — is the release that deletes the leftovers. The rest of this post substantiates each row.

What "bootstrap from v3store" actually means​

Since etcd 3.0, every cluster has carried two stores: the v2 store, a fossil from the pre-Kubernetes API, and the v3 store, the MVCC key-value engine that actually holds your cluster state. For several releases the server kept consulting the v2 store during startup — for membership data and v2 snapshot generation — even though nothing in Kubernetes reads v2 data anymore. Release after release chipped away at that dependency, and v3.7 finishes the job: the server bootstraps entirely from the v3 store.

For day-to-day operations this changes almost nothing, which is precisely why it deserves one careful paragraph. Membership data now has a single source of truth, and that single source is the same store your snapshots capture. Once a cluster is on v3.6 or later, the class of inconsistency where membership recorded in one store disagrees with the other cannot recur — the second store is out of the bootstrap path.

That sentence is worth believing because the previous migration proved what happens when it is not true. Clusters upgrading from etcd 3.5 (specifically versions between v3.5.1 and v3.5.19) straight to v3.6 failed with membership: too many learner member in cluster: voting members reverted to learners during the upgrade, the bootstrap refused to proceed with more than one learner, and long-removed "zombie" members reappeared in consensus. SIG etcd's standing guidance is unambiguous — upgrade to v3.5.26 or later first, so the cluster repairs itself, and only then move to v3.6. The lesson for 3.7 is the same shape: never jump more than one minor, never skip the patched intermediate, and treat the membership list (etcdctl member list) as a pre-upgrade check, not a post-upgrade surprise.

Snapshot/restore, compaction, corruption recovery: three verdicts​

This is the section the title promised, so here are the three verdicts plainly, each with the command it rests on.

Snapshot and restore: unchanged. The snapshot is still a point-in-time copy of the bbolt backend holding the v3 store, and v3.7 did not change its format. The procedure that worked on 3.5 and 3.6 works on 3.7: take the snapshot from a live member, check it, and restore it into a fresh data directory — never onto a live member, which corrupts Raft consistency.

etcdctl snapshot save /backup/etcd-snapshot.db
etcdctl snapshot status /backup/etcd-snapshot.db -w table
etcdutl snapshot restore /backup/etcd-snapshot.db --data-dir /var/lib/etcd-restored --timeout 60s

Two 3.7 details touch this flow without changing it. First, etcdutl commands — including the offline restore and status inspection you run during an incident — now accept a --timeout, so a repair command can no longer hang forever holding a backend lock while your control plane is down. Add the flag to every offline command in your runbook. Second, v3.7 still emits v2 snapshots for backward compatibility, with full removal deferred to v3.8. If any of your tooling consumes the v2 snapshot path, it keeps working today and breaks next minor — migrate it now.

Compaction and defragmentation: unchanged, still the likeliest thing to page you. Nothing in 3.7 alters auto-compaction, etcdctl defrag, or the quota machinery: the default backend quota is still 2GiB, crossing it still raises the NOSPACE alarm and makes the API server reject all writes with etcdserver: mvcc: database space exceeded, and the fix is still compact-then-defrag-every-member then disarm the alarm. The 3.7 performance work adjacent to this area — keys-only range reads served from the in-memory index, prioritized lease revokes, faster watch find() — lowers control-plane CPU. It does not move the quota, the alarm thresholds, or the defrag procedure. If your fleet has never raised NOSPACE on its Hetzner control-plane nodes, that is a property of your data size and retention, not of the etcd version — keep the defrag cron and the quota alert exactly where they are.

Corruption detection and recovery: unchanged, slightly less hang-prone. The recovery path is still snapshot-based: detect (consistency alarms, etcdctl endpoint status hash mismatches across members), then rebuild the cluster from a verified snapshot into fresh data directories on every member, using the same snapshot everywhere. What 3.7 improves is the tooling around the edges — etcdutl timeouts bound the offline inspection commands you run while diagnosing, instead of leaving you wondering whether the command or the disk is stuck. The v3store-only bootstrap additionally means a restored member starts through a simpler path with fewer legacy stores to disagree.

One deliberate non-verdict: RangeStream, the headline 3.7 feature, does not change backups at all. It streams large range reads in chunks instead of buffering whole result sets — a real fix for LIST-driven memory spikes on small control planes, consumed by Kubernetes 1.37 through the EtcdRangeStream gate — but snapshots, compaction, and recovery never go through the Range path. Do not let the headline feature distract the upgrade review; the store-bootstrap row in the table above is the one that concerns your restore procedure.

The fleet-wide backup drill, step by step​

A Cluster API fleet multiplies the backup problem: the management cluster holds the Machine and Cluster objects that describe every workload cluster, and each workload cluster holds its own etcd with tenant state. Losing either half without the other is a partial amnesia your runbook must cover. Run this drill once per fleet, before your next Kubernetes minor bump — it takes under an hour and it is the only proof your backups exist.

1. Inventory every etcd you own. List the management cluster plus each workload cluster, and for each, the endpoints, etcd version, and where its snapshots land. A cluster missing from this list is a cluster with no backup.
2. Snapshot each one from a live member. Run etcdctl snapshot save per cluster with a filename that records cluster, date, and etcd version. Copy each snapshot off the control-plane disk — a backup on the same volume as the database is a backup you lose in the same incident.
3. Verify every snapshot, not just the first. Run etcdctl snapshot status against each file and record the hash and revision. A snapshot that was never status-checked is a rumor.
4. Restore-drill one snapshot into a scratch directory. On a spare machine or an empty directory, run the etcdutl snapshot restore line from the previous section with --timeout set, start a single-member etcd on the restored directory, and read back a known key. You are testing the restore path, not the cluster — the drill passes when a cold read returns expected data.
5. Re-check alarms, quota, and defrag state per cluster. Run etcdctl alarm list and etcdctl endpoint status -w table on every member. Record database size against quota for each cluster; anything past half the quota gets a defrag scheduled before the upgrade window, not during it.
6. Write down the revision skew. Note each cluster's current revision alongside its snapshot's revision. After the drill you know, per cluster, how old a restore would be — which is the number that decides your snapshot frequency, not anyone's recommended default.

If any step fails — a snapshot that will not verify, a restore that will not start, a member already alarming — stop the upgrade planning and fix the backup first. An upgrade that breaks a cluster with a tested snapshot is an incident. An upgrade that breaks a cluster without one is a resume event.

What to do this week, and what v3.8 takes away​

If your fleet is still on etcd 3.5, the safe sequence is 3.5.26+ first, then 3.6, then 3.7 — the zombie-member saga of the last migration is the reason, and the July 23 patch releases (v3.7.1, v3.6.14, v3.5.33) are the maintenance tail that keeps each line safe to sit on. If you are on 3.6, the 3.7 upgrade is operationally calm: audit manifests for removed experimental flags, confirm no tooling depends on the v2 API packages or v2 snapshots, add --timeout to the offline commands in your runbook, and run the six-step drill above.

Then mark the calendar for v3.8. That is the release that deletes v2 snapshot generation and the --snapshot-count flag — the two leftovers 3.7 deliberately retained. Anything still referencing them after your 3.7 upgrade is technical debt with a published expiry date. The store migration that started back in the v3.4 era ends there; your fleet should arrive with verified snapshots, a drilled restore path, and nothing left that remembers the v2 store.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.