etcd's fsync Tax: Why Your Kubernetes Control Plane Needs Local NVMe, Not a Cloud Volume

Your Kubernetes API server isn't slow. Your etcd is on the wrong disk.

Every write to etcd — every Deployment scale, every Secret update, every lease heartbeat from every kubelet — hits disk through fsync before etcd acknowledges it. The etcd docs put a hard number on that cost: the p99 of wal_fsync_duration_seconds must stay under 10 milliseconds, and backend_commit_duration_seconds under 25 milliseconds. Miss that budget by a factor of two and you don't get a warning — you get leader elections, API timeouts, and controllers that look "flaky" while the disk quietly falls behind.

On Hetzner, the distance between "safe" and "never" is the difference between a local NVMe and a Cloud Volume. The fio test that predicts which side you're on takes 30 seconds to run, and the answer is already in this table:

If you only remember one number from this post, remember 10 milliseconds. Everything below explains how to measure it before your control plane tells you the hard way.

Why etcd Taxes the Disk Differently Than Your App​

Most workloads care about throughput — megabytes per second, IOPS under batch load. etcd cares about one thing: how long it takes to durably commit a tiny write.

etcd uses two disk paths, both synchronous:

• WAL fsync — every committed Raft log entry is appended to the write-ahead log and fdatasync'd before the peer acknowledges it. The write is small — on the order of a couple kilobytes — but it must be durable. A single slow fsync stalls that commit, and every API write that depends on it.
• Backend commit — periodically, etcd fsyncs its BoltDB-backed key-value store to disk. Same pattern, larger write, same sensitivity to latency.

Either path can gate availability. A disk that benchmarks at 300 MB/s sequential throughput can still stall etcd for 40 milliseconds per fsync if its synchronous latency is high — and throughput benchmarks won't show you that. The etcd FAQ is explicit: "To rule out a slow disk from causing this warning, monitor wal_fsync_duration_seconds (p99 duration should be less than 10 ms) to confirm the disk is reasonably fast," and for backend_commit_duration_seconds, "p99 duration should be less than 25 ms."

That is why a control plane that feels "flaky under load" is often not a CPU problem, not a network problem, and not a Kubernetes bug. It is a disk-latency problem wearing a Kubernetes costume. Controllers that list and watch aggressively make it worse — more writes, more fsyncs, faster queue backup — but the root cause was the disk the moment etcd was placed on it.

The One fio Command That Predicts etcd Health​

You don't need to run etcd to test whether a disk is fast enough for etcd. The etcd and OpenShift docs recommend the same fio pattern that IBM's cloud team documented for this exact check: small sequential writes with fdatasync after every write, at the write size etcd actually emits.

# Run on the path where /var/lib/etcd will live
mkdir -p ./test-data
fio --rw=write \
  --ioengine=sync \
  --fdatasync=1 \
  --directory=./test-data \
  --size=22m \
  --bs=2300 \
  --name=mytest

What each flag emulates:

• --ioengine=sync — synchronous I/O, no async batching. etcd issues synchronous writes.
• --fdatasync=1 — fdatasync after every write. This is the fsync tax in isolation.
• --bs=2300 — 2.3 KB block size. Small, like a WAL entry. Throughput-oriented 1 MB blocks hide latency; this size exposes it.
• --size=22m — enough writes to get a distribution, short enough to run anywhere.

Read the fdatasync percentiles at the bottom of the output, not the bandwidth. fio reports them in microseconds:

  fdatasync
    lat (usec): min=1200, max=42000, avg=2800, stdev=3100
   lat percentiles (usec):
    | 50.00th=[  2100],  95.00th=[  5200],  99.00th=[ 11500],  99.90th=[ 38000]

That 99.00th=[11500] is 11.5 milliseconds — already over the 10 ms budget. The disk in this trace would be marginal to unsafe for etcd under any real load. A healthy local NVMe on Hetzner reports a 99th closer to 1,500–3,000 µs (1.5–3 ms).

Run this before you place etcd, not after the API server starts timing out. Run it on the exact mount that will hold /var/lib/etcd — not on a neighboring volume — and run it under the I/O conditions the node will actually see (more on sensitivity below).

The Numbers: Local NVMe vs. Hetzner Cloud Volumes​

The critical distinction on Hetzner is not "SSD vs NVMe." It is local vs network-attached.

• Local NVMe on Hetzner Cloud (CX, CPX, CCX) and on dedicated machines (AX, EX) is a physical NVMe device in the hypervisor or server. A single node benchmarks around 30,000 random 4 KB IOPS and 2.5–3.6 GB/s sequential throughput on the cheapest CX22, with a 99th fsync near 1–3 ms in a clean test. That is the Ankra 2026 six-cloud benchmark window: "€5.49 buys local NVMe posting 2.5 to 3.6 GB/s sequential, 30k random IOPS, and a 2.5 ms fsync p99, on the cheapest node in either round by a factor of four."

• Cloud Volumes are Ceph-backed network-attached block storage. Hetzner publishes no IOPS guarantee, and community testing consistently reports roughly 300–400 IOPS for a modest volume and ~240 write IOPS through a Longhorn replicated layer — a 15.6× penalty versus NVMe-backed Longhorn on the same fleet. Latency follows the same split: median fsync on a quiet volume can look reasonable at 5–8 ms, but the 99th under ordinary neighbor contention lands at 15–40 ms, with spikes past 80 ms. Those are exactly the percentiles etcd's budget forbids.

The pattern holds across orders of magnitude: the MySQL-on-Hetzner storage comparison that ran fio head-to-head reported local NVMe at ~30,000 IOPS and 1,100 MB/s against a Cloud Volume at ~300–400 IOPS and 1.3 MB/s sequential in the same test harness — two orders of magnitude on throughput, and more than an order of magnitude on synchronous latency.

Why Cloud Volumes vary: sensitivity, not cherry-picking

A single number per disk type is misleading on Ceph. The same Cloud Volume will report different fsync latencies depending on three variables that all matter in production:

• Provisioned size and baseline IOPS — larger volumes get more backing placement groups and higher ceilings. A 10 GB test volume will show worse tail latency than a 100 GB volume on the same cluster, but both share the same network path and Ceph quorum overhead.
• Neighbor contention — Ceph is shared. A volume that tests at p99 12 ms at 3 a.m. can test at p99 35 ms when the underlying OSDs are busy. Local NVMe does not have this variable.
• Fill and replication pressure — Longhorn or any replicated layer on top of Volumes multiplies the fdatasync fanout. Three replicas on Ceph-backed disks is not three times slower — it is often 10–15× slower than the same replicas on local NVMe, because every replica pays the Ceph tail.

Local NVMe is not immune to variance either, but its variance is bounded differently: noisy neighbors on a cloud hypervisor can add a millisecond or two to the tail, and a degraded RAID1 rebuild on dedicated hardware shows up as a consistent offset, not a 5× spike. On Ceph, a 5× spike is not an outlier — it is the operating model under contention.

That variance is the structural reason for the placement rule below. It is not that a Cloud Volume can never pass a single fio run. It is that its p99 under contention routinely fails the budget a single-tenant disk passes even under load, and etcd has no ability to wait out a burst.

Decision Matrix: Where to Put etcd, Where Never​

The underlying principle is simple: etcd is the one workload in the cluster that does not tolerate a storage tail. Your app database can retry a slow write; etcd's quorum cannot wait out a slow fsync without the whole control plane noticing.

On a Cluster API fleet this choice is made once, in the machine image's mount layout or the KubeadmControlPlane's extraMounts, and then inherited by every control-plane node. Getting it wrong means every future etcd member is wrong. The fio check above is the pre-flight before you bake the image.

Catch It Before the API Server Times Out​

If etcd is already running, fio is the lab test; Prometheus is the vital sign. Three metrics tell you whether the disk is winning or losing, before clients see errors:

• etcd_disk_wal_fsync_duration_seconds_bucket — the direct fsync latency histogram. Alert on the 99th percentile.
• etcd_disk_backend_commit_duration_seconds_bucket — backend commit latency. Same histogram shape, looser threshold.
• etcd_server_leader_changes_seen_total — the symptom after the fact. A rising leader-change rate confirms the latency is already causing elections.

PromQL that matches the etcd and Red Hat alert recipes:

# WAL fsync p99 — budget is 10 ms (0.01 s)
histogram_quantile(0.99,
  rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
) > 0.01
 
# Backend commit p99 — budget is 25 ms (0.025 s)
histogram_quantile(0.99,
  rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
) > 0.025

Wire these as alerts, not dashboards alone. The etcd docs link high commit latency directly to "high request latency or make the cluster unstable," and OpenShift's etcd performance guidance treats sustained p99 above 10 ms as the "disk is too slow" signal — not a tuning hint. etcd_mvcc_db_total_size_in_bytes vs. etcd_server_quota_backend_bytes is the companion check: a growing DB amplifies every slow fsync's cost, and a defrag or compaction backlog on a slow disk makes the next spike worse.

When an alert fires, the remediation is placement, not parameter tuning. Moving etcd to a dedicated local disk — or, on cloud, to a larger instance type with locally attached NVMe rather than a larger Volume — is the fix the etcd FAQ itself suggests: "assigning a dedicated disk to etcd or using faster disk."

Fleet Practice on Hetzner With Cluster API and kubeadm​

For a self-hosted PaaS whose control plane runs on Hetzner, the operational checklist is short because the failure mode is narrow:

1. Give etcd its own mount on local disk. On cloud VMs, ensure /var/lib/etcd sits on the instance's local NVMe, not on a mounted Volume. On dedicated machines, prefer a dedicated partition or a second NVMe over a shared root filesystem that also absorbs image pulls and journal writes. In Cluster API, that mounts is declared in the KubeadmControlPlane or the node image's cloud-init, not hand-edited per host.

2. Never declare a Cloud Volume as the etcd dataDir. It is tempting because Volumes are resizable and "highly available," but availability of the block device is not the bottleneck — tail latency of its fsync path is. A Volume that survives a host failure by reattaching to another host still carries the same Ceph tail after it reattaches.

3. Verify with fio on the node, not on your laptop. SSH to a control-plane node and run the exact command from the section above against the live etcd mount. Record the 99th and 99.9th, not just the average. Re-run it under representative load — a kube-burner or a clusterloader2 smoke against the API server — to see the tail under contention. Local NVMe's tail should stay under 10 ms; a Volume's tail will not.

4. Scrape etcd metrics from the control plane. etcd exposes /metrics on its client port and optionally on --listen-metrics-urls. Scrape that endpoint into the same Prometheus that watches your tenants, and set the two alert thresholds from the previous section at 10 ms and 25 ms respectively. Third-party distribution guides (including Red Hat's OpenShift etcd practices) use exactly those PromQL shapes — they are the fleet-standard check, not a custom invention.

5. Test backups and restores on the same disk topology. etcd's snapshot and restore path also exercises the fsync budget. A backup that succeeds on a quiet volume and then restores slowly during an incident is not a backup guarantee. Test snapshot latency alongside wal_fsync so the restore does not become the second incident.

These are not performance tweaks. They are the difference between a control plane that recovers from a load spike and one that elects a new leader in the middle of your tenants' deploys.

The Control Plane Is Only as Fast as Its Slowest fsync​

Kubernetes abstracts a great deal — scheduling, networking, storage classes — but it does not abstract etcd's fsync tax. Every level of indirection you add between etcd and a local NVMe adds latency to the one path the control plane cannot make asynchronous.

Hetzner makes the right choice easy to buy and the wrong choice easy to reach for: a Cloud Volume is one click, resizable, and billed per gigabyte, while the local NVMe is just "the disk that came with the box." The fio run in this post is how you tell them apart by the number that actually gates control-plane health — not capacity, not throughput, but synchronous latency under 10 milliseconds.

Run the fio command on your control-plane image before you cut the cluster over. Keep etcd on local NVMe. Alert on the 99th before your tenants notice the 50th.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.