One Rack Switch Died in Nuremberg. Here's What a Hetzner Cloud Leaf Fault Actually Costs a Cluster API Fleet

At 08:37 UTC on July 24, 2026, a network switch serving one rack in Hetzner's Nuremberg (nbg1) data center faulted. The rack is identified on Hetzner's status page as nbg1-cloud1-leaf88 — a single top-of-rack "Cloud Leaf" switch. Every tenant whose servers happened to be wired through that leaf lost Load Balancer, Networks, Cloud Server, and Object Storage simultaneously, until Hetzner's own incident update read: "We have fixed the error. All systems are reachable again."

That's a real, dated, named incident, not a hypothetical. And the honest question for anyone running a Cluster API fleet on Hetzner isn't "should we add a second cloud vendor as a failover target" — multi-cloud abstraction is a deliberate non-goal for a platform like this, not an oversight. The question is narrower and more answerable: what does a single rack's switch failing actually do to a running Cluster API Provider Hetzner (CAPH) cluster, how often does it happen, and what mitigation does CAPH actually expose today versus what it doesn't.

What a "Cloud Leaf" Fault Actually Breaks​

A leaf switch is the top-of-rack device a slice of a Hetzner location's cloud fleet connects through — the first hop between a server's network interface and everything else. When it faults, the server itself doesn't necessarily go dark; its network path does. Hetzner's status page listed four affected systems for the July 24 incident, and each one maps to a distinct thing that stops working, not just "the internet is slow":

• Load Balancer — Hetzner's managed LB service, which is what most CAPH clusters put in front of an Ingress controller or Gateway API implementation.
• Networks — the private network layer node-to-node traffic, the CNI overlay, and kubelet-to-API-server calls all ride on.
• Cloud Server — the Hetzner Cloud API that creates, deletes, and queries servers — the thing CAPH's controllers actually talk to.
• Object Storage — Hetzner's S3-compatible bucket service.

Four different failure surfaces, one root cause, one rack. That combination is the whole reason this incident is worth more than a status-page footnote.

What It Actually Costs a Running CAPH Fleet​

This is the part a status page summary skips: what each of those four degraded systems does to a cluster that's already up and serving traffic when the leaf goes down.

Load Balancer degraded → traffic drops regardless of pod health. If a CAPH cluster's Ingress or Gateway API implementation routes through a Hetzner LB behind that leaf, requests fail or time out even though the backend pods are perfectly healthy. Nothing in the application layer is broken; the front door is.

Networks degraded → nodes flip NotReady. Any worker node physically connected through the faulted leaf loses its private-network path to the rest of the cluster. Kubelet heartbeats to the API server stop landing, the node controller marks it NotReady, and the scheduler starts evicting and rescheduling pods that were running there — a self-inflicted rebalancing storm layered on top of the actual outage. If a control-plane node's etcd member happens to sit behind the same leaf as another member, that's a live split-brain risk, not just a worker-node inconvenience.

Cloud Server API degraded → the control plane stalls, not just the data plane. This is the one that's easy to miss because nothing visibly crashes. CAPH's HCloudMachine reconciler talks to the Hetzner Cloud Server API to create, delete, and query servers. While that API is degraded, a MachineDeployment scale-out event queues instead of executing, and replacing a node that crashed for an unrelated reason during the same window simply doesn't happen until the API recovers. The cluster's self-healing loop — the entire premise of a declarative, Cluster-API-managed fleet — is itself paused.

Object Storage degraded → backups and registry pulls fail mid-window. Anything reading from or writing to Hetzner's S3-compatible storage during the fault — a registry cache, a Velero or etcd backup job, a tenant workload persisting objects — fails for the duration, not gracefully deferred, just failed.

Put together: a single rack switch dying doesn't just cost "some latency." It can simultaneously break inbound traffic, trigger a node-eviction storm, freeze the cluster's ability to self-heal, and fail whatever backup job happened to be running — four independent-looking failure modes with one shared cause.

Not a One-Off: Five Leaf Faults in Six Days​

The instinct is to read a single incident as a freak event. Hetzner's own status page says otherwise. In the six days from July 24 through July 29, 2026, its incident history logged five separate Cloud Leaf fault entries spread across all three of Hetzner's European locations:

That's not a single unlucky rack — it's a recurring pattern across Hetzner's entire European footprint, on a roughly one-a-day cadence during that stretch. Duration varies too: most of these resolve within hours, but a prior leaf fault — nbg1-cloud1-leaf107, starting June 29, 2026 — took nearly three days to fully clear, with Hetzner's update at the time reading only "our technicians are working hard on analyzing the cause of the interruption." A fleet planning around leaf faults has to budget for the multi-day tail, not just the typical case.

What CAPH's Failure Domains Actually See​

Here's the mismatch: Cluster API's failure-domain concept exists precisely to let a platform spread machines so one physical failure doesn't take the whole cluster with it. But the unit CAPH understands is the Hetzner location — fsn1, nbg1, hel1 — not the rack or leaf underneath it. HetznerCluster's controlPlaneRegions field accepts a list of locations, and CAPI's control-plane provider spreads control-plane machines across whatever list you give it. Worker MachineDeployments can independently target a region too.

Below that, Hetzner's own placement groups — the spread type — guarantee that servers in the same group land on different physical hosts, up to 10 servers per group. That's real isolation at the server level. But Hetzner's own documentation stops there: nothing in a placement group's contract promises separation at the rack or leaf level. Two servers in the same spread placement group can still end up behind the same leaf switch, because "different physical server" and "different top-of-rack switch" aren't the same guarantee.

In short: there is no CAPH-level or Hetzner-level primitive that lets an operator pin machines to different leaves within a single location. A cluster confined to nbg1 alone is always exposed to whatever leaf its nodes happen to land behind — placement groups included.

The Concrete Mitigation: Spread Across Locations, Not Providers​

The one lever that actually exists is coarser than "avoid this leaf" — it's "leave this location." Hetzner's eu-central network zone spans all three EU locations — fsn1, nbg1, and hel1 — as a single private network zone. Servers in any of the three can share one Hetzner Cloud Network and talk to each other over private IPs, routed through Hetzner's gateway. That means a CAPH cluster can list more than one location in controlPlaneRegions and split worker MachineDeployments across locations too, all while staying inside one Hetzner project, one network, one provider.

Since nbg1-cloud1-leaf88, fsn1-cloud2-leaf17, and hel1-cloud1-leaf16 are three different leaves in three different physical buildings, a fleet spread across all three locations doesn't share a fault domain with any single leaf incident — the July 24 Nuremberg fault would have degraded only the fraction of the fleet actually provisioned in nbg1, not the whole cluster. That's a meaningfully smaller blast radius than a single-location deployment, and it costs nothing in cross-cloud complexity: it's still one provider, one API, one billing relationship — the opposite of the multi-cloud abstraction layer this kind of platform deliberately doesn't build.

It's also not a free lunch. Cross-location traffic inside eu-central still routes through Hetzner's gateway rather than a flat LAN, and a fleet that's small enough to run comfortably on a handful of servers in one location takes on real operational overhead spreading across three — more failure domains to reason about, not fewer, even if each one is individually smaller.

The Honest Bottom Line​

Leaf-level isolation isn't a knob CAPH exposes, and it isn't one Hetzner's own placement groups promise either — that ceiling is real, and no amount of YAML changes it. What is available is location-level spread within eu-central, which turns "the whole fleet in one location goes down when one rack's switch does" into "the fraction of the fleet in that one location goes down." That's the actual mitigation on the table today, not a hypothetical future feature.

For a single-location fleet, the honest framing isn't "this won't happen to us" — Hetzner's own status page logged five of these in six days across three locations. It's "when it happens, it costs us a stalled control plane, a node-eviction storm, and a failed backup window for however long that leaf takes to fix, and sometimes that's three days, not thirty minutes." Whether that's an acceptable tradeoff against the operational cost of spreading a small fleet across locations it doesn't otherwise need is a real decision — but it should be made with the actual incident history in hand, not discovered the next time a rack switch in Nuremberg dies.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, on a Cluster API fleet you control the topology of. Decisions like where to draw a cluster's failure domains are exactly the kind of infrastructure tradeoff a self-hosted platform has to get right on behalf of every tenant routing traffic through it. Star the repo on GitHub or deploy your first app today.