Hetzner's 45-Hour ASH Load Balancer Outage: What a Managed LB Costs a Fleet That Doesn't Run Its Own

At 11:00 UTC on July 13, 2026, Hetzner's Load Balancer service in its ASH (Ashburn) location started failing. It didn't come back for roughly 1 day and 21 hours — call it 45 hours, almost two full days. Every tenant application running healthy, fully-provisioned pods behind that Load Balancer was still unreachable, because the thing that broke wasn't the compute. It was the single shared object sitting in front of it.

If you're running a Cluster API fleet on Hetzner and you provision ingress the default way — a Kubernetes Service of type: LoadBalancer, wired through hcloud-cloud-controller-manager to Hetzner's managed Load Balancer product — this incident is worth doing the arithmetic on. Not "was Hetzner down" (bare-metal and cloud providers all have bad days), but specifically: what does depending on a vendor-managed L4/L7 balancer cost you when that one product, and only that product, goes dark for two days — and what would it have cost to not depend on it at all?

What actually broke, and what didn't​

A Hetzner Cloud Load Balancer is not a load-balancing algorithm running on your nodes. It's a separate managed service — its own fleet, its own control plane, provisioned and reconciled for you by hcloud-cloud-controller-manager whenever a Service object requests type: LoadBalancer:

apiVersion: v1
kind: Service
metadata:
  name: tenant-app
  annotations:
    load-balancer.hetzner.cloud/location: ash
    load-balancer.hetzner.cloud/use-private-ip: "true"
spec:
  type: LoadBalancer
  ...

That annotation is the entire interface. Hetzner provisions the LB, terminates TLS, health-checks your backend nodes, and hands back one IP that DNS points at. It's convenient specifically because you never think about it — until the two days it's the one component in the request path you can't SSH into, can't restart, and can't route around.

During the ASH outage, the failure mode was exactly this split: your Deployments were Running. Your pods were passing readiness probes. kubectl get pods looked fine the entire time. None of that mattered, because the IP that DNS resolves to, the TLS termination, and the health-check-driven routing all lived inside Hetzner's Load Balancer product — and that product was the thing down. Every tenant sharing that one LB went dark together, for the full duration, regardless of how healthy their own workload was.

This wasn't a one-off blip​

It's worth being precise about what "45 hours" actually was, because it wasn't one continuous flatline — it was worse in a specific way. Hetzner's own status history shows ASH's Load Balancer service degrading twice on July 13: a shorter ~2 hour 20 minute disruption starting at 8:40 UTC, then a second, much longer disruption starting at 11:00 UTC that ran roughly 1 day 21 hours before full resolution. A tenant paged on the first incident, watched it clear, stood down — and then got paged again less than three hours later into an outage that outlasted most teams' entire on-call rotation before it was over.

That pattern matters for the argument here: this isn't a story about one freak multi-hour fluke. It's a shared managed service failing twice in one day, the second time for nearly two full days, with the tenant able to do nothing but watch a status page between updates. A team's own incident-response runbook — the one built around "diagnose, mitigate, communicate, resolve" on a timescale of minutes to hours — simply doesn't have a branch for "the fix is out of our hands for two days."

The blast-radius math​

This is the part that's easy to underweight until you put a number on it. A single shared Load Balancer means the outage's duration is a tax charged identically to every tenant behind it, and the total damage scales with how many tenants you've stacked behind that one IP.

Take the 45-hour outage against a monthly uptime SLA. A 99.9% ("three nines") monthly SLA budgets 43.2 minutes of downtime a month (0.1% of 30 days). This one incident burned 2,700 minutes — about 62x that entire monthly budget, in a single event. Expressed as monthly uptime, 45 hours out of 730 lands at roughly 93.8%, which isn't "two nines" either — it's below it. One incident alone would blow through a three-nines or two-nines SLA for the month.

And that 45-hour figure is constant per tenant regardless of fleet size — every tenant behind the LB eats the same downtime percentage. What scales with fleet size is the aggregate damage, in tenant-hours lost:

A platform with 20 tenants behind one shared Hetzner LB didn't have "an outage." It had 900 tenant-hours of simultaneous downtime from a single upstream failure, none of it caused by anything the platform team's own code, config, or capacity planning did wrong — and none of it something the platform team could act on, because the broken component wasn't theirs to touch.

What it costs to not depend on it​

The alternative is to stop asking Hetzner's managed LB product to be the thing standing between DNS and your ingress controller, and put an operator-managed L4 balancer — kube-vip or MetalLB — on nodes you already run instead.

Mechanically, this is well-trodden ground for bare-metal and Cluster API operators:

• kube-vip in ARP/L2 mode: one pod per eligible node runs a leader election; the leader holds a virtual IP and answers ARP for it. On leader failure, the next node claims the VIP and emits a gratuitous ARP. Observed failover in this mode averages around 3 seconds, with worst cases bounded by how long neighboring hosts hold a stale ARP cache entry (typically well under a minute).
• MetalLB does the equivalent for Service IPs specifically (not control-plane HA), in either the same L2/ARP mode or BGP, if your network supports peering.
• The floating/virtual IP itself is a Hetzner Floating IP — the first is free per project, additional ones run about €2/month, reassignable between servers in the same location in seconds.

That's the entire infrastructure bill: a couple of euros a month, replacing a Load Balancer product that costs roughly €7.49/month for the smallest tier post-2026 pricing. The real cost isn't the euros — it's operational: you now own the failover testing, the ARP/VIP monitoring, the on-call runbook for "what do we do when the VIP doesn't move," and the initial setup work of getting kube-vip or MetalLB correctly wired into your node pools. That's real engineering time a managed product was previously absorbing for you, and it's the honest price of the swap.

Making the swap without a cutover outage of your own​

If the math above lands, the migration itself doesn't have to be risky. The sequence that avoids trading one outage for another:

1. Deploy kube-vip (or MetalLB) alongside the existing Hetzner LB, pointed at the same backend Service, but bound to a new Floating IP — not the one DNS currently resolves to. Let it run in production, health-checked and monitored, for a full deploy-and-rollback cycle before it carries real traffic.
2. Load-test and kill-test it deliberately: pull the leader node, confirm the VIP actually migrates and gratuitous ARP propagates within the expected few seconds, and confirm TLS termination and health-check behavior match what the managed LB was doing (this is usually where the gap shows up — a managed LB's health-check semantics aren't always a 1:1 match for what you configure yourself).
3. Cut DNS over to the new Floating IP with a short TTL, watch traffic shift, and only then decommission the Hetzner-managed LB Service object.

None of that requires downtime, but skipping the kill-test step is exactly how a team ends up rediscovering an outage they thought they'd designed out — the ARP failover path is the one piece of this whole setup you're now responsible for validating yourself, and it should be validated well before the day you actually need it.

What you actually buy back​

Here's the reframe that matters more than the euros: DIY doesn't make you immune to outages — it changes whose outage it is, and who can act on it.

Running your own L4 balancer still depends on Hetzner's underlying compute and network staying up. If the ASH datacenter itself has a bad day, your nodes go down whether or not you're running a managed LB product on top of them — self-hosting the balancer doesn't insulate you from Hetzner's infrastructure, only from Hetzner's managed Load Balancer control plane specifically.

But that's exactly the layer that failed here. A kube-vip leader-election failure is a node-level or software-level event: your own health checks catch it, your own leader election reacts in seconds, and if automated failover somehow doesn't fire, your own on-call engineer can SSH in and move the VIP by hand. None of that requires waiting on anyone else's incident timeline. A managed-LB-product failure, by contrast, is something you can only watch — file a support ticket, refresh the status page, and wait however long it takes the vendor's own team to fix a system you have zero visibility or control into. That gap — between "my team can act in minutes" and "we wait on someone else's 45-hour timeline" — is the actual thing 45 hours of downtime bought a fleet that stayed on the managed product.

When the swap is worth it​

The honest threshold: if you're running a handful of low-traffic services for yourself, the managed LB's convenience almost certainly outweighs the ops burden of running your own — a single-digit number of tenant-hours at risk doesn't justify owning VIP failover. But once you're operating a multi-tenant Cluster API fleet where a shared LB's downtime multiplies across every tenant behind it — the 900-tenant-hour, 4,500-tenant-hour end of the table above — the calculus flips. A couple of euros and a kube-vip DaemonSet is cheap insurance against a dependency you can't debug, can't restart, and can't act on when it breaks for two days at a time.

Bex.co runs its own Cluster API fleets on owned Hetzner hardware precisely so that a tenant's ingress path isn't hostage to a vendor's managed control plane the operator can't touch. Check out the project on GitHub if you're evaluating what it takes to own that layer yourself.