MetalLB vs kube-vip on Bare Metal: The Two-Day Hetzner Outage That Settled a Fleet's Load Balancer Choice

On July 13, 2026, Hetzner's managed load balancers in Ashburn (ASH) stopped answering. Not for an hour — for close to 44 hours, every service that outsourced its front door to Hetzner's Cloud Load Balancer in that zone was degraded or down. Tenants could not fix it themselves, because the load balancer was not theirs. If your PaaS ingress ran through it, your PaaS was down. If your control-plane endpoint ran through it, your ability to heal the fleet was down.

That is the cost of a managed L4 you do not own: when it breaks, you wait. The outage made one question urgent for every team running a Cluster API fleet on Hetzner dedicated hardware: what does it take to own your own L4, and which of the two standard bare-metal answers fits which job?

You came for a concrete choice. Here it is, before the why.

A self-hosted fleet has two L4 jobs, and they want two different tools. Use kube-vip for the Kubernetes API VIP — the floating address kubelets and CAPH controllers use to reach the control plane — because it runs as a static Pod before the API server exists. Use MetalLB for tenant Service LoadBalancers — the many IPs tenant apps need — because it manages a real IP pool and speaks L2/BGP. The combination that survives an audit of failover speed, true load balancing, and day-two ops is almost always both, in different roles.

The two jobs bare metal forces you to solve​

A cloud provider hides two problems behind one product. On bare metal — Hetzner dedicated servers and private networks you own — you solve both explicitly.

Job 1: the control-plane endpoint. Three apiservers need one address. Kubelets, CAPH controllers, and every kubectl client point at it. It must exist before the first apiserver is up, or nothing bootstraps. It must float if the leader dies. Lose it and you cannot reconcile Machines — not just serve traffic.

Job 2: tenant LoadBalancers. Every type: LoadBalancer Service expects an external IP allocated automatically, announced to the local network, and reclaimed on delete. This is many IPs, not one. Lose it and tenants cannot expose apps — but the control plane still reconciles.

One managed Hetzner Cloud Load Balancer pretended both were one product. That is why its outage took both down together. Owning the layer means separating them.

MetalLB: the allocator that speaks ARP and BGP​

MetalLB runs inside Kubernetes. Two components, two responsibilities.

The controller (a Deployment with leader election) watches type: LoadBalancer Services, allocates an external IP from a pool, and writes it to status.loadBalancer.ingress. It does not touch packets. The speakers (a DaemonSet, one per eligible node) announce that IP to the local network and forward arriving traffic via kube-proxy.

How they announce divides MetalLB into two modes.

L2 mode: one leader ARPs, everyone else waits​

For each Service IP, exactly one speaker is elected leader. It answers ARP (or NDP for IPv6) with "that IP is at my MAC." All traffic for the Service hits that one node, then kube-proxy spreads it cluster-wide. If the leader dies, another speaker sends a gratuitous ARP — "same IP, new MAC" — and traffic moves.

Failover is typically 3 to 10 seconds (memberlist detection plus grat ARP). Fast enough for most web workloads, but not hitless: every TCP connection through that IP breaks when the MAC moves, because L2 is failover, not load balancing. The docs are explicit: "in a strict sense, MetalLB does not implement a load balancer for layer 2." All traffic through one node is also a throughput ceiling tied to that node's NIC. For dozens of Services under a few gigabits, that ceiling rarely matters. For heavy edge bandwidth, it is why BGP exists.

BGP mode: true ECMP, if your network will peer​

In BGP mode every speaker with a healthy endpoint peers with your upstream BGP router and advertises the Service IP as a /32. The router installs ECMP routes to every advertising node and hashes flows across them. Every node receives traffic. Failover is BGP withdrawal — often sub-second per flow.

The catch on Hetzner: the fabric does not expose a BGP session to tenants. You can run your own routers (Bird or FRR on dedicated boxes, or your own leaf switches) and peer there, but you cannot point MetalLB at Hetzner's fabric and say "peer with me." In practice almost every CAPH fleet on Hetzner runs MetalLB in L2 mode — it needs only a flat L2 segment and permission to send grat ARP. Choose BGP only if you brought your own BGP infrastructure.

Modern configuration uses CRDs, not the old ConfigMap:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: tenant-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.20.30.100-10.20.30.200

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: tenant-l2
  namespace: metallb-system
spec:
  ipAddressPools:
  - tenant-pool

Allocate once, and every LoadBalancer Service gets an address from that range.

kube-vip: the static Pod that solves bootstrap​

kube-vip floats a VIP with ARP or BGP, but its architecture is different in the way that matters. It runs as a static Pod — a manifest in /etc/kubernetes/manifests the kubelet watches directly, without an API server. That solves the chicken-and-egg: you need a VIP to reach the control plane, but you need the control plane to schedule the thing that provides the VIP. Static Pods break the cycle.

Leader election uses the standard Kubernetes mechanism (or Raft), bootstrapped from the local apiserver config so it starts before the VIP is held. The elected leader ARPs the VIP — 10.20.30.10 is at my MAC — and every kubelet's server: https://10.20.30.10:6443 reaches the live apiserver. If that node dies, another host wins election, grat-ARPs, and the VIP moves. Same 3-to-10-second order as MetalLB L2, same broken-connections-during-move semantics — but one layer earlier, before Kubernetes considers itself scheduled.

Generate it from the image itself, no Helm needed:

VIP=10.20.30.10
IFACE=eth0
KVVERSION=v0.8.3
ctr image pull ghcr.io/kube-vip/kube-vip:${KVVERSION}
ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:${KVVERSION} vip \
  /kube-vip manifest pod \
  --interface ${IFACE} \
  --address ${VIP} \
  --controlplane \
  --leaderElection \
  --arp | tee /etc/kubernetes/manifests/kube-vip.yaml

Drop that file on every control-plane machine. kubeadm init --control-plane-endpoint 10.20.30.10:6443 now has a stable endpoint even though only the first machine exists. CAPH does the same: controlPlaneEndpoint.host is the kube-vip address, rendered into machine bootstrap.

kube-vip can also serve tenant LoadBalancers with --enableLoadBalancer --services. It works for a handful of Services, but its Service mode is a single-IP leader election without MetalLB's pool management or share controls. At tens of Services, the pool allocator is the tool you want for Job 2.

Head to head: eight dimensions that decide​

This is the table the title owes you — practice, not README promises.

MetalLB owns the many-IP tenant problem and is the only one that can do true ECMP — but only with a peer Hetzner does not provide. kube-vip owns the before-Kubernetes-exists problem and is the only one that can give you a VIP without a control plane to schedule it. Neither covers both well. Together they do.

The verdict: use both, in different roles​

The boring architecture that has run longest on CAPH and bare-metal kubeadm is:

• kube-vip on every control-plane host as a static Pod, ARP, leader election. Owns exactly one address: the apiserver VIP. CAPH or kubeadm points controlPlaneEndpoint at it. If all apiservers die, kube-vip still runs — the kubelet still runs static Pods.

• MetalLB in metallb-system, L2 mode, IPAddressPool from your private range. Owns every other LoadBalancer Service. Tenants get IPs by declaring a Service; the platform never hand-manages one. One L2Advertisement on eth0 is enough.

Why not just kube-vip for everything? For five Services it is defensible. Beyond that, MetalLB's autoAssign, pool reclaim, and share selection repay the second component. kube-vip's Service mode has no IPAddressPool equivalent that scales.

Why not just MetalLB for everything? Because MetalLB cannot bootstrap the control plane. The topology that puts a Hetzner Cloud LB in front of the control plane and MetalLB behind it is exactly what went down for two days. If the front door's owner is not you, its outage is not yours to fix.

Why L2 not BGP on Hetzner? Unless you run your own BGP routers, there is nothing to peer with. L2's single-node ingress ceiling is real — benchmark node_network_receive_bytes on the leader — but it is a ceiling you own. For the fleet size Bex-class operators run (3 CP nodes, 3–10 workers, dozens to low hundreds of Services) L2 carries the load.

The converged fleet:

control plane (3× Hetzner, private net 10.20.30.0/24)
  /etc/kubernetes/manifests/kube-vip.yaml → VIP 10.20.30.10 (ARP, leader elected)
  apiserver at 10.20.30.10:6443 via controlPlaneEndpoint
 
workers (3–10× Hetzner, same private net)
  metallb-system/IPAddressPool 10.20.30.100-200 → tenant Services
  speaker DaemonSet, L2Advertisement on eth0
  ingress (Envoy Gateway / Cilium / NGINX Gateway Fabric) on one MetalLB IP

Two addresses, two components, two failure domains. One can move without the other. That separation is why July would not have taken this fleet down.

Migrating off Hetzner Cloud LB without a flag day​

You do not need to rebuild workload clusters.

1. Carve the range. From your existing private network, reserve one IP for kube-vip and a contiguous range for MetalLB. Do not overlap DHCP or any managed LB subnet you keep during cutover.

2. Install kube-vip. Render the manifest above and drop it to /etc/kubernetes/manifests/kube-vip.yaml on every control-plane machine. In CAPH this is a KubeadmControlPlane file or ignition snippet, not a Helm release. Verify kubectl --server https://<vip>:6443 get nodes reaches the API via the VIP before changing controlPlaneEndpoint.

3. Point the control plane. Update controlPlaneEndpoint to the kube-vip address and let CAPH roll the control plane. Keep the old LB in DNS with a low TTL until the roll completes.

4. Install MetalLB. helm install metallb metallb/metallb -n metallb-system --create-namespace, then apply the pool and advertisement. Test with a canary:

apiVersion: v1
kind: Service
metadata:
  name: lb-canary
spec:
  type: LoadBalancer
  selector:
    app: canary
  ports:
  - port: 80

Confirm EXTERNAL-IP populates, arping <ip> resolves to the leader's MAC, and cordoning that leader moves the MAC within seconds with only in-flight connections resetting.

5. Retire the managed LB. Remove hcloud-ccm annotations or HetznerCluster LB references, delete the managed object, reclaim the IP. No BGP session to de-peer — that is the point.

Two caveats: some virtualized private networks rate-limit grat ARP — watch tcpdump -i eth0 arp during a real failover before declaring done. And L2's single ingress node is observable — alert on metallb_speaker_announced and the leader's NIC saturation, and load-test one Service IP to that ceiling before tenants do.

What owning the front door buys you​

A two-day managed-LB outage does not prove managed services are bad. It proves the one network object every tenant and every controller shares should not be the one whose repair window you cannot control. Hetzner posted the incident — posting is not fixing — and for that window every fleet behind a managed front door elected to wait.

Owning the L4 trades one dependency for two components you operate. You now watch leader elections and speaker health yourself. It wins because both degrade in ways you can reason about — cordon a node, move a VIP, roll a DaemonSet — not in a way that needs a provider's incident commander to queue your recovery.

The July ASH outage was a reminder, not an exception. The question is not whether your provider will have its next L4 incident, but whether your fleet's front door will be among the objects it can take with it.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. The fleet underneath is Cluster API on Hetzner bare metal, where kube-vip holds the control-plane VIP and MetalLB owns tenant LoadBalancers — the same split this post recommends. Star the repo on GitHub or deploy your first app today.