Cilium LB-IPAM and BGP on Bare Metal: What Dropping MetalLB Looks Like on a Hetzner Fleet

MetalLB's 0.16 line now ships an FRR-K8s DaemonSet and webhook as its default BGP backend, even on clusters that only ever announce over L2. If your Hetzner dedicated fleet runs MetalLB purely for L2 announcements over a vSwitch, you are now carrying a second routing stack you never asked for — alongside the CNI that could have done the whole job. This post works out the concrete trade: what collapsing the load-balancer layer into Cilium's LB-IPAM changes for a Cluster-API fleet with no cloud load-balancer API, what the migration costs service by service, and where the one honest catch sits.

The short version: on Hetzner dedicated servers there is no cloud API that can materialize a load balancer, so every workload cluster already self-announces its service IPs, today usually through a MetalLB speaker pool answering ARP on a vSwitch VLAN. Cilium can absorb both halves of that job — LB-IPAM allocates the IPs, L2 announcements or the BGP control plane advertises them — and remove a DaemonSet from every node. But the BGP half needs a router to peer with, and Hetzner's documented dedicated-server primitives top out at routed subnets, the vSwitch L2 segment, and Robot-driven failover IPs. So the realistic migration is LB-IPAM plus L2 over the same vSwitch you already run, with BGP reserved for the day you operate your own routers.

The rest of this post substantiates every cell: the starting architecture, the three Cilium primitives, the Hetzner peering catch, a phased migration sketch, and the honest ledger.

The starting point: one subnet, one VLAN, one speaker pool​

Start with the premise the title assumes. On Hetzner Cloud, the cloud-controller-manager provisions managed load balancers for LoadBalancer services, the same way any cloud CCM does. On Hetzner dedicated servers there is no equivalent API — no call you can make that returns a load balancer — so each Kubernetes cluster announces its own service IPs onto a network Hetzner routes to you. The community's documented pattern is MetalLB in L2 mode over a vSwitch: order a routed public subnet, attach it to the vSwitch, and let the MetalLB speakers claim addresses with ARP (walkthrough from October 2025, using Calico over VXLAN plus WireGuard, kube-proxy IPVS with strictARP, and ingress-nginx).

The vSwitch itself is an 802.1Q VLAN (ID 4000–4091) spanning your dedicated servers, with free private addressing inside it and orderable public subnets usable from any member server. Public vSwitch traffic includes 1 TB per month, with each additional terabyte of outgoing traffic at €1.00, incoming and internal free (Hetzner vSwitch docs). The MetalLB side needs the usual ARP hardening on every node — arp_ignore, arp_announce, relaxed rp_filter — so VIPs move cleanly between speakers without conflicting replies from the pod network.

For a Cluster-API fleet, multiply this picture per workload cluster. Each cluster gets its own MetalLB pools and advertisements, typically carved as slices of the vSwitch-routed subnet, and each cluster's speaker pool is an independent failure domain. Each cluster is the migration unit — you move one at a time, each reversibly — and the cost multiplier: every component you delete disappears from every node of every cluster.

The 2026 pressure on this setup is not that L2 mode stopped working but that MetalLB's center of gravity moved: the 0.16 line makes FRR-K8s the default BGP backend, deprecates the legacy FRR mode, and ends the native Go BGP implementation, which means an L2-only install now has to consciously disable the BGP machinery to stay lean (MetalLB installation docs). Operators upgrading through 0.16 report exactly that maneuver: keep the native L2 speaker, switch the FRR-K8s subchart off. When your load balancer's default install assumes a BGP deployment you do not have, asking whether the CNI can take the whole job is not fashion — it is maintenance hygiene.

Three primitives, two jobs​

Cilium splits the load-balancer problem into allocation and advertisement, and the split is the first thing to internalize, because "replace MetalLB with Cilium" really means adopting one allocator and one announcer. LB-IPAM allocates: it assigns IPs from CiliumLoadBalancerIPPool objects to LoadBalancer services. It is always enabled but dormant — the controller wakes when the first pool appears — and pools support CIDR blocks, explicit start/stop ranges, per-pool service selectors, and the first/last-IP reservation for routed ranges (LB-IPAM docs). Advertisement is a separate choice — L2 announcements for same-subnet ARP/NDP, or the BGP control plane for routed propagation — and the two fail and upgrade independently, so don't conflate them when planning.

The BGP control plane speaks BGP from every cilium-agent to configured peers, advertising LB-IPAM-assigned addresses (and optionally pod CIDRs) wherever you tell it to. Configuration is the BGPv2 API — CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, plus node overrides — introduced in Cilium 1.16 (1.16 announcement), promoted from v2alpha1 to v2 in the 1.17 timeframe (promotion PR), with the old CiliumBGPPeeringPolicy v1 API deprecated and later removed so current docs describe only the v2 resources (configuration reference). Practical notes: the agent initiates peering by default and only listens when you set localPort (port 179 needs CAP_NET_BIND_SERVICE), and peers can be auto-discovered from the default gateway.

The MetalLB-to-Cilium mapping is nearly one-to-one, which is why the migration reads as tedious rather than scary:

One row deserves its own warning: L2 announcements still carry the Beta label in the Cilium docs, on the same page that documents LB-IPAM as the stable allocator. You are trading MetalLB's years-hardened L2 speaker for a younger implementation inside your CNI. Operators are making that trade in production — public migration logs show MetalLB-to-Cilium moves completed service by service, including one driven by MetalLB's L2 leader election breaking against Cilium Gateway API EndpointSlices (example migration, forcing-function case) — but go in with the Beta label priced into your canary plan, not discovered during it.

The Hetzner catch: BGP needs a peer, and that peer is yours​

Here is the sentence the "BGP peering straight from cilium-agent" framing needs: peering with what? Cilium's own docs describe the BGP control plane as the tool "for environments that support BGP" — a radio with nobody listening if no peer exists. Hetzner's documented dedicated-server networking gives you routed subnets, the vSwitch L2 segment, and failover IPs you re-point through Robot in 90–110 seconds (failover docs): no BGP session among them. So on Hetzner dedicated, the router on the other end is infrastructure you build — an FRR pair on gateway hosts, an OPNsense cluster, your own ToR — and you budget the routers before you budget the migration.

That constraint picks your advertisement path for you. The realistic Hetzner design keeps everything else about the current architecture and swaps only the announcer: same vSwitch VLAN, same routed subnet, LB-IPAM allocating from it, Cilium L2 announcements answering ARP where MetalLB speakers used to. BGP enters the picture only with your own routers, at which point you get what BGP is actually for — ECMP across nodes instead of one L2 leader per IP, and propagation beyond the broadcast domain. Do not adopt BGP for its aura; adopt it when you have peers and a reason.

The failover math makes the tiers concrete:

Two readings follow. First, L2-to-L2 migration changes who answers ARP, not failover speed — paging volume won't change because of the swap alone. Second, Robot failover is not a failover mechanism for tenant services at all; at 90-plus seconds and whole-subnet granularity it is a disaster-recovery lever, roughly two orders of magnitude slower than an ARP move.

Migration sketch: phased, reversible, service by service​

Assume one workload cluster, Cilium already the CNI, MetalLB currently announcing. The fleet version of this plan is just this plan with an order: lab cluster first, then one canary tenant cluster, then the rest, never starting the next cluster until the previous one's MetalLB is drained and deleted. Within a cluster, five phases:

Phase 0 — pin the version and enable the flags. Run something current — 1.18.2 or later carries the fix that stops LB-IPAM reallocating service IPs across operator restarts, the exact bug class that turns a routine rollout into a renumbering event — with bgpControlPlane.enabled and/or l2announcements.enabled set as your design requires. Verify with cilium status and the connectivity suite before touching any service. No pool object means LB-IPAM stays dormant, so this phase changes nothing observable.

Phase 1 — shadow pool, no traffic. Create a CiliumLoadBalancerIPPool covering addresses MetalLB is not handing out (a carved slice of the routed subnet, or a separate range), with a serviceSelector that matches nothing yet. This is also the moment to internalize the sharpest documented edge: updating a pool can reassign service IPs (upstream issue), so size pools once, deliberately, and treat later edits as renumbering events with a maintenance window, not casual kubectl applies.

Phase 2 — one canary service. Loosen the pool selector (or annotate one non-critical service explicitly) so exactly one LoadBalancer service gets a Cilium-allocated IP, announced via the L2 policy. Move a synthetic monitor or an internal tool first. Watch ARP behavior on the vSwitch VLAN and confirm datapath handling if you run kube-proxy replacement; MetalLB keeps announcing everything else, so either side can still roll back.

Phase 3 — drain MetalLB. Service by service, move annotations from metallb.universe.tf/* to lbipam.cilium.io/*, letting each service take a Cilium IP and confirming reachability before the next. Tenant-facing services go last, in announced windows, since each move is a brief re-convergence; when the last one migrates, delete MetalLB's pools and advertisements, then MetalLB itself — only now does the speaker DaemonSet leave every node.

Phase 4 (optional) — BGP to your own routers. If and when you operate routers, add CiliumBGPClusterConfig peering plus CiliumBGPAdvertisement for the LB pool, confirm sessions establish from the agents, and retire the L2 policy for the BGP-covered ranges. One published on-prem recipe for this shape — native routing, kube-proxy replacement, eBGP to an FRR router — dates from September 2025 against Cilium 1.17.4 (walkthrough); the CRD names have since settled on the v2 API, so translate accordingly.

Carry two cautions through every phase. First, the v1-to-v2 BGP API migration bites anyone following a pre-2025 guide: CiliumBGPPeeringPolicy manifests no longer apply on current releases, and mixing v1 and v2 resources was never supported. Second, 2026-vintage sharp edges still surface where allocation meets advertisement — one bare-metal distribution's tracker shows an LB-IPAM reassignment flap breaking BGP service advertisement (issue) — which is exactly why the canary phase exists and why pool sizing is a design decision, not a draft.

Honest ledger: what you gain, what you give up​

Gains, concretely: one fewer DaemonSet on every node of every cluster, and — since 0.16 — immunity to MetalLB's FRR-K8s default pulling a second routing stack into L2-only clusters. Your LB IPs live in the same eBPF datapath as the rest of Cilium, which means service handling without kube-proxy hops (if you run kube-proxy replacement) and Hubble visibility into the same flows your policies already see. Configuration collapses from two CRD families plus Helm values into one, and upgrades become one rollout instead of two. The Rafay team's 2025 assessment agrees: if Cilium is already your CNI on bare metal, activating LB-IPAM is the logical next step rather than a new dependency (analysis).

Costs, with equal concreteness: MetalLB is CNI-agnostic and Cilium LB-IPAM is not — today the swap is portable in one direction only, and a future CNI change reopens the whole question. Cilium's L2 announcements are Beta where MetalLB's L2 speaker is battle-hardened; the failure modes you have never seen are, by definition, the ones waiting. BGP semantics — timers, communities, graceful restart, BFD sessions — move from a dedicated speaker you could reason about in isolation into your CNI's control plane, where a misconfiguration shares fate with pod networking. And none of this buys ECMP or routed propagation on Hetzner until you build the routers; the L2-for-L2 swap is a simplification play, not a capability upgrade.

Who should migrate​

Migrate when three things are true: Cilium is already your fleet-wide CNI, your MetalLB usage is L2 announcements over the vSwitch with no BGP anywhere, and you can afford a service-by-service canary per cluster. Stay when any of them fails — multi-CNI fleets, real BGP peering you depend on, or tenant SLAs that cannot absorb a Beta announcer's learning curve — and if you stay on L2-only MetalLB, explicitly disable the FRR-K8s subchart so 0.16's defaults don't tax clusters that never speak BGP. Either way, the exercise is worth running as a design review: inventory your pools, price the FRR-K8s overhead you are now opting out of, and know which cluster would go first. The load balancer on a fleet with no cloud LB API is not a product you buy; it is a behavior your own components perform, and naming which component performs it is the entire decision.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.