Your Three Replicas Are All in FSN1: TopologySpreadConstraints, Hetzner Failure Domains, and the Placement Policy a Self-Hosted PaaS Owes Its Tenants

You ask a tenant how many replicas they want, they say three, and you picture one in Falkenstein, one in Nuremberg, one in Helsinki. Then you run kubectl get pods -o wide and all three are in FSN1. Nothing is broken. The scheduler did exactly what you asked. That is the problem: with no spread constraint, Kubernetes bin-packs, and three replicas on three nodes in one location is a perfectly valid answer to a request you thought meant "keep me alive when a location dies."

Here is the verdict up front. TopologySpreadConstraints with maxSkew: 1 across three Hetzner locations does not give two replicas three homes — it gives 1/1/0, with one location always empty. The euro cost of spreading across locations is close to zero on Hetzner's private networks, so the real price is operational: cross-location dependencies on every request path and control-plane chatter that now crosses location boundaries. And the placement policy a self-hosted PaaS owes its tenants is this: soft spread by default so scheduling never blocks, strict spread only as an opt-in for workloads with at least as many replicas as locations, and never a copied-paste "3-zone HA" snippet whose minDomains can wedge a second replica in Pending on a single-location fleet.

The baseline: three replicas, all in FSN1​

Before any constraint math, look at what the default scheduler does with a plain 3-replica Deployment on a fleet whose nodes all live in FSN1, or even on a fleet with nodes in three locations. The default scheduler scores nodes on resource fit, image locality, and spreading pressure from existing pods of the same workload — but nothing requires it to touch every location. Packing three pods onto the emptiest nodes frequently means packing them onto nodes in one location, especially when the other locations' nodes are fuller, newer, or still pulling images.

So the title image is the unconstrained baseline, and it surprises people twice. First, on a single-location fleet, a zone-spread constraint is a silent no-op: skew is measured per domain against the global minimum across eligible domains, and with one eligible domain the math is self-referential — always satisfiable, never blocking, never spreading. Second, when you later add Nuremberg and Helsinki nodes, the constraint wakes up and starts steering new pods to the emptier locations, while the pods you already run stay exactly where they are. Spreading is a scheduling-time decision, not a rebalancing one. If you want existing pods to move, that is the Descheduler's job, not the scheduler's.

That last row is the one that bites platform teams. You add two locations, add the constraint, and the dashboard still shows FSN1 holding everything until the next rollout.

maxSkew math in five minutes​

The whole feature is one formula. For whenUnsatisfiable: DoNotSchedule, maxSkew is the maximum permitted difference between the matching-pod count in the pod's target domain and the global minimum across eligible domains, as the official docs define it. A pod may land in a domain only if placing it there keeps that difference within maxSkew. With ScheduleAnyway, the same number is just a soft preference: the scheduler favors placements that reduce skew but schedules regardless.

Two things operators routinely get wrong. First, DoNotSchedule is the default. A constraint written without whenUnsatisfiable is a hard block, not a hint. Second, the headline math from the TODO that motivated this post:

Two replicas across three locations at maxSkew: 1 leaves one location empty — 1/1/0 satisfies the constraint, because the fullest domain (1) minus the emptiest (0) is exactly 1. Walk it forward: the third replica takes the empty location (1/1/1), the fourth stacks onto any location (2/1/1, skew still 1). The honest reading of 1/1/0: you still survive any single location loss, since at most one replica dies with it. What you do not have is what the tenant pictured — presence in every location — plus you have less headroom than 1/1/1 for the next failure. Never promise per-location presence to a workload running fewer replicas than locations.

Now the trap that actually wedges rollouts, and it is doc-grounded, not folklore. minDomains sets a floor on how many eligible domains must exist; when fewer exist, the global minimum is treated as zero. Copy a "3-location HA" snippet with minDomains: 3 onto a fleet whose nodes are all in FSN1 and watch:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    minDomains: 3
    labelSelector:
      matchLabels:
        app: tenant-api

Replica one schedules (1 against a treated-as-zero minimum: skew 1, allowed). Replica two has nowhere to go — the only eligible domain would hold 2 against 0, skew 2, over the max — so it sits in Pending forever, on a fleet with idle capacity, because of a YAML block that read as "best practice." The fix is boring and that is the point: minDomains must describe locations you actually run, not locations you aspire to. Without minDomains (it defaults to 1), the same constraint on a single-location fleet degrades to the silent no-op from the previous section instead of a wedge — still not HA, but at least not Pending.

What Hetzner's failure domains actually are​

Hetzner Cloud has locations (fsn1 Falkenstein, nbg1 Nuremberg, hel1 Helsinki, plus Ashburn, Hillsboro, Singapore), grouped into network zones — eu-central covers the three European sites, and a private network spans locations within one zone. That grouping is the entire physical vocabulary your placement policy gets: there are no availability zones inside one location. Within a location, the anti-affinity primitive is the spread placement group, which guarantees members land on different physical hosts — location-scoped, spread-type only, capped at 10 servers per group.

Three facts drive every placement decision, all worth stating with their scope attached:

• Inter-location private traffic is free. East-west chatter between fsn1, nbg1, and hel1 over the private network costs nothing extra, so "keep tenants in one location to save bandwidth" is not a bandwidth argument.
• EU servers include 20 TB of traffic per month, with €1 per TB over. Scope that sentence the way Hetzner scopes it: EU locations get the generous quota; US and Singapore quotas are far smaller. A tenant JSON API will never dent 20 TB; a tenant serving video might, wherever it runs.
• Per-location capacity fluctuates. Any given location can run short of a given server type, which is why operators keep fallback locations in their provisioning config rather than marrying one site.

And the labels your constraints match against are not something you hand-maintain. The Hetzner Cloud Controller Manager's zones interface sets topology.kubernetes.io/region and topology.kubernetes.io/zone on each node from the server's failure domain, and on the Cluster API side a MachineDeployment pins its pool with failureDomain: fsn1 (or nbg1, hel1) — one pool per location, each pool's nodes arriving already labeled with the domain the scheduler reasons about. The chain from "I want a pool in Helsinki" to "the scheduler knows what Helsinki is" is two links: the failureDomain pin at provisioning time, the CCM labels at runtime. If either link is missing — unlabeled nodes, or pools floating across locations — your zone constraint is matching against fiction.

What cross-location spread really costs​

Now the cost question the TODO item demands, answered in two currencies. In euros: for typical tenant east-west traffic, about zero. Private inter-location traffic is free, the public-egress quota per EU server is 20 TB before a euro is spent, and spreading three app replicas across three locations does not multiply any bill anyone in this architecture pays. If your total was tight before, it is tight by the same amount after. Single-location bin-packing wins on euros only at the margins — fewer cross-AZ-style transfer fees exist here to avoid in the first place, because Hetzner never charged them on the private fabric.

The real price is operational, and it lands in three places:

1. Every cross-location hop is a failure-mode dependency. A pod in hel1 talking to a database primary in fsn1 does not just get slower responses; it gains a partition mode in which the app is up and its data is unreachable. Spread the stateless tier all you like, but the stateful tier's location now dictates the failure story of every replica, and "the DB lives in fsn1" quietly re-centralizes the HA you just distributed.
2. Control-plane chatter crosses boundaries too. Kubelets talk to the API server constantly; spread nodes across three locations and some of that traffic permanently rides inter-location links. Keep etcd quorum members close together — a quorum stretched across Falkenstein, Nuremberg, and Helsinki pays for every commit with the slowest leg and risks losing quorum to a single inter-location partition. Management cluster in one location, workload nodes where you please, is the boring layout that survives.
3. Partial failure gets weirder, not rarer. Three locations triple the set of "half the fleet can see half the fleet" states your on-call has to reason about. That is a fair trade for surviving a full location loss — it is a bad trade for a tenant whose two replicas would have been perfectly safe, and far more debuggable, on two hosts in one location with a spread placement group between them.

The recommendation matrix falls out directly:

The placement policy a PaaS owes its tenants​

Tenants assume HA they never asked for because the platform never told them what the default is. The policy below is the default a self-hosted PaaS should ship — as cluster-level scheduler defaults so tenants inherit it without writing YAML, with strictness available as an opt-in, never as a copied snippet:

# Platform default: spread where possible, never block scheduling.
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: tenant-api
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: tenant-api

Host spread is soft so bin-packing and scale-up never wedge on a small fleet — remember, DoNotSchedule is the default, so every constraint you write without whenUnsatisfiable is a promise that scheduling can break. Zone spread is soft for the same reason, hardening to DoNotSchedule only when the tenant opts in and runs at least as many replicas as locations. minDomains appears nowhere in the default, because the default must be safe on a single-location fleet, where it quietly no-ops instead of wedging replica two in Pending.

Three rules sit above the YAML, and they are the part worth putting in your docs:

1. Replicas below locations means an empty location. Say it in the tenant-facing docs next to the replica slider, with the 1/1/0 row, before anyone learns it from a 2 a.m. page.
2. Defaults protect scheduling; strictness is opt-in. A platform default that can leave pods Pending on a healthy fleet is not a default, it is a landmine with documentation.
3. Name the locations in the contract. "Spread across zones" on Hetzner means fsn1/nbg1/hel1 via labeled pools, not an abstract topology. If the tenant cannot name which location holds their primary, neither can your incident response.

The through-line of all four sections is one sentence: the scheduler only knows the domains your provisioning and labeling actually built, the skew math only promises what its formula says, and everything else — the third location, the failover, the HA — is a platform policy you write down, default sensibly, and tell tenants about. Your three replicas are all in FSN1 until your policy says otherwise. Make it say otherwise on purpose.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.