Dora NodaYou did the big Hetzner migration. When the datacenter property disappeared from Servers and Primary IPs on July 1, 2026, you moved everything to the top-level location field, bumped your SDKs, and closed the ticket. The problem is that the July removal was the headline, not the whole changelog. Two smaller breaks landed in its footnotes — and a third one, a hard 410 Gone, is still ahead of you on October 1.
Here is the full damage list, up front, so you can stop reading as soon as your fleet is covered:
| Leftover | Live since | What it silently breaks | One-line fix |
|---|---|---|---|
Unassigned Primary IPs return assignee_type: "unassigned" (was "server") | Aug 1, 2026 | Scripts, Terraform, and SDK checks that test assignee_type == "server" | Null-check assignee_id; pass assignee_type explicitly on create |
Prometheus drops __meta_hetzner_datacenter for hcloud targets | 3.14.0 (Aug 17, 2026) | Relabeling rules, recording rules, and alerts keyed on the old label | Remap to __meta_hetzner_hcloud_location* before you bump past 3.13 |
GET /v1/datacenters returns 410 Gone | After Oct 1, 2026 | Terraform hcloud_datacenter data sources, hcloud datacenters commands, status tooling | Move every read to location / server_types now |
If your fleet is Cluster API with the Hetzner provider (CAPH), the hcloud cloud-controller-manager, and kube-prometheus-stack — the standard self-hosted shape — all three rows deserve ten minutes of your time. The rest of this post is the what, the why, and a copy-paste runbook.
Break 1: unassigned Primary IPs stopped saying server
On April 27, 2026, Hetzner made assignee_type optional when creating a Primary IP, with a warning attached: IPs created without it would report assignee_type: "server" only until August 1, after which the value flips to "unassigned". On August 1 the flip happened, exactly as announced. The changelog table is admirably explicit:
assignee_type | assignee_id | |
|---|---|---|
| Previously | server | null |
| As of 01 Aug. 2026 | unassigned | null |
The stated reason is forward-looking: Hetzner wants to assign Primary IPs to resource types other than servers eventually, and a field that says server for an IP attached to nothing cannot survive that future. Fair enough. But every piece of automation written against the old behavior encodes the assumption "server with a null id means unassigned" — and that automation just started lying.
Concretely, three surfaces are exposed. First, shell and Python glue: anything that filters hcloud primary-ip list output or walks client.primary_ips.get_all() looking for assignee_type == "server" to find free IPs now matches nothing, or worse, matches only assigned ones depending on which side of the comparison you wrote. Second, Terraform: the provider changelog carries the same warning, and configurations that create Primary IPs without an assignee_type attribute now govern resources whose read-back value differs from anything the config implies. Third, your own operators and controllers: any Go code against hcloud-go that branches on PrimaryIP.AssigneeType needs the new enum member in its switch.
The fix is two lines of discipline. When reading, treat assignee_id == null as the definition of "unassigned" and stop branching on the type string. When creating, pass assignee_type explicitly — --assignee-type server in the CLI, the matching argument in every SDK — so no future default change moves your fleet again. The SDK changelogs (Go, Python, CLI, Terraform, Ansible) all say the same thing in the same words; when five maintainers agree, take the hint.
Note what this break is not: there is no evidence that the hcloud cloud-controller-manager mis-reconciles IPs over this change, and the CCM's own changelog for the period is about the datacenter-to-location mapping, not Primary IP types. Audit your assumptions rather than assuming a CCM bug — check the CCM and hcloud-go versions you run, then grep your own code, which is where the == "server" checks actually live.
Break 2: Prometheus 3.14 deleted __meta_hetzner_datacenter
The datacenter removal had a downstream consumer nobody owns: your monitoring. Prometheus's Hetzner service discovery derived its __meta_hetzner_datacenter label from the same API property Hetzner deleted, so upstream had to follow. The timeline ran in two steps. Prometheus 3.11.0 (April 2, 2026, PR #17850) deprecated the label for hcloud targets with a "will stop working after 1 July 2026" notice. Then 3.14.0 (August 17, 2026, PR #19269) dropped it outright: "[CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API."
The failure mode is silence. A relabeling rule with source_labels: [__meta_hetzner_datacenter] does not error after the upgrade — the label is simply absent, so keep/drop/replace actions quietly do the wrong thing. Recording rules that aggregate by datacenter produce empty series. Alerts that group by it stop grouping, which pages whoever is on call with either nothing or everything, depending on the rule.
And the deprecation covered more than one label; the full remap from #17850 is:
| Before (deprecated, then removed) | After |
|---|---|
__meta_hetzner_datacenter (hcloud) | gone — use __meta_hetzner_hcloud_location |
__meta_hetzner_hcloud_datacenter_location | __meta_hetzner_hcloud_location |
__meta_hetzner_hcloud_datacenter_location_network_zone | __meta_hetzner_hcloud_location_network_zone |
Two exceptions keep this from being a blind find-and-replace. The robot role (dedicated servers, not cloud) keeps working under the renamed __meta_hetzner_robot_datacenter label — only hcloud targets lost the short name. And if you run VictoriaMetrics' vmagent instead of Prometheus, the same upstream removal hits you through a different path: issue #10909 tracks vmagent silently dropping the __meta_hetzner_hcloud_datacenter_location* meta labels because the JSON path they were read from no longer exists. Same grep, different binary — check both if you run both.
The practical rule: do not bump kube-prometheus-stack (or raw Prometheus) past 3.13 until the remap is applied and promtool check rules passes. The label work is a fifteen-minute config edit; the outage from a silently wrong keep rule is a 3 a.m. education in PromQL empty-set semantics.
The deadline still ahead: /v1/datacenters goes 410 on October 1
The July removal took datacenter out of the Server and Primary IP objects, but the standalone endpoints survived — until now. On June 2, 2026, Hetzner deprecated GET /v1/datacenters and GET /v1/datacenters/{id}, with removal after October 1, 2026, after which "requests to these endpoints will return HTTP 410 Gone." Unlike the label drop, this one fails loudly: anything still polling those endpoints starts getting 410s on day one.
The blast radius is wider than it looks because the endpoints were convenient. The Terraform provider's hcloud_datacenter and hcloud_datacenters data sources are deprecated on the same schedule. The hcloud datacenters list and hcloud datacenters describe CLI commands are deprecated. Ansible's datacenter_info module is deprecated. And bespoke tooling breaks too: hetzner-radar, the community availability tracker, had to move its cloud-status reads to the supported server_types endpoint precisely because its datacenter reads were about to 410.
For a CAPH fleet, the highest-value check is your provisioning path: Cluster API manifests, Terraform modules, and any preflight scripts that resolve a datacenter name to validate capacity or placement. The documented replacement mapping points every read at location (for geography) or server_types (for availability). You still have days, not months — this is the one row in the table where the fix window is still open, so take it first if you only have one maintenance slot.
The pre-upgrade runbook: six greps before your next bump
Everything above compresses into a checklist you can run against your fleet repos in order. Do it before your next CCM or kube-prometheus-stack upgrade, not after.
1. Find Prometheus rules keyed on the dead labels. Search every repo that holds Prometheus config, Helm values, or mixin-style rule files:
grep -rn "__meta_hetzner_datacenter\|__meta_hetzner_hcloud_datacenter" \
--include="*.yml" --include="*.yaml" --include="*.libsonnet" .Remap every hit per the table in Break 2. Remember the robot exception: __meta_hetzner_robot_datacenter is the correct new name there, so don't "fix" it back.
2. Find code branching on assignee_type. Cover Terraform, scripts, and controllers:
grep -rn "assignee_type\|assigneeType\|AssigneeType" \
--include="*.tf" --include="*.py" --include="*.go" --include="*.sh" .Reads should key on assignee_id == null; creates should pass the type explicitly.
3. Find reads of the dying endpoints. Catch Terraform data sources, CLI calls, and direct API use:
grep -rn "hcloud_datacenter\|datacenter_info\|/v1/datacenters\|datacenters describe\|datacenters list" .Every hit must move to location or server_types before October 1.
4. Find lingering SDK field access. The SDKs removed Server.Datacenter / PrimaryIP.Datacenter in favor of Location, calling it non-breaking "since the property was already removed from the Hetzner Cloud API" — which is true for the API and cold comfort for your build:
grep -rn "\.Datacenter\|server\.datacenter\|primary_ip\.datacenter" \
--include="*.go" --include="*.py" .5. Validate and version-check. After editing rules, run promtool check rules on every touched file. Then confirm what you actually run: prometheus --version (or your kube-prometheus-stack chart's appVersion — anything at 3.14+ already dropped the label), hcloud version, your CCM image tag, and the hcloud-go version in go.mod. Kubermatic's machine-controller needed its own tracking issue for this migration; every consumer of the API needs one, including yours.
6. Fix in this order. October-410 reads first (hard deadline, loud failure), Prometheus relabeling second (gate your 3.14 bump on it), assignee_type assumptions third (silent wrong answers, no version gate — they broke on August 1 regardless of what you run).
Why this keeps happening (and the pinning discipline that ends it)
None of these breaks was a surprise. Hetzner announced the datacenter phase-out in December 2025, the assignee_type flip in April 2026, and the endpoint deprecation in June 2026 — each with a date, each kept. Prometheus deprecated for a full minor-release cycle before removing. The pattern is not vendor carelessness; it is the normal churn of an API growing up (datacenters folding into locations, IPs outgrowing servers), landing on fleets that pinned the big migration and skimmed the footnotes.
Two habits end the cycle. First, subscribe to the Hetzner changelog RSS feed and treat every entry tagged with your API surfaces as a ticket, not a newsletter — the April assignee_type notice was a three-month warning that most fleets spent ignoring. Second, make SDK bumps a scheduled chore with a grep attached: every hcloud-go, CCM, Prometheus, or provider upgrade gets the six greps above run against the fleet repos before rollout. The hcloud CCM team modeled the right behavior on the datacenter removal itself — they statically compute the old datacenter value from location to keep topology.kubernetes.io/zone labels stable, buying every downstream consumer a compatibility shim. Compat shims buy time, though, not exemption; the shim covers the CCM's labels, not your alert rules.
The deeper lesson generalizes past Hetzner: on a self-hosted fleet, your cloud API is a dependency with a changelog, and "we own the machines" never meant "the API stands still." Pin it like any other dependency — subscribe, grep, bump on purpose — and the next footnote stays a footnote.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
