Dora NodaIn August 2026, a Telegram bot that manages Hetzner Cloud servers went blind overnight. Every server showed a white "unknown" flag, the plan-change screen listed no plans at all, and the prices it quoted came from the wrong city. Nothing in the bot had changed. What changed was one field in the Hetzner Cloud API: datacenter was gone, and location had taken its place.
The bot is hetzner-server-usage, a Python tool its author uses to watch traffic, change plans, and track overage across a personal Hetzner fleet. Its August repair commits are the most instructive post-mortem of Hetzner's rename to date — not because the bot is famous, but because it failed the way fleet automation fails: silently, in the read path, weeks after the provider's changelog announced the change. This post walks the full failure cascade, the money bugs hiding underneath it, and the runbook your own capacity and cost tooling needs before October 1, when the old datacenters endpoint starts answering 410 Gone.
What Hetzner actually changed
The rename was not a surprise. It was a nine-month deprecation with four dated changelog entries — which is exactly why it is worth studying: every team that broke had the schedule and broke anyway.
| Date | Changelog entry | Operator impact |
|---|---|---|
| 2025-12-16 | Phasing out datacenters | datacenter on Servers and Primary IPs deprecated; top-level location is the replacement |
| 2026-04-01 | Datacenter deprecations | Availability data moves from Datacenter.ServerTypes to ServerType.Locations |
| 2026-06-02 | Datacenters deprecated | The /datacenters endpoint itself deprecated; returns 410 Gone after October 1, 2026 |
| 2026-07-01 | Removing datacenters | datacenter property removed from Server and Primary IP responses (now null) |
The mapping itself is trivial: the location name is the datacenter name up to the dash, so fsn1-dc14 becomes fsn1 and nbg1-dc3 becomes nbg1. Hetzner's own tooling treated the follow-through as routine. The hcloud CLI and Terraform provider both shipped the removal in v1.67.0 with the note "we do not consider this a breaking change," since the property was already gone from the API. That sentence is the whole lesson in miniature: when the vendor does not consider it a breaking change, your automation is the only thing left that can break.
The failure cascade, symptom by symptom
The August 11 repair commit (d3cb0ff, "Fix location lookups, price reporting and monthly overage accounting") describes a cascade with a single root: everything that read datacenter.location.name now got an empty string, because datacenter comes back null. Four symptoms, one cause:
| # | Symptom | Root cause | Fix |
|---|---|---|---|
| 1 | Every server showed the white "unknown" flag | Flag lookup keyed on datacenter.location.name, now "" | Resolve location through one helper that reads either API shape; derive the flag from the location's own country code instead of a hardcoded table |
| 2 | Change Plan found no available plans at all | Plan list filtered on the datacenter's available_for_migration list, which no longer resolves | Key plan availability off the location; offer every plan and mark the ones Hetzner hides rather than filtering them invisible |
| 3 | Per-location prices fell back to the wrong entry | Server prices were read from prices[0] — always Falkenstein, whatever the server's real location | Match the prices[] entry to the server's actual location; never index [0] |
| 4 | Primary IP creation failed every time with 422 invalid_input | POST /primary_ips used to take a datacenter and now takes a location; the picker sent nbg1-dc3 where the request needs nbg1 | Send location on the write path and offer locations in the picker (fixed two days later in ab10bbd) |
Two details in that table deserve emphasis. First, symptom 4 is a write-path failure in a cascade that otherwise only corrupted reads: the rename broke display code silently and creation code loudly, and the loud one was found two days later than the silent ones. Audit your writes, not just your dashboards. Second, the hardcoded flag table in symptom 1 is the kind of code that works until a provider adds a region — deriving display from the location object's own country code means the next Hetzner region renders correctly on day one instead of going blank.
The money bugs hiding underneath
A cost tool that misreads locations does not just look wrong — it bills wrong. The same repair exposed three cost-accuracy bugs that had been overcharging or misattributing money independent of the rename:
1. Grandfathered prices are invisible to the API. Hetzner's API reports only the current list price, but a long-standing account keeps the price it signed up at — and nothing in the API exposes that number. Any cost tool built purely on API prices silently re-prices old servers at today's rates. The fix was a Prices screen with per-server-type overrides, keeping the API price as the default. If your fleet cost tooling has no override mechanism, every provider price change rewrites your history.
2. Overage double-counted across month boundaries. The tracker bucketed overage by calendar month but snapshotted Hetzner's traffic counter, which resets on the billing cycle rather than on the 1st. A month that opened before the reset re-absorbed the previous month's traffic and billed it again when the reset landed — August 2026 held EUR 15.90 that was really July's. The fix accumulates overage as deltas against a persistent last-seen counter, so each month only carries what was actually sent in it, and migrates existing history to remove the double count. Calendar months are a reporting convenience; billing cycles are the accounting truth. Never bucket one by the other.
3. A hardcoded 20 TB traffic limit. Traffic allowances came from a constant, but US and Singapore locations include only 1 TB — and servers created mid-month get a prorated slice. The fix reads each server's own included_traffic. Any constant in cost code that the API already returns per resource is a wrong number waiting for a region to expose it.
None of these three required the rename to be wrong. The rename just forced someone to read the code.
Six fix patterns for your own fleet tooling
Generalized from the commits above, here is the checklist for any tooling that reads a cloud API for capacity or cost — a CAPH fleet's dashboards and autoscalers included:
| # | Pattern | Where to grep |
|---|---|---|
| 1 | Read both API shapes during migration through one helper, not ad-hoc at every call site | datacenter reads in fleet repos; every ["datacenter"], .datacenter, __meta_hetzner_datacenter reference |
| 2 | Derive display values (flags, region names) from the location object, never from a hardcoded table | hardcoded region/country maps; anything that renders a code it fetched |
| 3 | Match price entries to the resource's location; never take prices[0], and never fall back to a wrong price silently — fail loud or mark unknown | price-indexing code; any fallback chain that ends in a guess |
| 4 | Audit write paths, not just reads: creation calls take location now (nbg1, not nbg1-dc3) | server/IP/volume creation code, pickers, Terraform modules pinning old provider versions |
| 5 | Key plan-availability lookups off ServerType.Locations, not the datacenter's migration list | plan-change, rightsizing, and capacity-planning code |
| 6 | Bucket money by billing cycle, accumulate counters as deltas, and read limits per resource | overage/cost aggregation; hardcoded traffic, storage, or quota constants |
Pattern 3 is the one to internalize as a rule: a cost tool that cannot determine a price must say "unknown," not print a confident wrong number. prices[0] always being Falkenstein meant every non-Falkenstein server in that bot was mispriced for as long as the code ran — and nothing looked broken.
Pin the provider changelog in your upgrade runbook
The hetzner-server-usage cascade was not an isolated incident; July 2026 moved the entire Hetzner ecosystem at once. Every one of these shipped in the same few weeks:
| Project | Change | Shipped |
|---|---|---|
| hcloud CLI v1.67.0 | --datacenter flags removed in favor of --location | July 24, 2026 |
| terraform-provider-hcloud v1.67.0 | datacenter attributes removed from Server and Primary IP resources | July 2026 |
| Ansible hetzner.hcloud | datacenter removed from server and primary IP modules | July 28, 2026 |
| Prometheus | __meta_hetzner_datacenter discovery label dropped | July 2026 |
| hcloud-go v2.38.0 | Datacenter.ServerTypes deprecated for ServerType.Locations | 2026 |
That is the most aggressive provider deprecation cycle Hetzner has run in years, and it still has one stage left: after October 1, 2026 — days from now — requests to the /datacenters endpoints return 410 Gone, and the hcloud datacenter commands go with them. Four runbook items before then:
- Subscribe to the changelog as a monitoring input.
docs.hetzner.cloud/changelogannounced every stage above months ahead. A provider changelog belongs in the upgrade runbook with an owner and a review cadence, not in a bookmark nobody opens. - Grep every fleet repo for
datacenterbefore October 1. Reads, writes, labels, dashboards, Terraform, Ansible, and cost scripts. Anything still hitting/datacentersbreaks loudly in days. - Pin minimum client versions in CI. hcloud-go, hcloud CLI, Terraform provider, Ansible collection — assert floors that know about
locationso a stale pinned version cannot silently resurrect the old shape. - Treat "not a breaking change" as a breaking change. When the vendor's removal note says the property was already gone from the API, that sentence is addressed to you: your automation was reading a field that no longer exists, and the only question is whether it failed loud or wrong.
Providers will keep renaming things. The teams that survive it are the ones whose tooling reads the changelog before the API response changes — and whose cost code says "unknown" instead of guessing Falkenstein.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
