Hetzner Deleted the Datacenter Field: Why Your Prometheus Labels Went Quietly Empty (and the 30-Second Check)

On July 1, 2026, Hetzner removed the datacenter property from its Cloud Servers and Primary IPs API responses — and nobody's pager went off. That silence was the incident. Every target kept scraping, every dashboard kept rendering, but every Prometheus label derived from servers[].datacenter.location quietly became the empty string on fleets running an unpatched scraper.

This is the story of a cloud API rename as a monitoring incident in slow motion: a deprecation announced in December 2025, fixes shipped by both major scrapers months before the cutoff, and still a long tail of fleets that sailed past July 1 with location-aware alerts, dashboards, and routing silently matching nothing. If you run hetzner_sd_configs anywhere — Prometheus or vmagent — this post gives you a 30-second check, the label mapping, and the full audit checklist.

The 30-second check: are your location labels empty?​

One caveat first, because it shapes the check: __meta_* discovery labels never reach storage. They exist only during relabeling, so you cannot query them in PromQL. What you can query is whatever label your relabel_configs derived from them — typically something like location. Run this against your Hetzner scrape job:

count by (location) (up{job="hetzner"})

If you see a row with an empty location next to your healthy fsn1, nbg1, hel1 rows, you are affected: targets are UP, but their location metadata evaporated. (Adjust job and the label name to yours; or open the Prometheus UI's Targets page and look for blank location-derived labels.) No empty bucket? You are either patched or never keyed on these labels — skim the checklist at the end anyway, because October 1 brings the next cutoff.

The fix is a label-name migration, not a value remapping. Hetzner's own changelog notes the new top-level location field carries the same data that used to sit under datacenter.location, so fsn1 stays fsn1 — only the meta label path changes:

Patched scrapers emit the new canonical labels: Prometheus 3.11.0 and later, and vmagent/vmsingle v1.143.0 and later (backported to the v1.136.x and v1.122.x LTS lines). Details and the old-label compatibility story are in the fix section below.

What Hetzner actually changed​

The whole saga fits in one timeline. Four of these rows are Hetzner API changes; two are the scraper fixes that defined your upgrade window:

Two things stand out. First, the upgrade window was generous: nearly three months from the Prometheus fix, nearly two from the vmagent fix, to the removal date. This was never a surprise cut — it was a slow-motion incident for anyone not watching the changelog. Second, the rename keeps echoing: the August assignee_type change alters a different field's values (not just its path), and the October endpoint removal will hard-fail anything still polling the old datacenters endpoints instead of merely returning empty labels.

The same rename rippled through the whole Hetzner client ecosystem — hcloud-go, hcloud-python, the hcloud CLI, terraform-provider-hcloud, and the Ansible collection all shipped datacenter-to-location migrations. If you template monitoring configs from Terraform or scripts against the API, those need the same audit.

Why nothing alerted you: the anatomy of a silent label loss​

Every layer in the chain did exactly what it was designed to do, and the failure slipped through all of them. Walk through it:

1. The API stayed green. Hetzner returned 200 OK with valid JSON — the datacenter key was simply absent. No error code, no failed request, just a key that stopped being there.
2. Service discovery didn't fail. hetzner_sd reads servers[].datacenter.location.*. With the key gone, the Go structs unmarshalled to zero values: empty strings. Discovery still returned every server, so no prometheus_sd_failed_configs, no failed-scrape alert.
3. Relabeling succeeded — with empty values. A rule like this kept working, in the sense that it ran without error:

relabel_configs:
  - source_labels: [__meta_hetzner_hcloud_datacenter_location]
    target_label: location

Every target got location="". Targets page all green, up == 1 everywhere.

4. Downstream matching silently stopped. This is where the damage lands, and it lands far from the cause:
   • An alert like up{location="fsn1"} == 0 never fires — not because FSN1 is healthy, but because no series carries location="fsn1" anymore. Absence of fire reads as health.
   • A Grafana variable built on label_values(up, location) gains a blank option; dashboards filtered by location show empty panels.
   • Any automation keyed on the derived label — per-location alert routing, capacity dashboards feeding an autoscaler's headroom math — operates on an empty set.

The cruelest part: this failure mode is invisible to the monitoring watching itself. up is 1, scrape durations are normal, rule evaluation succeeds. The only signal is a value-shaped hole — series that should exist but don't — and most alert setups don't assert on the presence of their own label dimensions.

The August assignee_type change is the same genre of failure with a twist: instead of a missing path, a changed value. Previously an unassigned Primary IP reported assignee_type: "server" with assignee_id: null; now it reports assignee_type: "unassigned". Any filter, cleanup script, or cost report that selected unattached IPs via assignee_id == null is fine — but anything matching assignee_type == "server" to mean "attached" now silently includes every unassigned Primary IP in the account. Check both halves of that predicate wherever you query Primary IPs.

The fix, per scraper​

Prometheus: upgrade to v3.11.0 or later. PR #17850 added __meta_hetzner_hcloud_location and __meta_hetzner_hcloud_location_network_zone as the canonical labels, and renamed the robot-role label to __meta_hetzner_robot_datacenter (old name kept for compatibility). The old hcloud datacenter labels are deprecated.

vmagent and vmsingle: upgrade to v1.143.0 or later (or LTS v1.136.9 / v1.122.22). The fix for issue #10909, in PR #10910, mirrors Prometheus: it reads the new top-level location API field, emits the two new canonical labels, and keeps populating the deprecated __meta_hetzner_hcloud_datacenter_location* labels from the new field so existing relabel rules keep working past July 1.

That last point deserves emphasis because it defines your migration order:

1. Upgrade first. On a patched scraper, even your old relabel rules produce correct labels again, because the deprecated labels are backfilled from the new API field. This alone restores your alerts and dashboards.
2. Then migrate rules to the canonical labels. The deprecated names work today, but they are deprecated — the next cleanup release can drop them, and you will have the same silent-empty incident all over again. Update source_labels to __meta_hetzner_hcloud_location and __meta_hetzner_hcloud_location_network_zone, plus the robot-role rename if you scrape dedicated servers.

If you run Prometheus Operator, kube-prometheus-stack, Grafana Alloy, or a vendored vmagent, the fix version is whatever ships the underlying discovery code — check the bundled Prometheus/VictoriaMetrics version, not just the wrapper's. And if you pinned an old scraper image "because monitoring config never changes," this incident is the argument for putting your SD components on a normal upgrade cadence.

The full audit checklist​

Work top to bottom; each item catches a different blast radius of the same rename:

1. Run the 30-second check from the top of this post against every Hetzner scrape job (all Prometheus servers, all vmagents, all tenants). An empty location bucket anywhere means that scraper is unpatched.
2. Grep every relabel config for datacenter: grep -rn "hetzner_hcloud_datacenter\|hetzner_datacenter" /etc/prometheus/ /etc/vmagent/. Migrate matches to the canonical labels from the mapping table.
3. Audit alert selectors and recording rules that filter on location-derived labels. Any location="fsn1"-style matcher has been matching nothing since July 1 on unpatched scrapers — after upgrading, confirm those alerts can fire again (and consider whether any incident in the gap was missed because of it).
4. Check Grafana template variables and dashboard filters built on label_values(..., location) or the network-zone label. Remove stale blank options and verify location-scoped dashboards populate.
5. Verify automation keyed on location labels: per-location alert routing (Alertmanager routes, PagerDuty urgency rules), capacity/headroom dashboards, and anything an autoscaler consumes. Silent-empty labels here mean decisions made on empty sets.
6. Audit Primary IP assignee_type filters for the August 1 value change. Any logic treating assignee_type == "server" as "attached" now also matches unassigned IPs; prefer testing assignee_id == null for "unassigned," and update IP-cleanup automation and cost reports accordingly.
7. Search for direct API consumers of the old paths: scripts or exporters reading servers[].datacenter, Terraform datacenter attributes on servers and primary IPs (long since deprecated in the provider — move to location), and anything polling GET /v1/datacenters, which starts returning 410 Gone after October 1.
8. Re-run the Section 1 query after upgrading and confirm the empty bucket is gone. Then add a standing guard: an alert that fires when count(up{job="hetzner", location=""}) > 0, so the next silent-empty regression pages you instead of hiding.
9. Calendar the remaining cutoffs: October 1, 2026 for the datacenters endpoints (410 Gone), and note Hetzner's September announcement that the deprecated field on Images/Server Types/Load Balancer Types goes away November 2, 2026 in favor of deprecation — the rename wave is not over.

Treat cloud changelogs as monitoring inputs​

The general lesson is short: your cloud provider's changelog is an input to your monitoring system, not background reading. A field rename in an API you never call directly still reaches you through service discovery, and it arrives as the worst kind of failure — valid responses, green targets, hollow labels.

Three habits make the next one boring instead of silent:

• Subscribe to the machine-readable changelog. Hetzner publishes an RSS feed for its Cloud changelog; pipe it into the channel where your team actually reads operational notices, and treat deprecation entries as tickets, not FYIs.
• Put service discovery on an upgrade cadence. Prometheus and vmagent both shipped this fix months before the cutoff. The fleets that got bitten were not missing information — they were running SD code from before the fix existed. A quarterly bump of your scraper, with a staging diff of discovered labels, closes this class of gap.
• Assert on your own label dimensions. The standing guard from checklist item 8 — alert when a derived label you depend on goes empty — costs one rule and converts every future silent-empty regression from "wrong dashboards for weeks" into "a page within minutes."

Cloud APIs will keep evolving under your feet; locations replaced datacenters this year, and something else gets renamed next year. The fleets that notice are the ones that monitor their monitoring.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. If you run your PaaS on Hetzner, the fleet it provisions is exactly the fleet this checklist audits. Star the repo on GitHub or deploy your first app today.