Hetzner Killed the datacenter Field: Is Your CAPH Fleet's CCM Still Reading It?

If your Cluster API Provider Hetzner (CAPH) fleet has been running quietly for the last few months, there's a one-line check worth running right now: kubectl get deployment -n kube-system hcloud-cloud-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}'. If that tag is v1.30.0 or older, you're carrying a bug that's about to stop being theoretical.

Hetzner Cloud deprecated the datacenter property on Servers and Primary IPs back on December 16, 2025, replacing it with a top-level location field. The deprecated field is scheduled for removal from the API "after 1 July 2026" — a date that, as of this post, has already passed. That's not a hard cutover at midnight; Hetzner's deprecation language leaves the exact removal moment unannounced, which is worse for operators, not better. The field can disappear from API responses at any time now, with no further warning.

What actually changed in the API​

Hetzner's datacenter object was always a bit of a compound field: it bundled a datacenter name like fsn1-dc14 with a nested location object describing the city-level region (fsn1, nbg1, hel1, and so on). The December 2025 change split that apart — location is now a standalone top-level attribute on both Server and PrimaryIP resources, and datacenter is marked deprecated in its place.

This isn't an isolated cleanup. The same wave deprecated the GET /v1/datacenters and GET /v1/datacenters/{id} listing endpoints, which are scheduled for removal after October 1, 2026 — a second, later deadline that's easy to lose track of once the first one is handled. Any tooling that lists datacenters directly (custom inventory scripts, Terraform data sources, homegrown capacity dashboards) needs its own separate check.

For a Cluster-API-provider-Hetzner fleet, the piece that actually touches datacenter on every reconcile isn't your Terraform state or your CAPH controller — it's hcloud-cloud-controller-manager (hcloud-ccm), the in-cluster component that turns Hetzner server metadata into Kubernetes node labels.

The exact failure: a nil pointer panic, not a warning​

hcloud-ccm uses the datacenter field to populate the standard Kubernetes topology labels on every node it manages:

• topology.kubernetes.io/region ← the location (fsn1)
• topology.kubernetes.io/zone ← the datacenter (fsn1-dc14)

Versions of hcloud-ccm at v1.30.0 and earlier read server.Datacenter directly to build the zone label. Once the field stops coming back in the API response, that code path dereferences a pointer that's now nil. The maintainers documented the resulting crash explicitly in the v1.30.1 release notes:

"Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference"

That's not a degraded-mode warning logged to kubectl logs that someone might notice during a routine check — it's the controller process crashing. In cloud-controller-manager's architecture, the same component that sets zone/region labels is also responsible for clearing the node.cloudprovider.kubernetes.io/uninitialized taint that every CAPI-provisioned node starts with. A crash-looping ccm doesn't just mean stale labels on existing nodes; it means any new node joining the cluster after the field disappears never finishes cloud-provider initialization — it sits tainted, unschedulable, invisible to your autoscaler's success metrics, while kubectl get nodes shows it as Ready and everything looks fine until pods refuse to land on it.

Existing, already-initialized nodes keep their current zone/region labels and keep working — which is exactly why this bug is so easy to miss in a fleet that isn't actively scaling. It surfaces the first time CAPH's MachineHealthCheck replaces a node, or your autoscaler adds capacity, days or weeks after the API change actually took effect.

Why CAPH fleets specifically carry this risk​

On a managed Kubernetes offering, the platform vendor upgrades the cloud-controller-manager for you, on their schedule, without you noticing. On a CAPH fleet, the ccm version is something you pinned — usually in one of these three places:

1. A HelmChartProxy or ClusterResourceSet referencing a specific hcloud-cloud-controller-manager Helm chart version
2. A static manifest checked into the same repo as your Cluster/HetznerCluster CRs, applied once when the cluster was bootstrapped
3. A machine image or cloud-init template that bakes in a specific manifest URL, set up months ago and never revisited

None of those three update themselves. The version that was current when you stood up the cluster is the version you're still running — unless something forced a Renovate/Dependabot bump or someone manually bumped it during an unrelated upgrade.

Check your fleet now:

# what's actually deployed
kubectl get deployment -n kube-system hcloud-cloud-controller-manager \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
 
# where the version is pinned in your repo (adjust path to your CAPH setup)
grep -rn "hcloud-cloud-controller-manager" --include="*.yaml" --include="*.yml" .

If the deployed tag is v1.30.0 or earlier, upgrade to v1.30.1 or newer before your next node replacement or scale-up event, not after.

What v1.30.1 actually does — and why the fix matters for existing nodes too​

The fix isn't just "stop crashing." The hcloud-ccm maintainers had a second constraint: fleets already have NodeAffinity rules, topologySpreadConstraints, and PodDisruptionBudgets written against existing zone label values like fsn1-dc14. Simply switching the zone label to the new bare location value (fsn1) the day the field disappeared would silently break every one of those scheduling rules — a second, quieter breakage layered on top of the first.

Instead, v1.30.1 ships a static mapping table from location names to their legacy datacenter names, so the topology.kubernetes.io/zone label keeps producing the same fsn1-dc14-style value for datacenters that existed before the change — sourced from the mapping table, not a live API field. New Hetzner locations added after the mapping table was built will report the bare location name with no -dcNN suffix, which is worth knowing if you're about to expand into a newly launched Hetzner region.

Practically, this means the upgrade is close to invisible for existing infrastructure: your zone labels don't change, your scheduling rules don't need editing, and the only observable difference is that the controller no longer panics once Hetzner flips the switch on the underlying field.

If you're already stuck: recognizing and unwinding it​

If you're reading this after the fact — nodes added recently aren't picking up workloads — the symptoms to check for are specific enough to confirm the diagnosis in under five minutes:

# crash-looping ccm pod is the first tell
kubectl get pods -n kube-system -l app.kubernetes.io/name=hcloud-cloud-controller-manager
kubectl logs -n kube-system -l app.kubernetes.io/name=hcloud-cloud-controller-manager --previous | grep -i panic
 
# nodes stuck uninitialized never get scheduled onto
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints \
  | grep uninitialized

The fix is the same upgrade, applied after the fact rather than ahead of it: bump the hcloud-ccm image to v1.30.1+, let the deployment roll out, and the controller's next reconcile pass clears the uninitialized taint on every affected node without requiring a node replacement. Nodes that were provisioned and stuck are recovered in place — you don't need to cordon-and-drain your way through the fleet, which is the detail that turns this from an afternoon incident into a five-minute non-event, provided you catch it via the check above rather than via a ticket about pods stuck in Pending.

The maintenance bill self-hosting doesn't remove — it relocates​

This is a small item, deliberately. It's one Kubernetes patch version, on one component, driven by one upstream API deprecation. That's exactly the point: a managed PaaS absorbs this kind of change on your behalf as a matter of course. A self-hosted, Cluster-API-managed fleet moves the obligation from "read a vendor's release notes" to "read a cloud provider's API changelog, on a schedule, and cross-reference it against every component in your stack that touches that provider's SDK." Nobody pings you when your specific hcloud-ccm version becomes incompatible with a field Hetzner removed six months after you deployed it — it's on you to have found out.

The concrete practice worth adopting, whether or not this specific bug ever bites you:

• Track Hetzner's Cloud API changelog the same way you'd track a dependency's release notes — it's the one place deprecations like this and the October 1, 2026 GET /v1/datacenters removal get announced first.
• Put hcloud-cloud-controller-manager, the hcloud Terraform provider, and any custom tooling that calls the Hetzner API on Renovate/Dependabot, not on "someone remembers during the next unrelated upgrade."
• Treat the Terraform provider the same way — v1.58.0+ preserves the datacenter value in state and derives it from location on the API side, so an un-upgraded provider hits an analogous break.

None of this is a Cluster API or Hetzner-specific failure mode; it's the standard cost of owning the control plane instead of renting one. It's a cost worth paying deliberately, not one worth discovering the first time an autoscaler adds a node that never comes ready.

Bex.co runs Cluster-API-managed fleets — including on Hetzner — as the substrate for its open-source, AI-native PaaS: push a git repo, get a running HTTPS service on machines you own. Tracking exactly this class of upstream API churn is part of what we handle so a fleet operator doesn't have to catch it manually. Star the repo on GitHub or deploy your first app today.

Sources​

• Hetzner Cloud API Changelog
• hcloud-cloud-controller-manager releases
• hcloud-cloud-controller-manager repository
• Cluster API Provider Hetzner (CAPH)
• terraform-provider-hcloud releases