Hetzner Killed the `datacenter` Field on Schedule. CAPH's Provider Code Survived — Its Vendored SDK Didn't Keep Up

Nine days ago, Hetzner's Cloud API quietly stopped returning the datacenter property on every Server and Primary IP it hands back. No maintenance window, no status-page incident, no crash report — just a field that used to be there and now isn't. If you run a Kubernetes fleet on Hetzner through Cluster API Provider Hetzner (CAPH), that's the kind of change you'd expect to find out about the hard way, months after the deprecation notice shipped and forgotten.

So we went and checked. We cloned CAPH's main branch, grepped every line of its controller and provider code for the word "datacenter," and read the actual vendored client library it ships. The answer surprised us: CAPH's own reconciliation logic never touched the field Hetzner just removed. It was never at risk of the failure mode everyone assumes — a nil-pointer panic three months after the fact. The real problem is narrower, more mundane, and still unresolved: CAPH is running a dependency snapshot from three weeks before Hetzner even announced the change, and the fix has been sitting in an open, CI-failing pull request since June 3.

Here's the audit, with file paths and PR numbers, not vibes.

The Deprecation Timeline, in Full​

Hetzner phased this out in three dated steps, all documented in its public API changelog:

Two separate deadlines, two separate blast radii: the July 1 change affects anyone reading a Server or Primary IP object; the October 1 change affects anyone calling the /datacenters listing endpoints directly. A fleet can be exposed to one, both, or neither — which is exactly why a grep-level audit is more useful here than a policy summary.

What Actually Changes on the Wire (and Why "Panic" Is the Wrong Word)​

Before the change, a Server response nested location data three levels deep:

{
  "id": 42,
  "datacenter": {
    "name": "fsn1-dc8",
    "location": { "name": "fsn1", "city": "Falkenstein", "network_zone": "eu-central" }
  }
}

After July 1, that whole datacenter object is gone; only a flattened, top-level location remains. The official Go client, hcloud-go, already reflects this — Server.Datacenter now carries an explicit deprecation comment:

// Deprecated: [Server.Datacenter] is deprecated and will be removed after 1 July 2026.
// Use [Server.Location] instead.
Datacenter *Datacenter

The TODO framing we started from expected this to surface as "a runtime nil-field panic months after the deprecation notice shipped." That's not quite how Go's encoding/json behaves, and the distinction matters operationally. Missing JSON keys don't error — they decode into the zero value. Worse, hcloud-go's own schema converter builds the Datacenter struct unconditionally, whether or not the API actually sent one:

func (c *converterImpl) DatacenterFromSchema(source schema.Datacenter) *Datacenter {
    var hcloudDatacenter Datacenter
    hcloudDatacenter.ID = source.ID
    hcloudDatacenter.Name = source.Name
    hcloudDatacenter.Location = c.LocationFromSchema(source.Location)
    return &hcloudDatacenter
}

So the actual failure mode isn't a crash. It's a non-nil pointer to an empty struct — Datacenter{ID: 0, Name: "", Location: Location{}} — silently returned to any code that still reads server.Datacenter.Name or .Datacenter.Location.Name after the field disappears from the wire. No stack trace, no error log — just an empty string flowing into whatever depended on it (a node label, a topology annotation, a log line). That's a harder bug class to catch than a panic, which is exactly why it's worth auditing rather than assuming.

The Audit: Does CAPH Read datacenter.location?​

We cloned github.com/syself/cluster-api-provider-hetzner at its current main and ran the check that actually answers the question, across every non-vendored, non-test Go file:

$ grep -rniI "datacenter" --include="*.go" . | grep -v vendor | grep -v _test.go
(no output)

Zero matches. CAPH's controllers, pkg/services/hcloud/, and api/ packages never reference the word "datacenter" at all — not in a field access, not in a status struct, not in a log message. The only hit anywhere in the tree is a canned JSON fixture in a test file (pkg/services/hcloud/server/server_suite_test.go), a copy-pasted example API response used to verify JSON unmarshalling completeness, not application logic.

That's because CAPH was already writing to the newer, flatter shape on the request side, long before this deprecation existed. Server creation targets hcloud.Location directly:

// pkg/services/hcloud/server/server.go:1725
Location: &hcloud.Location{
    Name: hm.Spec.Location,
},

And load balancer creation does the same:

// pkg/services/hcloud/loadbalancer/loadbalancer.go:431
Location: &hcloud.Location{Name: string(hc.Spec.ControlPlaneLoadBalancer.Region)},

Neither code path ever calls the deprecated Client.Datacenter.List/.Get methods either — so CAPH doesn't touch the endpoints being removed on October 1 either. Verdict: CAPH's own reconciliation loop was never exposed to either deadline. If your fleet only runs vanilla, upstream CAPH, this specific deprecation doesn't touch your control plane.

The Real Gap: A Client Library Stuck in November 2025​

Here's where the "already safe" story gets a caveat. CAPH's go.mod pins:

github.com/hetznercloud/hcloud-go/v2 v2.32.0

That release shipped November 25, 2025 — three weeks before Hetzner even announced the deprecation. hcloud-go's own changelog shows exactly which release picked up the fix:

We checked whether this was just a main-branch lag — it isn't. CAPH's latest tagged release, v1.1.7 (published June 17, 2026, two weeks before the deadline), still pins v2.32.0. Compare that to the two other Hetzner-ecosystem projects a typical CAPH fleet runs alongside: hcloud-cloud-controller-manager and terraform-provider-hcloud are both already on hcloud-go v2.44.0. CAPH is the outlier, twelve releases behind its own sibling tooling.

To be precise about what that staleness does and doesn't buy you: since CAPH's controllers never read Server.Datacenter (see the audit above), an old hcloud-go pin doesn't break the reconciliation loop today. What it does mean is that CAPH is running a client library snapshot that predates the vendor's own migration path — brittle in the sense that any custom patch, fork, or webhook that does read server.Datacenter.Location.Name (a homegrown node-labeling controller is the obvious candidate) inherits the silent-empty-struct behavior described above, with no upstream fix available to pull in.

The Fix Is Already Written — and Stuck in CI Since June​

This part isn't hypothetical. There's a real, currently open pull request: #2069, a Renovate-authored dependency bump titled "Update Update Golang Dependencies group (minor)," opened June 3, 2026. It bumps hcloud-go from v2.32.0 straight to v2.44.0, alongside a handful of other minor updates.

As of this writing — five weeks after it was opened, and nine days after Hetzner's July 1 deadline passed — it's still open, and its CI is red across the board: Lint Pull Request, Test Code, Verify Pull Request, and Build and push manager image all fail. The actual failure isn't an API incompatibility; it's mechanical. Verify Pull Request's log shows make generate and verify-generated-files.sh failing because the bump touched only go.mod/go.sum — nobody ran go mod vendor or regenerated the CRD manifests that CAPH's build expects to match. A bot can propose the version bump; it can't regenerate 13 files of vendored source and controller-gen output on its own, and apparently nobody has picked that up since.

That's the actual, present-tense state of the "fix" the TODO for this piece asked about: known, proposed, dated, and stuck on a mechanical step, not a design problem.

What to Actually Do Before October 1​

The July 1 deadline has already passed with no live impact on stock CAPH — but two things still need attention before October 1 closes the loop on the second wave:

• If you run CAPH from a fork or vendor it from source, don't wait on #2069. Pull the hcloud-go bump to v2.44.0 yourself, then run go mod vendor && make generate locally and commit the regenerated vendor tree and CRD manifests — that's the step the open PR is missing, not a code change.
• Audit any code that talks to the Hetzner API directly and bypasses CAPH's own client — a bootstrap hook, a Talos or cloud-init template, a custom webhook, a one-off script in a machine image build — for JSON paths like .datacenter.location.name. Those broke live on July 1, and CAPH's own clean bill of health doesn't cover them.
• Track the October 1 GET /datacenters removal separately. CAPH's controllers don't call those endpoints today, so nothing forces the bump before then — but "nothing forces it yet" is also the exact state that let this pin fall twelve releases behind in the first place.

The broader lesson for anyone operating Cluster API against a cloud provider's API directly: targeting the abstraction a provider is visibly steering you toward (Location, set at request time) rather than reading vendor internals back out of a response object is what actually insulated CAPH here — not luck, and not a fix that landed in time. Depending on owned infrastructure through a provider's own API doesn't remove this kind of maintenance burden; it just makes it visible in a go.mod diff instead of a vendor's changelog you never read.

Bex.co takes the same approach at the platform layer — Cluster API against Hetzner's own APIs, no extra abstraction layer sitting between your fleet and the provider's changelog. Star the repo on GitHub or push a git repo and get a running HTTPS service on machines you own.