410 Gone on October 1: The Pre-Deadline Hetzner Datacenter Audit for Cluster API Fleets

On October 1, 2026, a Hetzner Cloud API call that returned 200 OK for years will start returning 410 Gone — permanently, with no fallback, and no retry that can save it. The bill for ignoring the warning is not a deprecation header you can triage next sprint. It is a provisioning failure at 3 a.m. when Cluster API tries to reconcile a machine that references a datacenter that no longer exists.

Hetzner did not do this abruptly. The timeline was published in four deliberate steps across ten months. Most fleets still have at least one datacenter: nbg1-dc3 hiding in a checked-in manifest, a Terraform state file, or a Python script that calls server.datacenter — and after October 1, every one of those becomes a hard error.

This is the 15-minute audit you run before the deadline does it for you.

TL;DR — run this now, fix what it finds before October 1:

# 1. Cluster API & raw manifests (CAPH, kubeadm, Fleet)
rg -n "datacenter" --glob '*.{yaml,yml}' | grep -v "location"
 
# 2. Terraform (code + state)
rg -n "datacenter" --glob '*.tf' --glob '*.tfvars'
jq -r '.. | strings | select(contains("datacenter"))' terraform.tfstate | sort -u
 
# 3. Ansible
rg -n "datacenter(_info)?" --glob '*.{yml,yaml}'
 
# 4. Application code (Go / Python clients)
rg -n "Datacenter|datacenter" --glob '*.{go,py}'
 
# 5. Live API sanity check (expect 410 after Oct 1, 200 before)
curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $HCLOUD_TOKEN" \
  https://api.hetzner.cloud/v1/datacenters
# Today: 200 — After Oct 1: 410

If any of the first four commands produce output, you have work to do. The rest of this post maps where that output hides, why 410 is not a 404, and the exact replacement for each hit — plus a CI gate that makes the fix stick.

The four-step deprecation you were supposed to notice​

Hetzner published the datacenter phase-out as a stack, not a single announcement. Each step removed a different surface. Treating them as one event is why teams patch one layer and miss the next.

Note the subtlety: the July 1 removal already broke any client that sends datacenter. The October 1 removal breaks any client that reads GET /datacenters. If you fixed your Terraform hcloud_server blocks in June but left a data "hcloud_datacenter" "dc" lookup for a validation script, you feel fine today and fail in October.

The client libraries tracked the API within weeks:

• hcloud-go v2.33.0 deprecated ServerCreateOpts.Datacenter; v2.45.0 (July 2026) removed Server.Datacenter entirely — the release notes explicitly say "since the property was already removed from the API, we do not consider this a breaking change."
• hcloud-python v2.13.0 deprecated server.datacenter; v2.23.0 removed it with a migration snippet from server.datacenter to server.location.
• hcloud CLI v1.59.0 deprecated --datacenter; v1.67.0 removed --datacenter and stopped printing Datacenter in server describe.
• terraform-provider-hcloud v1.58.0 deprecated datacenter on servers and primary IPs, v1.66.0 deprecated hcloud_datacenter data sources, v1.67.0 removed datacenter from those resources.

If your lockfiles predate those versions, you are shipping the deprecated path even if your manifests look clean.

Why 410 Gone is not "just a 404"​

HTTP distinguishes two kinds of absence:

• 404 Not Found — "I don't know this resource. It may have never existed, or you typed it wrong, or it moved. Try again."
• 410 Gone — "This resource existed, I intentionally deleted it, it has no forwarding address, and it is not coming back." (RFC 7231 §6.5.9)

The practical difference for an operator:

• Retrying a 404 sometimes makes sense — a DNS blip, a typo in an ID, an eventual-consistency window.
• Retrying a 410 never makes sense — the server is telling you the endpoint itself is retired. Your controller's exponential backoff just burns API rate limit and logs.

Hetzner chose 410 deliberately. The datacenters endpoints do not return 404 as if you mistyped a datacenter name. They return 410 to signal a contract removal. Terraform's hcloud_datacenter data source, CAPH's old provisioning paths, and any homegrown script that enumerates available server types per datacenter will not get an empty list they can handle gracefully. They get an error whose correct handling is to change the code that called it, not to add a retry.

You already saw this pattern on July 1. Teams running the Drone autoscaler on the vendored pre-v2 hcloud-go client discovered on July 2 that every server create with datacenter now returns invalid_input — not a warning, a rejection. The datacenters endpoint on October 1 is the same class of failure, just one layer removed: not "your field is rejected" but "the collection you list does not exist."

Where datacenter hides in a real fleet​

A Cluster-API-on-Hetzner fleet typically has six places to check. The table doubles as a fix map.

1. CAPH / Cluster API​

Early CAPH examples and many copy-pasted cluster-template.yaml files contained:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HetznerMachineTemplate
metadata:
  name: workers
spec:
  template:
    spec:
      datacenter: nbg1-dc3   # <- remove
      location: nbg1         # <- use this
      serverType: cpx31

And at the cluster level, some older HetznerCluster specs referenced datacenter to pin control planes. Current CAPH expects failureDomain or controlPlaneRegions that resolve to a location, not a datacenter slug. If you generated your manifests with clusterctl generate cluster --infrastructure hetzner before mid-2026, regenerate against the latest template and diff.

Also check HetznerMachine objects that were created before July 1 and now have stale status: kubectl get hetznerclusters -A -o yaml | rg datacenter catches persisted status that controllers may still try to read.

2. Terraform​

This is where most fleets have the longest tail. Three distinct hits:

# a) The attribute on resources you create
resource "hcloud_server" "cp" {
  # datacenter = "nbg1-dc3"   # <- delete
  location   = "nbg1"
  server_type = "cpx31"
  image      = "ubuntu-24.04"
}
 
# b) The data source you use for lookups
# data "hcloud_datacenter" "dc" {    # <- delete, 410 after Oct 1
#   name = "nbg1-dc3"
# }
data "hcloud_location" "loc" {
  name = "nbg1"
}
 
# c) Capacity / placement logic that enumerated server types per datacenter
# Before:  data.hcloud_datacenter.dc.server_types.supported
# After:   data.hcloud_server_type.cpx31.locations[*].available

Do not forget state. Even after you fix *.tf, terraform.tfstate may still contain datacenter in attributes and tfstate lineage:

# Find stale state - should be empty
jq -r '.resources[]? | select(.type | test("hcloud_")) | .instances[].attributes | keys[]?' \
  terraform.tfstate | rg datacenter
 
# Check remote state if you use S3 / Terraform Cloud
terraform state list | rg datacenter
terraform show -json | jq '.. | objects | select(has("datacenter"))'

The provider's v1.67.0 release intentionally left datacenter in state for reduced blast radius, which means terraform plan after upgrading may still show the field as present until you terraform apply once with the corrected config.

3. Ansible​

The collection hetzner.hcloud marked datacenter_info as Removed in: major release after 2026-10-01:

# Before
- hetzner.hcloud.datacenter_info:
  register: dcs
 
# After
- hetzner.hcloud.location_info:
  register: locations

If you have custom server_info tasks that filter by datacenter, switch the filter to location. The module's deprecation message links to the same June 2 changelog entry.

4. Application code​

Go before/after from the hcloud-go migration guide:

// Before
server, _, _ := client.Server.GetByID(ctx, 42)
fmt.Println(server.Datacenter.Location.Name)
fmt.Println(primaryIP.Datacenter.Location.Name)
 
// After
server, _, _ := client.Server.GetByID(ctx, 42)
fmt.Println(server.Location.Name)
fmt.Println(primaryIP.Location.Name)
 
// Create path
client.Server.Create(ctx, hcloud.ServerCreateOpts{
    Name:       "worker-1",
    ServerType: &hcloud.ServerType{Name: "cpx31"},
    Location:   &hcloud.Location{Name: "nbg1"}, // was Datacenter
    Image:      &hcloud.Image{Name: "ubuntu-24.04"},
})

Python equivalent: server.datacenter -> server.location.

Check vendored clients: go.mod pinning hcloud-go below v2.33.0 or requirements.txt with hcloud < 2.13.0 means the symbol exists in code even if your import path still compiles.

5. CLI and shell wrappers​

The CLI change breaks scripts that construct hcloud server create strings:

# Before
hcloud server create --name worker-1 --server-type cpx31 --image ubuntu-24.04 --datacenter nbg1-dc3
 
# After
hcloud server create --name worker-1 --server-type cpx31 --image ubuntu-24.04 --location nbg1

Search your Makefile, justfile, scripts/*.sh, and GitHub Actions workflows — the flag appears there more often than in YAML.

6. Service discovery and controllers​

Two easy-to-miss dependents:

• VictoriaMetrics / Prometheus hetzner_sd_configs that previously labeled targets by __meta_hetzner_datacenter; after July 1 the meta label is __meta_hetzner_location. Dashboards and alerting rules that group by (datacenter) silently return empty series.
• hcloud-cloud-controller-manager — Hetzner explicitly required >= v1.30.1 before July 1 to handle the location transition. Older CCM versions reconcile nodes against the removed field and can leave Node objects with mismatched providerID formats.

The migration map you can keep​

Copy this table into your runbook. It is the only mapping Hetzner publishes implicitly across three changelog entries.

A useful mental model: datacenter was a physical hall; location is a routable region. Hetzner's physical topology used datacenter slugs like nbg1-dc3 for individual halls inside a region like nbg1. The API now only exposes the region because placement within a region is an implementation detail Hetzner no longer wants tenants to pin.

Pin the fix in CI — so it does not drift back​

A grep check that blocks datacenter from re-entering main is cheaper than a 3 a.m. page. Two options, pick one.

GitHub Actions (fail the PR)​

# .github/workflows/hetzner-datacenter-guard.yml
name: hetzner-datacenter-guard
on: [pull_request]
 
jobs:
  no-datacenter:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Fail if `datacenter` is still referenced
        run: |
          set -euo pipefail
          echo "Scanning for stale datacenter references..."
          # Allow this file and changelog links to mention the word for documentation
          if rg -n "datacenter" \
              --glob '!**/hetzner-datacenter-410-gone-audit.md' \
              -g '!CHANGELOG.md' \
              --glob '*.{yaml,yml,tf,tfvars,go,py,sh,md}' .; then
            echo "::error::Found 'datacenter' reference. Use 'location' (see https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters)"
            exit 1
          fi
          echo "No stale datacenter references found."

Allowlist deliberately narrow: only the post you are reading now gets to say the word. Everything else must migrate. If you have intentional documentation that must mention datacenter (like a runbook), add that single path to the allowlist — not a blanket ignore.

Pre-commit hook (fail locally)​

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: hetzner-no-datacenter
        name: forbid Hetzner datacenter
        entry: bash -c 'rg -n "datacenter" --glob "*.{yaml,yml,tf,go,py,sh}" . && { echo "use location, not datacenter"; exit 1; } || exit 0'
        language: system
        pass_filenames: false

Run once to confirm you are clean, then enforce:

pre-commit run hetzner-no-datacenter --all-files
# expected: no output, exit 0

Version pinning deserves its own guard. Add to CI:

# Fail if lockfiles pin a pre-migration client
for f in go.mod requirements.txt package.json .terraform.lock.hcl; do
  [ -f "$f" ] || continue
  if rg -n "hcloud-go@v2\.(2[0-9]|3[0-2])|hcloud.*<.*2\.13|terraform-provider-hcloud.*1\.5[0-7]" "$f"; then
    echo "Stale Hetzner client pinned in $f — bump to hcloud-go >=2.45.0 / hcloud-python >=2.23.0 / provider >=1.67.0"
    exit 1
  fi
done

What to do in the next 39 days​

1. Today (15 minutes): run the five greps in the TL;DR. Open one PR per surface (manifests, Terraform, Ansible, go/python) — small PRs merge faster before a deadline than one big one.
2. This week: bump hcloud-go, hcloud-python, terraform-provider-hcloud, and hcloud-cloud-controller-manager past the migration versions. Run terraform plan on every workspace; empty diff is the expected result.
3. Before Sep 20: run curl https://api.hetzner.cloud/v1/datacenters in a nightly cron that alerts on 200. When it flips to 410, you want to know because your dashboard flipped, not because a customer's machine did.
4. After Oct 1: keep the CI guard. The API will not come back.

The broader lesson is not specific to Hetzner. Datacenter -> location, type -> generation (see Hetzner's June 15 CPX/CAX reshuffle), zone -> region — providers phase out the most physical naming first because it leaks implementation details they want freedom to rearrange. A Cluster-API-based fleet that pins the physical slug rather than the logical location inherits someone else's floor plan. A fleet that targets location pins the contract that survives a hall being rebuilt.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Cluster API owns the machine lifecycle under the hood, so when Hetzner renames a field, you fix a location string rather than reprovision a datacenter. Star the repo on GitHub or deploy your first app today.

Sources​

• Hetzner Cloud API Changelog — Phasing out Datacenters in favor of Locations (Dec 16, 2025) — https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters
• Hetzner Cloud API Changelog — Datacenter server_types information and recommendation fields are deprecated (Apr 1, 2026) — https://docs.hetzner.cloud/changelog#2026-04-01-datacenter-deprecations
• Hetzner Cloud API Changelog — Deprecation Data Center endpoints (Jun 2, 2026) — https://docs.hetzner.cloud/changelog#2026-06-02-datacenters-deprecated
• Hetzner Cloud API Changelog — Remove Datacenters properties for Primary IPs and Servers (Jul 1, 2026) — https://docs.hetzner.cloud/changelog#2026-07-01-removing-datacenters
• hcloud-go v2.45.0 release — Remove datacenter property — https://github.com/hetznercloud/hcloud-go/releases/tag/v2.45.0
• hcloud-python v2.23.0 release — Remove datacenter property — https://github.com/hetznercloud/hcloud-python/releases/tag/v2.23.0
• terraform-provider-hcloud v1.58.0 / v1.66.0 / v1.67.0 releases — Deprecation and removal of datacenter — https://github.com/hetznercloud/terraform-provider-hcloud/releases
• hcloud CLI v1.67.0 release — Remove datacenter flags — https://github.com/hetznercloud/cli/releases/tag/v1.67.0
• hetzner.hcloud Ansible collection — datacenter_info deprecation — https://docs.ansible.com/projects/ansible/latest/collections/hetzner/hcloud/datacenter_info_module.html
• HTTP 410 Gone semantics — https://http.dev/410
• Drone autoscaler issue #154 — Vendored hcloud-go sends datacenter, fails with invalid_input after Jul 1 — https://github.com/drone/autoscaler/issues/154
• VictoriaMetrics issue #10909 — hetzner_sd_configs datacenter -> location — https://github.com/victoriametrics/victoriametrics/issues/10909
• Kubermatic machine-controller issue #2053 — Hetzner datacenter deprecation timeline — https://github.com/kubermatic/machine-controller/issues/2053