CAPH Stopped Checking Your Server Type: What Hetzner's Hardware Churn Teaches About Pinning Machine Templates

In October 2025, Hetzner deprecated a swath of its Cloud server types — the workhorse cx22 and cpx11 among them, with removal set for January 1, 2026 — and shipped a reworked lineup in their place. That same month, Cluster API Provider Hetzner (CAPH) merged a change that deleted its own hardcoded server-type allowlist: since v1.0.7, hcloudMachine.spec.type accepts any string, and a typo or a retired type sails through kubectl apply only to die later at Hetzner's API.

If you run a Cluster-API-managed fleet on Hetzner, here is the replacement discipline in four lines: pin every MachineTemplate to an exact server type, validate that type against the live Hetzner API in CI, check per-location availability (not just existence), and watch the deprecation field — because the guardrail you relied on is gone and it is not coming back.

This post covers what exactly changed, why a hardcoded list never stood a chance, what the new failure mode looks like in practice, and the concrete CI recipe that replaces the deleted check.

What exactly changed: an allowlist became a free-form string​

Before the change, CAPH's HCloudMachine API type carried a kubebuilder validation annotation — an enumerated list of acceptable spec.type values baked into the CustomResourceDefinition. Apply a Machine with type: cxx99 and the Kubernetes API server rejected it instantly, before any controller ever saw it. That is the behavior every CAPH operator internalized: if kubectl apply accepted your MachineTemplate, the server type was at least a real one.

Pull request #1694, "Allow all values for HCloudMachine type," removed that annotation. The PR body is worth quoting in full because it is refreshingly blunt:

The list of valid machine types gets changed by the Hetzner from time to time. CAPH no longer validates this string. It is up to the user to use a valid type as not all types are available in all the locations.

The change shipped in CAPH v1.0.7, and the field's own documentation now reads as a polite warning label:

The list of valid machine types gets changed by Hetzner from time to time. CAPH no longer validates this string. It is up to you to use a valid type. Not all types are available in all locations.

Note the last sentence — it flags the second dimension of this problem. Even a perfectly spelled, non-deprecated type can fail if it is not available in the location your Machine is landing in. The old allowlist never checked that either; it could not, because availability varies per datacenter and changes over time. The new world just makes the gap visible instead of partially papered over.

Why a hardcoded list never stood a chance​

CAPH did not remove the validation on a whim. Hetzner's server catalog is a moving target, and a provider that hardcodes it signs up for a maintenance treadmill it will always lose. The timeline tells the story:

The October 2025 event was the forcing function. It was not a quiet tweak: Flatcar, Azure Container Linux CI configs, Pulumi examples, Molecule plugins, and image-upload tooling all had to bump defaults from cx22 to cx23 and cpx11 to cpx22 within weeks, because after January 1, 2026, new orders on the old types fail outright.

Any CAPH release whose CRD still enumerated the old list would have been wrong in one direction; any release enumerating only the new list would have been wrong for clusters still lawfully running the old types until removal. An allowlist that is wrong in both directions simultaneously is not a safety check — it is a liability. Deleting it was the honest fix.

There is a second, quieter reason the list had to go: Hetzner now scopes availability per location. A type can exist, be non-deprecated, and still not be orderable in fsn1 while being fine in nbg1. No static enum can express "valid, except where it isn't." Only a live query against the API can answer that question, which means the check was always in the wrong layer.

The new failure mode: rejection moves from apply time to reconcile time​

Here is the concrete behavior change, stated as plainly as possible:

• Before: type: cxx22 (note the typo) → kubectl apply fails immediately with a validation error. You fix it in seconds, in your terminal, with full context.
• After: type: cxx22 → apply succeeds. The Machine object is created. CAPH's controller picks it up, calls Hetzner's API to create the server, and Hetzner answers with invalid_server_type — "the server type does not fit for the given server or is deprecated." The Machine stalls. If this template backs a MachineDeployment that the autoscaler is trying to scale up at 3 AM, you find out from an alert, not from your shell.

The same applies to a retired type with no typo at all. A template pinning cx22 that applied cleanly for a year starts failing the moment Hetzner removes the type — and nothing in your Git history changed. Downstream projects hit exactly this: one automation tool's changelog notes "Hetzner deprecated these on Jan 1, 2026 — new orders fail," and a pricing-table fix describes the API rejecting provisioning with a "server type is deprecated" error after the cx22 generation was retired in early 2026. Your fleet's templates are exposed to the same cliff on whatever schedule Hetzner sets next.

This is strictly worse as a debugging experience. Apply-time errors are synchronous, local, and carry the full schema context. Reconcile-time errors are asynchronous, surface through controller events and Machine status conditions, and arrive mixed in with every other reason a Machine can fail to provision — rate limits, image problems, network attachment failures. The signal did not disappear, but its signal-to-noise ratio collapsed.

The replacement discipline: validate against the live API in CI​

CAPH moved the check; it did not eliminate the need for it. The right layer is your own pipeline, querying the source of truth — Hetzner's API — at the moment you change a template, not the moment a controller reconciles it. Four practices, in ascending order of rigor:

1. Pin exact types, never generations. Write cx23, not "whatever replaces cx22." The October 2025 rework proved that Hetzner retires whole generations; a template that names a generation by implication rots silently. Pinning makes the dependency explicit and greppable, so the next deprecation announcement maps to a concrete list of files to change.

2. Gate template changes on a live lookup. The hcloud CLI exposes exactly what you need. Since the 2026 CLI updates, hcloud server-type describe prints an Available and Recommended line for every location — the per-location truth the old CAPH allowlist could never encode. A minimal CI step greps that output directly:

# Fail unless $SERVER_TYPE is Available in $LOCATION
hcloud server-type describe "$SERVER_TYPE" \
  | grep -A1 "Location:[[:space:]]*$LOCATION" \
  | grep -q "Available:[[:space:]]*yes"

Prefer JSON? The API's GET /v1/server_types returns each type with a locations array carrying available and a per-location deprecation field (null means not deprecated), so the equivalent script-friendly gate is:

curl -sH "Authorization: Bearer $HCLOUD_TOKEN" \
  "https://api.hetzner.cloud/v1/server_types?name=$SERVER_TYPE" \
  | jq -e --arg loc "$LOCATION" '
      .server_types[0].locations[]
      | select(.name == $loc)
      | select(.available and .deprecation == null)'

If either query fails, the template change does not merge. This restores apply-time-style feedback at PR time, which is arguably better than what the old annotation provided.

3. Watch the deprecation field, not the deprecated field. Hetzner announced in September 2026 that the flat deprecated response field disappears on November 2, 2026. For server types, deprecation lives on the per-location entries. Any check you write today should read the deprecation field — otherwise your shiny new CI gate breaks two months after you build it.

Hetzner's own SDKs model this correctly: hcloud-go ships a ValidateServerType helper that fails on deprecated types, hcloud-python raises ValueError("server type ... is deprecated"), and the Terraform provider's changelog documents a check-block pattern asserting is_deprecated is false. Copy the pattern your stack already uses; the point is that validation lives at the provider edge, evaluated against live data.

4. Re-validate on a schedule, not just on change. A template that passes CI today can fail in production next quarter with zero commits in between — the cx22 retirement proved it. A weekly cron that re-runs check #2 against every type pinned in your fleet's templates turns "Hetzner removed a type" from a 3 AM page into a Monday-morning ticket. This is the one practice the old annotation could never have given you, and it is the highest-value item on this list.

The design lesson: provider-owned enums do not belong in CRDs​

Step back and the episode is a clean case study in API design for infrastructure providers. CAPH's allowlist encoded someone else's product catalog in its own schema. That creates a coupling where the downstream project must cut a release every time the upstream vendor renames a SKU — and in the window between the vendor's change and the provider's release, the validation is either blocking legitimate new types or blessing retired ones. CRD validation is versioned with your platform, but the thing it validates is versioned with someone else's price list. Those two clocks will never agree.

The correct layer for this check was always a live one: an admission webhook that queries the Hetzner API, an SDK-side helper evaluated at deploy time, or — most pragmatically — CI against the live catalog. All three read the current truth instead of a snapshot frozen at CRD-generation time. CAPH deleting the enum was not the removal of safety; it was the removal of safety theater, relocating the real check to where the data lives.

Expect this pattern to repeat. Hetzner's catalog will keep churning — new generations, per-location rollouts, GPU types with their own availability quirks — and every infrastructure tool that snapshots the catalog instead of querying it will hit the same wall. If you maintain anything with a Hetzner server type baked in — Terraform modules, Pulumi examples, image builders, internal CLIs — audit it for the same hardcoded-list smell. The ecosystem-wide scramble of late 2025, when project after project bumped cx22 to cx23 in the same quarter, is what that smell costs at scale.

Looking ahead​

The November 2026 removal of the legacy deprecated field is the next scheduled cliff: any tooling still reading the flat boolean instead of the per-location deprecation entries will go blind to deprecations overnight. If your fleet's templates survived the cx22 retirement, use the quiet months to build the CI gate described above — before the next generation turnover tests whether you actually did.

The deeper shift is worth naming: as Hetzner scopes more of its catalog per location — availability today, deprecation detail tomorrow — "which machine type" stops being a static string in a template and becomes a small query against a live system. Fleets that treat it as data to be refreshed will ride out the next rework; fleets that treat it as text to be typed will meet invalid_server_type at the worst possible hour.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.