CAPH Quietly Renamed Its Annotations and Finalizers — Here's Every Old-to-New Key

If a Grafana alert or Alertmanager rule on your Cluster API Provider Hetzner (CAPH) fleet ever matched on the literal string error.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io — the annotation CAPH used to set on a bare-metal host stuck needing human intervention — it stopped firing in August 2024. Not with a warning, not with a deprecation window, not with anything louder than a merged pull request. CAPH renamed that annotation to capi.syself.com/permanent-error and never looked back. The old key is just gone: nothing in the controller writes it anymore, and nothing reads it either.

That's the shape of the problem this post covers: CAPH renamed its annotation and finalizer keys to match upstream Kubernetes naming conventions, and the change is old enough now that most people have forgotten it happened — which is exactly when it's most likely to still be silently broken in whatever automation was written against the old strings. Here's the full mapping, sourced directly from the CAPH pull requests that shipped it, followed by the one detail that actually determines whether you need to do anything about it: finalizers healed themselves, annotations didn't.

The Full Old → New Mapping​

CAPH (syself/cluster-api-provider-hetzner, the Cluster API infrastructure provider a self-hosted Hetzner-backed fleet runs its node provisioning through) shipped this in two pull requests three days apart: #1455 renamed finalizers, #1458 renamed annotations. Both merged between August 20–23, 2024, and shipped in v1.0.0-beta.43 — before CAPH's own 1.0 GA that October. Every key that changed:

Annotations:

Finalizers:

Two things worth clocking in that table before moving on. First, the secret finalizer didn't just flip prefix and suffix like the others — its whole shape changed, from <owning-resource-domain>/secret to a flat infrastructure.cluster.x-k8s.io/caph-secret, so a naive find-and-replace on the pattern won't catch it. Second, the two remediation finalizers weren't renamed at all; CAPH stopped setting them, full stop. If a script expects to find either string on a live HCloudRemediation or HetznerBareMetalRemediation object, it won't — not because the string changed, but because the resource no longer carries a finalizer there.

Why It Happened, and How Quietly It Shipped​

This wasn't CAPH inventing a convention. Kubernetes itself flagged the gap in issue #119445: the API server never enforced that custom-resource finalizers be domain-qualified (example.com/name), even though the docs always said they should be, so plenty of controllers — CAPH included — shipped bare, non-qualified-looking strings like hetznercluster.infrastructure.cluster.x-k8s.io that happened to contain a dot but didn't follow the domain/name shape the convention actually specifies. Cluster API's own project hit the same issue under #10914 ("our finalizers are not domain-qualified"), which is what CAPH's maintainers cited when they opened their own tracking issues, #1450 for finalizers and #1457 for annotations, in mid-August 2024.

From issue to merged fix took about a week. Both PRs shipped with test coverage and a new docs/caph/03-reference/08-annotations.md reference page, which is more diligence than a lot of dependency-internal renames get. What they didn't ship is a blog post, a mailing list note, or anything else that would surface the change to someone who doesn't read every PR title in a provider they depend on. If you weren't specifically watching syself/cluster-api-provider-hetzner's release notes the week of August 30, 2024, you found out about this rename the same way most people find any silent breaking change in a dependency: when something stopped working and you went looking for why.

It's not an isolated incident, either — it's the same pattern CAPH has repeated since. A year later, v1.0.5 (July 2025) shipped a fix for a Hetzner Cloud API change that had broken control-plane deletions, leaving machines stuck in a terminating state, again disclosed as a release note rather than an incident writeup. Different surface, same lesson: a provider you depend on for machine provisioning can change its own contract on its own schedule, and the notification is whatever you happen to read in its changelog.

The Asymmetry That Actually Determines Your Risk​

Here's the detail that matters more than the table itself: these two renames were not handled the same way, and CAPH's current source — as of this writing, two years after the original PRs — still shows the difference plainly.

Finalizers got a back-compat shim. Every renamed finalizer constant has a matching Deprecated*Finalizer constant sitting right next to it in the source, and the reconcile loop for every affected resource type runs an add-new/remove-old pattern on every pass — controllerutil.AddFinalizer(obj, NewFinalizer) paired with controllerutil.RemoveFinalizer(obj, DeprecatedFinalizer). That means any CAPH-managed resource that's still being actively reconciled — which is to say, almost every resource in a running fleet — gets its finalizer silently migrated forward the next time the controller touches it. Pull an object created back on beta.42 today and its finalizer already reads the new string, not because anyone ran a migration script, but because the controller has been quietly fixing it up on every reconcile since.

Annotations got no such shim. The old constants — WipeDiskAnnotation, RebootAnnotation, PermanentErrorAnnotation, and the rest — were deleted from the source in the same PR that introduced the new ones. There's no DeprecatedWipeDiskAnnotation anywhere in the codebase, no code path that reads the old key and translates it, nothing. An old-style annotation key sitting on a live object today is just inert text CAPH will never look at again.

Put together, that asymmetry tells you exactly where to look: the finalizer rename is CAPH's own problem, and CAPH already solved it for you. The annotation rename is everyone else's problem — anything watching for these strings from outside CAPH's reconcile loop never got the memo, because there was no memo to get.

The Audit to Run Before You Trust Either List​

The finalizer table above is close to reference material — worth knowing, unlikely to bite you. The annotation table is the one to actually act on, because nothing self-heals it. Grep your own surface for the four old-style annotation strings, not the docs:

grep -rn \
  "wipedisk.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|\
ignore-cd.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|\
reboot.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|\
reboot.hcloud.infrastructure.cluster.x-k8s.io\|\
error.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io" \
  --include="*.yaml" --include="*.sh" --include="*.go" --include="*.rego" .

Run that across your Grafana dashboards' query definitions, Alertmanager routing rules, backup/DR scripts that snapshot or skip resources by annotation, and any custom controller or admission webhook that watches CAPH objects. PermanentErrorAnnotation's old string is the one most worth losing sleep over: it's explicitly documented as "auto-remove: disabled" — the annotation is meant to stay on the object until a human clears it after fixing whatever's actually wrong with the machine, which is precisely the kind of long-lived, page-a-human signal an external alert is likely to be keyed on. WipeDiskAnnotation and RebootAnnotation, by contrast, are transient and auto-removed by the controller within a single reconcile cycle, which makes them far less likely to have anything external watching for them in the first place.

Guarding Against the Next One​

The specific strings in the tables above are a known, closed issue — CAPH isn't going to rename these particular keys again. What's not closed is the pattern: any provider you depend on can do this again to a different key, on its own schedule, disclosed as nothing more than a PR title. The cheapest guardrail is turning the audit grep from something you run once, reactively, into something CI runs on every change to the fleet's automation:

# .github/workflows/caph-annotation-lint.yml
- name: Fail on deprecated CAPH annotation strings
  run: |
    if grep -rn \
      "wipedisk.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|ignore-cd.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|reboot.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io\|reboot.hcloud.infrastructure.cluster.x-k8s.io\|error.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io" \
      --include="*.yaml" --include="*.sh" --include="*.go" --include="*.rego" . ; then
      echo "Deprecated CAPH annotation string found — see docs/caph/annotations.md" && exit 1
    fi

That single check would have caught the stale monitoring query from this post's opening line the same week the rename shipped, instead of letting it sit dead for two years. It's a five-line job, not a project, and it generalizes: the same pattern — a small denylist grep wired into CI, refreshed whenever you notice a provider dependency renaming something — is cheaper than finding out a permanent-error alert has been silently dark the next time a bare-metal host actually needs a human.

Why a Provider's Renaming Is Your Maintenance Cost​

None of this required CAPH to ship a bug. It required nothing more than running an old-format annotation string in a query or script that nobody revisited after August 2024 — the same failure mode as the Hetzner Cloud API deprecations that hit CAPH-managed fleets earlier this month, just one layer further from the surface. That's the cost of running the infrastructure provider yourself instead of renting a platform that runs it for you: a hosted PaaS absorbs its own provider's internal renames on your behalf, invisibly, because tracking a dependency's changelog is their job, not yours. A self-hosted Cluster API fleet makes that job yours by default.

It's a fair trade for owning the machines instead of renting someone else's black box — but only if "grep the fleet for stale provider-internal strings" gets added to the same recurring maintenance cadence as dependency bumps and CVE patching, rather than treated as a one-time cleanup you either did in 2024 or didn't.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, on the same Cluster API foundation this post is auditing. Star the repo on GitHub or deploy your first app today.