CAPH v1.1.7 and the 429 That Ate a Healthy Machine: Why a Rate-Limited FindServer Marked 31 HCloudMachines as Gone

A ready server vanished without a single API call saying it was gone. The Hetzner Cloud API answered 429 Too Many Requests, and the Cluster API Provider Hetzner (CAPH) controller wrote that answer down as "this machine does not exist."

On a management cluster watching over tenant workloads, the next step is automatic: a MachineHealthCheck sees a machine whose infrastructure reference no longer exists, marks it unhealthy, and orders a replacement. A node serving live traffic gets drained and deleted because a throttle, not a termination, happened for 30 seconds. In the incident that surfaced as syself/cluster-api-provider-hetzner#1966, that exact chain hit 31 HCloudMachines across 8 clusters simultaneously after a roughly 24-hour rate-limit window.

CAPH v1.1.7, released June 17, 2026, is a three-line patch release that erases that failure mode and two quieter ones beside it. If your fleet runs on CAPH — as every bex-managed Hetzner fleet does — this is the release where "read the changelog even on patch day" stops being hygiene and becomes the thing that keeps a healthy box from being reprovisioned under you.

This post lands the deliverable up front: what v1.1.7 shipped, verbatim why the ready-machine path was wrong, and a concrete operator checklist to audit before the next 429.

What v1.1.7 Actually Shipped: Three Fixes, One Lesson​

Full diff: v1.1.6...v1.1.7. The changelog is short enough to quote in whole — which is exactly why it is easy to miss if you skim patch notes for features.

All three are correctness, not features. v1.1.7 exists because a throttle on a shared API taught a controller to lie about existence, silence about bare-metal failure, and trust a stale cache during the one operation — rollout — where staleness hurts.

The Dangerous One, Line by Line: Why a 429 Became "Not Found"​

Hetzner Cloud's API is shared per project: 3,600 requests per hour, refilling at 1 per second (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset headers on every response). CAPH, the Hetzner Robot API, and any other caller against the same project share the bucket. The CAPH rate-limit docs are explicit: on 429 the controller sets a HCloudRateLimitExceeded condition to true, blocks that one object for a window, and requeues it after 30 seconds so another HCloudMachine can still reconcile normally.

The bug was the interaction with Status.Ready:

Before (buggy, from #1966):

if s.scope.HCloudMachine.Status.Ready {
    hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "findServer")
    return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
}
// ...
// If HCloudMachine is ready, stop reconciling.
return reconcile.Result{}, nil // ← no requeue; stale conditions never cleared

In practice the second path dominated. findServer hit a 429, HandleRateLimitExceeded correctly noted the throttle, but because the machine was already Ready, the controller exited with no requeue. The Ready condition stayed True, the HCloudRateLimitExceeded condition stayed set, and the "does this server exist?" check above it had already written the answer as "no" on the throttled path. From the outside the machine looked simultaneously ready and non-existent — a contradiction the next reconciler, MachineHealthCheck, resolves by replacing the node.

The report on v1.0.12 and v1.1.0-alpha.2 notes the same shape survived two minor lines, and one operator saw it sweep 31 machines at once. The sweep matters because the Hetzner limit is per-project, not per-machine. One burst — say, HetznerCluster updates triggering HCloudMachine watches (itself a documented source of excess GET /v1/servers/:id calls in #926) — can throttle many machines in the same hour, and every one of them takes the buggy branch together.

After (v1.1.7):

hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "findServer")
return reconcile.Result{RequeueAfter: 30 * time.Second}, nil

No Ready gate. Every 429 records the condition and requeues. The 30-second delay is not a guess — it matches the existing backoff CAPH already used for rate-limited objects, long enough for 30 tokens to refill at 1/sec but short enough that a real missing server is noticed quickly.

Two details that make this not cherry-picking:

• A ready machine throttling is the typical case. A fleet that has been stable for hours has almost all machines Ready. An idle fleet that only just provisioned nodes would have many NotReady machines that did take the requeue path and self-healed. The bug punished stability.
• The fix does not retry immediately. CAPH does not busy-loop on 429. It still parks the single throttled object and lets other machines reconcile. The cost of the fix is one delayed requeue, not a retry storm — which is why #926 and the newer exponential-backoff work in #1757 remain complementary rather than redundant.

The Other Two Fixes Deserve a Read Too​

Bare-metal provisioning that never emitted an event​

On HetznerBareMetalHost (HBMH — Robot API, not Cloud), a failed provisioning path simply returned an error without recording a Kubernetes Event. kubectl get events --field-selector involvedObject.name=<host> stayed empty while HetznerBareMetalHost sat in Failed. PR #2101 adds the missing Eventf in pkg/services/baremetal/host/host.go. The change is one file, one purpose: if a fleet mixes Cloud and bare metal (or a future pool does), a bare-metal failure now shows up where every operator already looks first.

Why it landed in the same patch: the same release that taught a Cloud machine not to lie about existence also taught a Robot host not to swallow the reason it couldn't become one. Both are "the fact the operator needs is one hop further away than the condition says."

A stale LB status that skipped a real attach​

During a control-plane rollout, CAPH removes a target from the Hetzner Cloud Load Balancer and re-adds it (often with the same IP and server ID). HetznerCluster.Status.ControlPlaneLoadBalancer.Target is only updated when the HetznerCluster controller itself reconciles — independently of the HCloudMachine controllers driving the rollout. #2094 and #2107 show the resulting false positive: the stale Status still lists the target as present, the attach step is skipped, and the new control-plane node never actually joins the LB.

The fix is a deliberate trade: when ServerAvailableCondition is not yet True (mid-rollout), fetch live LB targets via the HCloud API; once True (steady state), trust the cached Status and avoid one API call per reconcile. One extra API call per machine lifecycle, only when it matters, in exchange for not silently dropping a control-plane node out of the LB.

Together the three tell a coherent story: a platform that thinks it is paying Hetzner for machines is actually paying for a reconciler's opinion about machines, and that opinion's correctness lives in a handful of branches that only run under load, during rollout, or on bare metal.

Why "CNCF-Adjacent and Hetzner-Maintained" Is Not a Substitute for Reading the Release​

CAPH is Syself-maintained, not Kubernetes SIG-owned. It follows the CAPI v1beta1 contract on the v1.1.x line, shimmed through Cluster API's temporary v1beta1 compatibility layer (supported through CAPI v1.15, but upstream docs explicitly call it not recommended long-term). That layer lets v1.1.x keep working on CAPI v1.11–1.15, but it also means a v1.1.7 that reads as "patch" is still operating on a contract upstream is already steering operators away from.

Two context pieces make changelog discipline non-optional:

• CAPH stopped validating hcloudMachine.spec.type in late 2024 (#1694). The allowed-values list could not keep pace with Hetzner's own catalog changes — most recently the June 15, 2026 switch from hardware-type SKUs to generation SKUs (CPX Gen2, CX Gen3/CAX on Ampere). A typo'd or newly-retired server_type in a HetznerMachineTemplate now fails at Hetzner's API, not at kubectl apply. If your templates still hardcode cpx31 from February, June's catalog may not contain it at the new price. The provider deliberately pushed that check onto operator CI.
• Patch does not mean small blast radius. v1.1.7 is a patch — v1.1.6 → v1.1.7 — but the per-project token bucket makes a single 429 a fleet-wide event. A release whose title looks like "a fix and two cherry-picks" can be the difference between one machine self-healing in 30 seconds and a whole pool replacing itself.

The honest read from the bex perspective: owning the machines does not spare you from upstream's reconciliation opinions about them. Self-hosting removes the hosted PaaS's margin-protecting price hikes, not its subscription to reading a provider changelog the way you would read the kernel changelog before a node OS bump.

Operator Checklist: What to Audit Before the Next 429​

This is a bex-fleet-specific checklist, but any CAPH fleet will recognize it. Run it before you cut the next MachineDeployment rollout through a shared Hetzner project.

1. Pin the version, read the three PRs​

# What are you actually running today?
kubectl -n caph-system get deployment caph-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
 
# What did you skip?
# Open https://github.com/syself/cluster-api-provider-hetzner/compare/v1.1.6...v1.1.7
# Read #2099, #2101, #2107 in full — not the release title alone.

If you are on <= v1.1.6 (or still on v1.0.x), you carry all three bugs. The 429 false negative alone justifies a bump; the bare-metal and LB fixes are free correctness on the same binary.

2. Audit every hardcoded server_type​

grep -R "server_type:" config/ clusters/ 2>/dev/null
grep -R "spec:\s*type:" templates/ 2>/dev/null | grep -i -E "cpx|cx|cax|ccx"
# Cross-check against https://docs.hetzner.com/cloud/servers/ and your live project:
hcloud server-type list -o noheader | cut -f1 -d" "

June 2026's generation rename (CPX Gen2 x86 as "Cloud Regular Performance", CX Gen3 x86 / CAX on Ampere as "Cloud Cost-Optimized") means a string that applied cleanly in May can 404 in August. CAPH will no longer tell you at admission time.

3. Make 429 visible and cheap to spot​

The Hetzner headers tell you how close you were before you throttled:

curl -s -D - -H "Authorization: Bearer $HCLOUD_TOKEN" \
  https://api.hetzner.cloud/v1/servers?per_page=1 -o /dev/null | \
  grep -i ratelimit
# RateLimit-Limit: 3600
# RateLimit-Remaining: 1842
# RateLimit-Reset: 1721828912

Wire these into the dashboard you already watch for fleet health:

• Alert when RateLimit-Remaining < 500 on any project token.
• Alert when any HCloudMachine or HetznerCluster exposes HCloudRateLimitExceeded == True.
• Keep controller logs for findServer 429s (the reconcile path that actually throttles under scale).

One project token means one burst is a fleet event. Treat a 429 the way you treat a NodeNotReady — a signal, not a success path that happened to retry.

4. Tune MachineHealthCheck so a throttle is not a death sentence​

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: workers
spec:
  clusterName: bex-prod
  selector:
    matchLabels: { cluster.x-k8s.io/deployment-name: workers }
  checks:
    unhealthyNodeConditions:
      - type: Ready
        status: "Unknown"
        timeout: 5m
      - type: Ready
        status: "False"
        timeout: 5m
  remediation:
    triggerIf:
      unhealthyLessThanOrEqualTo: 50%  # never replace >50% at once

The point is not to disable remediation — bare-metal really does fail and should be replaced — but to cap the blast radius of a correlated false positive. With v1.1.7 the correlated false positive should not happen, but a 24-hour shared-bucket throttle from another caller (CI, autoscaler, rogue script) can still set many HCloudRateLimitExceeded conditions at once. A rolling throttle that parks machines for 30 seconds each is self-healing; a MachineHealthCheck that replaces half a pool at once is not.

Also watch the sibling bug #1891 taught: unauthorized (bad token) errors are handled before rate-limit errors, and a 429 must not clear the HCloudCredentialsInvalid condition. If you rotate HCloud tokens via automation, test that rotation fails into CredentialsInvalid, not into RateLimitExceeded.

5. Plan the v1beta2 hop deliberately​

v1.1.x is the last v1beta1 line. v1.2.x is the v1beta2 contract aligned with CAPI v1.11+. The upgrade is not just a binary swap — CRD v1beta1/v1beta2 coexistence, MachineHealthCheck spec reshaping (nodeStartupTimeout string → checks.nodeStartupTimeoutSeconds int, among others in k8s-cluster-api#), and condition renames all travel together. Put the provider upgrade in the runbook before the CAPI v1.12 in-place-update feature set, not after — v1.1.x's compatibility shim is supported only until CAPI v1.15 and is explicitly not the long-term path.

The Takeaway Isn't "Patch Faster" — It's "Treat Patch Notes as Part of the Fleet's Control Loop"​

A 3,600-request-per-hour bucket sounds generous until you remember it is per-project, not per-controller, and every HetznerCluster status write that triggers a HCloudMachine watch adds GET /v1/servers/:id calls you never listed in a capacity plan (the GET /v1/servers/:id amplification in #926 is a real trace, not a hypothetical). CAPH already brute-forces 429s by parking the throttled object for 30 seconds — a reasonable choice that keeps correctness local to one machine. But a single branch that skipped that park when Ready == true turned a local backoff into a global lie about existence.

The fix is small. The discipline it asks for is not:

• Read patch releases as if they could touch reconcile correctness — because on infrastructure providers, they do.
• Make 429 as visible as 5xx in your dashboards. A throttle that writes the wrong status is worse than one that simply slows you down.
• Own the validation CAPH deliberately stopped owning. Hardcoded server_type strings and bare-metal event silence are now operator problems by design.

Bex chose CAPH because it puts declarative machine lifecycle on owned Hetzner hardware instead of behind a vendor console. That bet holds — the upstream is maintained, the fix landed within the v1.1.x patch train, and a bex fleet that tracks it stays on the right side of the throttle. The price of that bet is just what this incident names honestly: you own the reconciliation loop in the same sense you own the hardware. The hardware answers 429 when you ask too much; the loop decides whether to hear "try again" or "gone."

Sources: CAPH v1.1.7 release and diff v1.1.6...v1.1.7; issues #1966, #2094; PRs #2099, #2101, #2107; Hetzner Cloud rate limiting (3600/hour, 1/sec refill) and CAPH rate-limit docs; GET /v1/servers/:id amplification in #926; unauthorized-before-rate-limit fix #1891; server_type validation removal #1694; CAPH v1beta2 migration notes (v1.1.x on v1beta1 compat layer through CAPI v1.15).

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.