hcloud-cloud-controller-manager's Experimental Server Cache: Fewer Hetzner API Calls, and the Staleness Question It Reopens

Every reconcile loop on a Hetzner fleet spends the same currency: Hetzner Cloud API calls, drawn against a project-wide budget of 3,600 requests per hour. On June 12, 2026, the controller that spends a large share of that budget started spending less — hcloud-cloud-controller-manager v1.32.0 introduced an experimental server cache between the node controllers and the API, enabled by default. Six days later, v1.33.0 renamed its tuning knob, extended it to the routes controller, and fixed the exact staleness bug a cache in this position invites.

Here is the bottom line up front: upgrade to v1.33.0 or later (not v1.32.0), keep the default all mode at 10 seconds, and dashboard the new hit-rate metric. The savings are real and scale with fleet size — the worked math below takes a 20-node cluster's node-lookup spend from roughly the entire hourly budget to about a tenth of it — but the cache reopens a question every machine-lifecycle decision previously answered with a live API call: what if the server this entry describes no longer exists?

Everything below is the evidence for that table: where the budget goes, what the cache changes, the not-found bug that shipped with it, and the upgrade checklist.

Why the landlord-API budget bites a CAPH fleet​

Hetzner's Cloud API allows 3,600 requests per hour per project, refilling gradually at about one request per second. That budget is shared by everything in the project holding your API token: CAPH provisioning and reconciling machines, the cloud-controller-manager's node and route controllers, the CSI driver, the load-balancer controller, the autoscaler, and any operator scripts you run. There is no per-controller reservation. Whoever polls hardest eats first, and whoever arrives after the budget is gone gets throttled.

On a Cluster-API-managed fleet, the structural spender is reconciliation itself. CAPH's controllers requeue and re-check machine state on a loop; the CCM's node and node-lifecycle controllers look up the Hetzner Server object behind every Kubernetes Node by ID or name. Steady state is affordable. Churn is not: a rolling upgrade, a MachineHealthCheck remediation wave, or an autoscaler scale-up multiplies Nodes × lookups × re-syncs into a burst that lands all at once — precisely when you most need the API responsive.

This is not theoretical. A CAPH issue from this year describes 31 HCloudMachines across 8 clusters stuck with stale error conditions after a roughly 24-hour rate-limit event: the fleet burned through its landlord budget, throttling cascaded into reconciliation errors, and the errors outlived the throttle. When the API says "slow down," a fleet that reconciles by polling does not gracefully degrade — it queues, retries, and spends even more.

The CCM maintainers have been on an API diet all year, and this post is the second half of that story. The first half: v1.29.0 (December 2025) replaced the routes controller's fixed 30-second network poll with watch-based reconciliation, and Kubernetes 1.36 later added the metric proving it worked — a change we covered in July. The server cache is the same diet applied to the other big spender: per-Node server lookups.

What v1.32.0 actually shipped​

The problem statement in PR #1252 is one sentence: the InstancesV2 controller looks up Servers by ID or name in every method of the controller, generating significant API traffic. The fix is a cache that sits between the node and node-lifecycle controllers and the Hetzner API, serving those lookups from memory.

Three modes, one environment variable:

The window defaults to 10 seconds (HCLOUD_SERVER_CACHE_TTL in v1.32.0; renamed in v1.33.0, see below), and the release notes explicitly discourage values above a minute. Note the deliberate asymmetry: the cache is enabled by default — the maintainers call the implementation safe in practice — while simultaneously experimental, with breaking changes permitted within minor releases. That combination is the honest version of "we are confident enough to turn it on, not confident enough to freeze the interface." The v1.33.0 rename, six days later, is exhibit A for why the caveat exists.

The worked savings: before/after call math​

Model a typical CAPH fleet with stated assumptions so you can re-run them: each Node's reconcile path performs ~3 server lookups across the node and lifecycle controller methods, and each Node re-syncs roughly every 60 seconds during active churn. Uncached cost is multiplicative: N nodes × 3 lookups × 60 syncs per hour.

Two things to read off this table. First, the shape of the win: uncached spend grows with the fleet, cached spend is flat at about one lightweight list call per 10-second window (~360/hr) whenever there is reconcile traffic. At 20 nodes the node lookups alone consume the entire project budget uncached; cached, they cost a tenth of it — leaving headroom for CAPH provisioning, the load-balancer controller, and the autoscaler to do their jobs during the same event.

Second, the sensitivity that keeps the table honest: a 3-node cluster saves only ~180 calls/hr, because the list-all snapshot has a floor cost. The cache is not magic for tiny fleets; it is throttle insurance whose value grows with N. And the assumptions cut both ways — quiet clusters that re-sync rarely spend less uncached than modeled, while a remediation storm with tight requeues spends more. The invariant is the shape (multiplicative → flat), not any single cell.

Mode one sits between the columns: per-server entries avoid the full snapshot fetch but can still cost up to N individual GETs per window. Its value is freshness granularity per server, not maximum savings — and as the next section shows, per-entry caching is also where the sharpest staleness edge lived.

The staleness question: a cached not-found that isn't​

Every entry in this cache is a claim — "server X looks like this" — that decays from the moment it is stored. The default 10-second window bounds that decay tightly. But the first implementation had a failure mode worse than slow decay: in mode one, when a refresh found the server not found in the API, the entry was not directly removed. Expired entries were evicted only after an hour, and only when a refresh had actually found a value — so a deleted server could keep being served from cache for up to ~1 hour after the API truthfully reported it gone.

That is the "cached not-found that isn't": a lookup path answering "here is your server" for a machine that no longer exists. The decisions downstream of these lookups — node lifecycle handling, existence checks during deletion, address and metadata reads while a machine is churning — previously assumed a live API answer. A stale exists is the most dangerous direction for staleness to point, because every consumer proceeds as if the machine is still there.

PR #1271, merged June 17 and shipped in v1.33.0, fixes it at the narrowest point: when the API returns not-found, never return an expired entry. The cache no longer contradicts a live deletion signal. This is also the concrete reason the upgrade target is v1.33.0 and not v1.32.0 — the experimental cache is worth running, but worth running with the version where "deleted means deleted" holds.

Sizing guidance follows directly. Keep the default 10-second max-age unless you have measured a reason: it bounds snapshot staleness to a window shorter than almost any remediation loop's requeue, while still collapsing the multiplicative lookup cost. Reaching for a minute or more trades throttle headroom you can measure against lifecycle races you will discover in production — exactly the trade the maintainers warn against. And keep off documented as your incident escape hatch: if node behavior ever looks cache-suspicious at 3am, one variable restores live answers while you investigate.

The v1.33.0 upgrade checklist​

Four items, in dependency order:

1. Rename the knob — the old one is silently ignored. v1.33.0 renamed HCLOUD_SERVER_CACHE_TTL to HCLOUD_SERVER_CACHE_MAX_AGE, and the old variable is no longer recognized. If you tuned the TTL in v1.32.0 and upgrade without renaming, you do not get an error — you get the 10-second default wearing your tuning's clothes. Grep your manifests for the old name as part of the upgrade; the within-minor breakage the release notes warned about arrived in six days, and it arrived quietly.

2. Know the routes controller joined the cache. v1.33.0 extended the shared server cache to the routes controller, which may override the default with a longer max-age for its lookups. Route data changes less often than node state, so a longer window is defensible — but it means two subsystems now share one cache with different freshness expectations. After upgrading, node lookups and route reconciliation draw from the same snapshot mechanism rather than separate code paths.

3. Migrate the metric, then alert on it. The legacy hcops/AllServersCache.* operation series are gone, superseded by cloud_controller_manager_server_cache_requests_total, partitioned by subsystem (instances_v2, routes), mode, and result (hit/miss). Update dashboards referencing the old series, then add the hit-rate query that tells you the cache is earning its keep:

sum by (subsystem) (rate(cloud_controller_manager_server_cache_requests_total{result="hit"}[5m]))
  /
sum by (subsystem) (rate(cloud_controller_manager_server_cache_requests_total[5m]))

A healthy fleet shows a high hit ratio on instances_v2 during churn — that is the multiplicative lookup cost being absorbed. A hit ratio collapsing toward zero with the mode set to all means lookups are missing the snapshot window (or the cache is effectively off), and your API spend is back to multiplicative. Pair it with an alert on Hetzner 429 responses, which is the landlord telling you the budget math failed.

4. Note the stability signal. v1.34.0 (July 17), v1.35.0 (August 11), v1.36.0 (August 26), and v1.37.0 (September 11) touched load-balancer behavior, zone labels, and IPv6 — nothing about the cache. Three months without a cache change is a reasonable signal that the v1.33.0 shape has settled, even though the experimental label (and its within-minor-breakage permission) still stands. Target v1.37.0 latest; floor v1.33.0.

What to do Monday morning​

In the order you need them:

1. Target v1.33.0 or later — never stop at v1.32.0. The cache without the #1271 not-found fix can serve a deleted server from memory; the fix is the version boundary that matters.
2. Rename HCLOUD_SERVER_CACHE_TTL to HCLOUD_SERVER_CACHE_MAX_AGE in every manifest, and verify the old name is gone. Silent fallback to the default is the upgrade's one real footgun.
3. Keep the defaults (all, 10s) unless measurement says otherwise. Ten seconds of bounded staleness buys a ~10× lookup reduction at 20 nodes; longer windows trade measured savings for unmeasured races.
4. Dashboard the hit-rate metric and alert on 429s. The PromQL above proves the cache is absorbing churn; the 429 alert catches the budget failing across every spender sharing the project token.
5. Budget the API across clusters sharing a project. The 3,600/hr limit is per project, not per cluster. Two 20-node clusters in one project each cost ~360 cached lookup calls/hr — fine — but CAPH provisioning bursts, LB operations, and CSI calls all draw from the same pool. The cache buys headroom; it does not mint budget.

The deeper lesson is about where reliability comes from on owned infrastructure. Nobody provisioned a bigger API quota or bought a second project. A cache, a 10-second window, and a one-line invariant — deleted means deleted — turned the fleet's loudest poller into a flat line. That is what the landlord-API era rewards: not more budget, but fewer questions asked of it.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Fleets like the one above are exactly what it manages: declarative machines on Hetzner, reconciled without burning the landlord's API budget. Star the repo on GitHub or deploy your first app today.

Sources​

• hcloud-cloud-controller-manager v1.32.0 release notes — experimental server cache, modes, defaults (June 12, 2026).
• hcloud-cloud-controller-manager v1.33.0 release notes — TTL→MAX_AGE rename, routes adoption, metric migration, #1271 (June 18, 2026).
• PR #1252: cache server lookups to reduce API calls — InstancesV2 per-method lookup problem statement.
• PR #1271: when API returns not found do not return an expired entry — mode one stale-entry fix.
• Hetzner Cloud API rate limiting — 3,600 requests/hour per project, gradual refill.
• CAPH #1966: machines stuck after rate-limit event — 31 machines across 8 clusters after a ~24h throttle.