Kubernetes 1.36 Ships the Metric That Finally Proves Your Route Controller Isn't Wasting API Calls

Here's a number worth sitting with: for five months, Hetzner's own cloud-controller-manager quietly stopped polling its network API every 30 seconds — and there was no way to prove it. hcloud-cloud-controller-manager v1.29.0 shipped watch-based route reconciliation, on by default, on December 18, 2025. Kubernetes didn't give operators a metric to verify that change actually worked until v1.36, five months later, in May 2026.

That gap is the story. The old behavior was a fixed-interval loop: every 30 seconds, regardless of whether a single node had joined, left, or changed its pod CIDR, the route controller fired a GET /v1/networks/{id} call at the Hetzner API. That's 120 calls an hour, per cluster, forever, just to confirm nothing happened — against a project-wide budget of 3,600 requests/hour. Run ten small clusters in one Hetzner project (a common way to keep cost down on owned hardware instead of paying for ten separate accounts) and idle route polling alone was eating 1,200 calls an hour — a third of your entire budget, spent on a no-op, before your autoscaler, your load-balancer controller, or a single real node provisioning event touched the API.

Kubernetes 1.36's new route_controller_route_sync_total metric is the first tool that lets you actually watch that number instead of trusting the changelog. This post covers what the metric measures, how to check whether your own CCM exposes it, and — the part that actually matters operationally — a concrete alert that uses it to catch a stuck node join before it turns into a silent networking outage.

What route_controller_route_sync_total Actually Counts​

The metric is an alpha counter, added to k8s.io/cloud-provider under KEP-5237, and it increments once every time the route controller runs a reconciliation pass. It exists to answer one question: is your route controller reconciling because something changed, or because a timer went off?

Kubernetes' own worked example, from the official v1.36 blog post, makes the contrast concrete. With the old fixed-interval behavior (still the default upstream), a quiet cluster with zero node changes looks like this:

# 10 minutes elapsed, no node changes
route_controller_route_sync_total 60
 
# 20 minutes elapsed, still no node changes
route_controller_route_sync_total 120

The counter climbs at a constant rate no matter what's happening in the cluster — one sync every 10 seconds, whether or not there's anything to sync. Flip on the CloudControllerManagerWatchBasedRoutesReconciliation feature gate (KEP-5237, alpha since v1.35) and the same quiet cluster looks like this instead:

# 10 minutes elapsed, no node changes
route_controller_route_sync_total 1
 
# 20 minutes elapsed, still no node changes — unchanged
route_controller_route_sync_total 1
 
# a node joins — counter increments
route_controller_route_sync_total 2

Flat until something real happens, plus one mandatory reconcile at a randomized 12–24 hour interval (jittered at controller startup so a whole fleet doesn't reconcile in lockstep). That's the entire mechanism: a fixed poll became an event listener on node adds, node deletes, and changes to a node's .spec.podCIDRs or .status.addresses.

For hcloud-ccm specifically, that generic counter maps to something you can point at directly. Hetzner's own v1.29.0 release notes describe the pre-fix behavior in exactly these terms: "a GET /v1/networks/{id} call is triggered every 30s, even when no changes have occurred." One route-sync equals one Hetzner API call in the old code path. So the metric isn't measuring an abstraction — for a Hetzner-backed fleet, it's a direct, 1:1 proxy for real requests hitting your rate-limited API budget. Watching this counter stay flat is watching your API budget stop leaking.

Check Before You Build On It​

Before wiring up any alert, confirm the metric is actually there. hcloud-ccm v1.31.0 (May 2026) added Kubernetes 1.36 support, but "supports 1.36" and "vendors the exact k8s.io/cloud-provider revision that added this alpha metric" aren't guaranteed to be the same statement — CCM release cadences and upstream library bumps don't always move in lockstep. Don't assume; check:

curl -s localhost:8233/metrics | grep route_controller_route_sync_total

Port 8233 and the /metrics path come straight from hcloud-ccm's own Helm chart (HCLOUD_METRICS_ADDRESS), and it's the same endpoint the chart's optional PodMonitor scrapes. If the grep comes back empty, the alpha metric isn't in your CCM build yet — the watch-based behavior may still be running (hcloud-ccm turned it on by default independent of this metric), you just can't observe it directly yet. Wait for a newer patch release rather than building an alert on a metric name that doesn't exist.

The Alert: Catching a Stuck Node Join Before It's an Outage​

Here's the failure mode this metric actually exists to catch. A new node joins your Cluster API Provider Hetzner (CAPH) fleet. The route controller is supposed to notice the new node, sync its pod CIDR into a Hetzner Cloud Network route, and let Cilium (or whatever CNI you're running with native routing) forward pod traffic to it. If that sync silently fails or hangs — a locked network, a rate-limit backoff, a controller stuck on a stale informer cache — the symptom isn't an error in your logs. It's pods on the new node that can start, pass health checks, and still be completely unreachable from the rest of the cluster, because the route to get traffic there was never created.

This isn't a hypothetical. Issue #115 against hcloud-ccm documents a real instance of exactly this class of failure: the route controller assigned a pod subnet to the wrong node — 10.224.0.0/24 routed to a node whose pods actually lived on 10.224.1.0/24 — with no error surfaced anywhere except downstream connectivity failures. Whether the specific bug is a wrong assignment or a sync that never happens at all, the observable symptom is identical from the outside: a node that looks healthy in kubectl get nodes and is unreachable over the pod network. That's precisely the gap a route-sync counter closes — it gives you a signal that fires before someone notices the outage, instead of a log line you only find after.

Before this metric existed, you had no clean signal to catch that. You'd find out when a request started timing out and someone went spelunking through CCM logs. Now you can correlate two series: kube_node_created (from kube-state-metrics, telling you when a node object was created) against route_controller_route_sync_total (telling you whether the route controller has done anything since).

- alert: CCMRouteSyncStuckOnNewNode
  expr: |
    (time() - kube_node_created) > 300
    and
    max_over_time(rate(route_controller_route_sync_total[2m])[5m:]) == 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "New node {{ $labels.node }} joined 5+ minutes ago with no route-controller activity"

Two threshold choices matter here, and both are deliberate rather than arbitrary. First, the 5-minute window: CAPH node bootstrap plus CNI agent startup typically lands in the 2–3 minute range, so 5 minutes gives real provisioning time to finish before the alert can fire, without waiting through a full incident to notice.

Second, the rule checks for zero sync activity following a node-creation event, not merely "the counter is flat" — because under the watch-based feature gate, a flat counter is the expected steady state 99% of the time by design. Alerting on flatness alone would just be re-implementing the noisy fixed-interval behavior you were trying to get away from. The signal that actually means something is a new node with no corresponding sync activity in the window right after it appeared — a leading indicator for exactly the "route never got created" failure the CCM's own logs won't clearly surface.

Why a Cluster Primitive This Basic Took Until 2026​

It's worth asking why a counter this simple didn't exist years ago. The cloud-controller-manager was split out of the monolithic kube-controller-manager back in Kubernetes 1.11, in 2018. In the eight years since, the components that stayed inside core Kubernetes — the scheduler, the kubelet, kube-controller-manager itself — accumulated deep, well-documented Prometheus metrics as a matter of course, maintained by SIG Scheduling and SIG Node with dedicated attention.

Cloud-provider-specific controllers didn't get that same treatment, because they aren't one component maintained by one team — they're 50-plus independent implementations (AWS, GCP, Azure, Hetzner, OpenStack, and every other cloud-provider-* repo) each maintained by whoever staffs that vendor's Kubernetes integration, with wildly uneven resourcing. A metric has to be proposed and merged into the shared k8s.io/cloud-provider library before any individual CCM can expose it — and nobody prioritized "let operators verify the route controller is behaving" until KEP-5237 made the watch-based rework itself the forcing function. The fix arrived because someone needed to prove their own optimization worked, not because CCM observability was due for attention on its own schedule. It's the last major control-plane subsystem to get baseline reconciliation metrics, and it got there by accident of a different feature's rollout.

What This Means for a Fleet on Owned Hardware​

If you're running Cluster API Provider Hetzner against real machines instead of a managed control plane, this is one more item for the pre-flight list: check that your CCM build exposes the metric, wire up the alert above, and stop trusting a release note's word for what your route controller is actually doing in production. The gap between "the changelog says this is fixed" and "I can see it's fixed on my own cluster" is exactly the kind of thing that's invisible until a node join goes silently wrong at 2 a.m.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, backed by the same Cluster API primitives this post digs into. Star the repo on GitHub or deploy your first app today.

Sources:

• Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager
• Kubernetes v1.35: Watch Based Route Reconciliation in the Cloud Controller Manager
• KEP-5237
• hcloud-cloud-controller-manager v1.29.0 release notes
• hcloud-cloud-controller-manager v1.31.0 release notes
• hcloud-cloud-controller-manager Helm chart values (metrics config)