Kubernetes v1.36 Route Sync Metric: Catch Your CCM Wasting API Calls on Hetzner

Your Cloud Controller Manager calls your infrastructure provider's API every 10 seconds to reconcile routes — even when nothing changed. Kubernetes v1.36 finally gives you a number that proves it: route_controller_route_sync_total, an alpha counter that ticks once per route sync. On an idle cluster with the default settings, expect about 60 ticks every 10 minutes; with v1.35's watch-based reconciliation gate enabled, expect about one. If you run your own CCM on Hetzner hardware instead of a managed control plane, that counter is the cheapest A/B test and the earliest stuck-reconciler tripwire you have.

What the route controller actually does on Hetzner​

On Hetzner Cloud, the route controller inside the hcloud Cloud Controller Manager (HCCM) programs native routes into your Private Network — one per node, mapping that node's pod CIDR (a /24 by default, carved out of your cluster CIDR) to the node as gateway. It is enabled by default whenever networking is enabled and you attach a Private Network, and you opt out with HCLOUD_NETWORK_ROUTES_ENABLED=false. Two constraints shape everything else: Private Networks are IPv4-only, so the controller manages IPv4 routes exclusively, and each Hetzner network caps out at 100 attached servers, 100 routes, and 50 subnets.

The division of labor is worth stating plainly:

Cilium's native routing mode exists precisely to ride on these provider-programmed routes instead of wrapping every packet in VXLAN. That makes the route table a load-bearing part of your data plane — and the reconciler that maintains it worth watching.

Here is the part hyperscaler users never think about. On EKS, GKE, or AKS, the CCM is the vendor's problem: you cannot see its flags, its logs, or its metrics endpoint. On owned Hetzner machines with HCCM, you run that binary yourself — you choose the feature gates, you scrape its /metrics, and every wasted API poll spends your own provider quota. That is also why an upstream counter beats any vendor dashboard here: the Hetzner Console shows you routes-as-state, a snapshot of what exists right now. Only this counter shows you reconcile behavior over time — how often the loop runs, whether it runs when nothing changed, and whether it stopped running at all.

What the counter measures — and what it doesn't​

Provenance first, so you can cite it: the metric is route_controller_route_sync_total, introduced as alpha in Kubernetes v1.36 in the shared k8s.io/cloud-provider route controller, announced by Lukas Metzner (Hetzner) in a Kubernetes blog post republished May 15, 2026. It exists to validate the CloudControllerManagerWatchBasedRoutesReconciliation feature gate from v1.35 (KEP-5237).

Semantics, precisely: the counter increments once per reconcile pass that syncs routes with the cloud provider — including passes where nothing needed changing. It counts sync attempts, not routes. It does not tell you how many routes exist, whether they are correct, or whether the last API call succeeded. Treat it as a tachometer for the reconcile loop, not an odometer for your route table.

The baseline math falls straight out of the defaults. The fixed-interval loop runs every --route-reconciliation-period, which defaults to 10 seconds — six passes a minute, 60 per 10 minutes, 8,640 per day, on a cluster where zero nodes changed. Every one of those passes is at least one round trip to the Hetzner Cloud API, spending quota that autoscaling, load-balancer updates, and your own automation also draw from. That steady 60-per-10-minutes tick is the number the watch-based gate is trying to kill.

The A/B test: flipping the watch-based gate​

Up to v1.34, the route controller only knew the fixed-interval full reconcile: look at every route in the cluster CIDR and every node, then create, update, or delete to match — on every tick, changed or not. The v1.35 gate switches that to watch-based reconciliation: the loop fires when nodes are actually added, removed, or updated, plus a low-frequency periodic sweep that still cleans up stale routes. The reconcile logic itself is unchanged; only the trigger changes.

Enabling it on HCCM is one Helm value, since the chart passes feature gates through to the binary:

args:
  feature-gates: "CloudControllerManagerWatchBasedRoutesReconciliation=true"

The A/B test the metric was built for is a single query, run before and after the flip:

rate(route_controller_route_sync_total[10m])

On an idle cluster, the published example shows the shape of the win: gate off, the counter climbs 60 → 120 over 20 changeless minutes; gate on, it sits at 1 and only moves when a node joins. But idle is the flattering case, so run the second row of the test under churn — the case a Cluster API fleet actually lives in. Roll N nodes (a MachineDeployment rollout, a zone evacuation): with the gate off, the rate stays pinned near 6-per-minute background regardless of what you do; with the gate on, expect roughly N increments for N node events plus the quiet periodic baseline. If the gate-on rate under churn looks identical to the gate-off rate, the gate is not engaged — check the flag actually reached the binary. And set your storm threshold from this same data: sustained rate well above the idle baseline, with no matching Node add/remove/update events to explain it, means either real churn you should know about or an update loop stuck retrying — both worth paging on, which is the next section.

Wiring it into Prometheus alerts​

Two alerts cover the failure modes that matter. Both are regime-dependent — write the variant that matches your gate setting, because watch-based reconciliation deliberately idles near zero.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ccm-route-reconciler
spec:
  groups:
    - name: ccm-routes
      rules:
        # Fixed-interval regime (gate off): the loop must tick every ~10s.
        # Silence under zero churn is impossible here, so flatline = wedged.
        - alert: CCMRouteReconcilerStalled
          expr: rate(route_controller_route_sync_total[5m]) == 0
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "CCM route reconciler has not synced for 10 minutes"
        # Either regime: sustained syncs far above baseline with no node
        # churn to explain them = stuck update loop or runaway churn.
        - alert: CCMRouteSyncStorm
          expr: rate(route_controller_route_sync_total[5m]) > 0.5
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "CCM route sync rate 5x above the 10s-period baseline"

For the watch-based regime, the stall alert needs a companion: near-zero is normal at rest, so only fire when Nodes changed but the counter didn't move (changes() over your node count joined against a flat counter — kube-state-metrics gives you the node side). And every regime needs the correctness companion the counter alone cannot provide: compare desired routes (Ready nodes × one /24 each) against actual routes in the Hetzner network (hcloud network describe, or an exporter if you run one). Desired-vs-actual mismatch is the investigate signal; the counter tells you whether the reconciler is even trying.

The blackhole scenario this pair catches is concrete. A node leaves, its route goes stale, and the departed node's pod CIDR still points at a dead gateway. Cilium keeps delivering pod traffic into that route because the network says it is valid. The counter flatlines while the route count disagrees with the node count — either signal alone is ambiguous, but together they say "the reconciler stopped converging" before your tenants' cross-node traffic says it louder.

One operational note for fleets: run one cluster per Hetzner region and you get one counter per cluster for free, since each CCM scrapes independently. Dashboard the per-cluster rates side by side; a region whose idle baseline diverges from its siblings is telling you its gate flags, CCM version, or API latency differ — exactly the drift a fleet view exists to catch.

Limits and honest caveats​

Alpha means alpha: the metric name, labels, and even existence can change before beta, so pin your dashboards and alerts to the CCM version you validated and re-check on every upgrade. The counter cannot prove routes are correct — pair it with the desired-vs-actual check above, always. And two Hetzner-specific ceilings deserve explicit statements. First, a scope correction to the "multi-region" framing: a Hetzner Private Network spans locations within a single network zone (eu-central covers Falkenstein, Nuremberg, and Helsinki, for example), not across regions — true multi-region means separate clusters with separate networks, each with its own counter. Second, the 100-routes-per-network cap bounds how large a natively-routed cluster can grow; route hygiene stops being cosmetic well before you hit it, because every stale route is a slot a real node cannot use.

The cheapest observability you will add this quarter​

The arc here is short: a Hetzner engineer felt the pain of a reconcile loop polling a rate-limited API every 10 seconds, upstreamed the watch-based fix in v1.35, then upstreamed the counter that proves the fix works in v1.36. If you run HCCM, your rollout is three steps — scrape the metric, record your gate-off baseline, flip the gate and watch the rate collapse — and your quota savings scale with every idle hour your clusters sit stable. When the gate eventually defaults on, the same counter graduates from A/B instrument to drift detector: any return to the old cadence means something regressed. Give feedback upstream via #sig-cloud-provider or KEP-5237 if your numbers disagree with the published ones; that is what alpha metrics are for.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.