Skip to main content

How SNCF Keeps 200+ Kubernetes Clusters in Sync With Zero Drift — and Why the Same Loop Works on Three Hetzner Boxes

8 min readDora NodaDora Noda
Share

France's national railway operator updates every single one of its 200+ Kubernetes clusters every month, and every one of them stays in sync with its declared desired state in between. No snowflake clusters. No manual patch nights. No "we'll get to that cluster next quarter." For a company whose software runs the signaling, ticketing, and scheduling systems behind millions of daily passenger trips, that's not a nice-to-have — it's the same reliability bar as the trains themselves.

SNCF won CNCF's 2026 Top End User Award for exactly this: a fleet spanning 200+ clusters across AWS, Azure, and a private OpenStack cloud, running 70% of the company's 2,000 applications, all reconciled through Cluster API (CAPI) instead of hand-managed one at a time. The mechanism behind "zero drift" isn't exotic — it's the same reconciliation loop Cluster API Provider Hetzner (CAPH) gives any team running Kubernetes on owned hardware. What SNCF proves at national-infrastructure scale is that the loop doesn't bend or degrade as the fleet grows — it just keeps running.

What "Zero Drift" Actually Means Operationally

Every fleet of long-lived servers accumulates drift eventually: a manual kubectl edit here, an emergency patch applied to one node and forgotten, a config that diverged from what's in git six months ago and nobody noticed because nothing broke — yet. Drift is invisible until the day it isn't, usually during an incident, when the runbook assumes a configuration that quietly stopped being true.

SNCF's own prior state is the before-picture. Before this platform, their Kubernetes footprint lived in a restrictive DMZ landing zone with mandatory VM usage — a setup the platform team internally described as "a monster" that was extremely difficult to maintain and operate. Getting out of that took a two-year evaluation of bare-metal Kubernetes options before landing on the current stack.

Cluster API's answer to drift is structural, not procedural: a cluster's entire desired state — control plane version, node count, machine images, everything — lives as Kubernetes-native custom resources (Cluster, KubeadmControlPlane, MachineDeployment) in a management cluster. CAPI's controllers run a continuous reconciliation loop against those objects, the same pattern every Kubernetes controller uses for every other resource: compare declared state to actual state, and correct the difference. There's no separate "drift detection" step to schedule or forget — reconciliation is the mechanism, running constantly, not audited for periodically.

That's the structural reason "zero drift" is achievable at 200+ clusters and not just achievable in a slide deck: nobody has to remember to check. The controller does.

The Stack Underneath the Loop

CAPI is the orchestration layer, not the whole stack. Here's what SNCF runs underneath it:

LayerToolRole
IaaSOpenStack (private cloud) + AWS/AzureCompute, network, and storage primitives across public and private environments
Node OSTalos LinuxAPI-only, immutable OS — no SSH, no shell, no manual node access to drift from
OrchestrationCluster APIDeclarative cluster lifecycle, the reconciliation loop itself
NetworkingCiliumeBPF-based networking and observability across the fleet
PolicyKyvernoPolicy-as-code enforcement for compliance across every cluster

Two details matter beyond the checklist. First, Talos removing shell access doesn't just harden the node — it removes the exact failure mode that causes drift in the first place: someone SSHing in during an incident and leaving a change that never makes it back into git. (We covered Talos's architecture in more depth in a companion post if you want the full picture of what disappears when a node has no shell to escalate from.) Second, CAPI running against a private OpenStack cloud let SNCF turn on node autoscaling in their own datacenters — a capability that used to be exclusive to public cloud provisioning APIs, now available on hardware SNCF owns and controls for data-sovereignty reasons.

The Monthly Update Mechanic, Step by Step

"Updated monthly" sounds like a cadence choice. Mechanically, it's simpler than that — it's what happens when you bump a version field and let the controller do the rest:

  1. The change lands in git. A KubeadmControlPlane or MachineDeployment spec gets its version field bumped (a new Kubernetes patch release, an updated Talos image, a config change) — one commit, reviewed like any other infrastructure change.
  2. CAPI's controllers pick it up. The management cluster's reconciliation loop notices the declared spec no longer matches the running cluster's actual state.
  3. Machines roll, one cluster at a time, across the fleet. Depending on the provider and upgrade strategy, CAPI either replaces Machines (the traditional model) or — as of CAPI v1.12 — updates them in place, without the delete-and-recreate cycle every earlier release required.
  4. Status reports back as Kubernetes objects. No side channel, no separate change-tracking spreadsheet — the cluster's own API server reflects what version it's actually running, queryable the same way you'd query any other resource.

Nothing in that sequence depends on fleet size. It's the same four steps whether the management cluster is reconciling 3 clusters or 200 — which is precisely why SNCF's 70%-of-2,000-applications migration didn't require a proportionally larger operations team to keep every cluster current. The loop doesn't get more expensive to run per cluster as the fleet grows; it gets marginally more expensive to design changes for, because a bad version bump now has 200 clusters to land in instead of one.

The Same Loop on Three Hetzner Boxes

Here's the part that matters if you're not running a national railway: the reconciliation primitive SNCF is using is the exact same one CAPH exposes for a self-hosted fleet on Hetzner. A version bump looks identical at any scale:

yaml
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: prod-control-plane
  namespace: default
spec:
  version: v1.34.2          # bump this...
  replicas: 3
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: HCloudMachineTemplate
      name: prod-control-plane-hcloud
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prod-md-0
spec:
  template:
    spec:
      version: v1.34.2      # ...and this, then `kubectl apply` and walk away

Commit that diff, apply it, and CAPH's controllers reconcile your management cluster's declared state against whatever's actually running on your Hetzner boxes — the identical loop, the identical guarantee that nothing drifts silently between now and the next bump. A single CAPI management cluster comfortably handles on the order of a hundred workload clusters; a 1-3 node self-hosted PaaS fleet isn't testing that ceiling.

What genuinely doesn't carry over is everything around the reconciliation primitive, not the primitive itself:

SNCF (200+ clusters)Small CAPH fleet (1-3 nodes)
Reconciliation mechanismCAPI controllers, declarative specIdentical
Blast radius of a bad bumpStaged rollout, canary clusters first, one bad change can't hit all 200 at onceThe whole fleet is the blast radius — no canary tier to absorb a mistake
Multi-provider abstractionAWS + Azure + private OpenStack behind one APISingle provider (Hetzner); no cross-cloud abstraction to manage
Policy enforcementKyverno gates changes fleet-wide before they reconcileUsually none — a small team applies the manifest and watches

A national-rail fleet earns staged rollouts and policy gates because a bad reconciliation touching 200 clusters at once is a genuinely different failure than a bad reconciliation touching one. A 3-node fleet doesn't need that machinery to get the core benefit — it needs the reconciliation loop itself, which is the part that scales down for free. The honest version of "the same loop works on three Hetzner boxes" is: the mechanism that eliminates drift is scale-invariant; the safety rails you'd want around a mistake are not, and a small team should just test a version bump against a staging cluster before it hits production, since there's no canary tier doing that automatically.

This is the same primitive bex builds its own Hetzner fleet management on — CAPH underneath, git-push deploys on top, so an app's App custom resource reconciles the same way a KubeadmControlPlane does: declare the version, let the controller close the gap. Star the repo on GitHub if you want to run that loop on your own machines.

Reconciliation Doesn't Care How Big the Fleet Is

The headline number from SNCF's case study isn't 200 clusters, or 2,000 applications, or even the CNCF award — it's that a system responsible for national rail infrastructure trusted the exact same declarative reconciliation loop that a three-person team can run on a handful of Hetzner boxes. CAPI didn't need a different architecture to get safe at that scale; SNCF added staged rollouts and policy gates around it, not a replacement for it.

That's the strongest argument for adopting Cluster API on a small self-hosted fleet today, before you ever need the scale: the primitive you're learning now is the same one that just passed the hardest production test CNCF handed out this year. There's no migration to a "real" orchestration model waiting for you at cluster 50 or cluster 200 — you're already running it.


Sources:

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex