Blue-Green Deploys on Cluster API: Flipping a Tenant's Traffic Between Two MachineDeployments Without Touching the Load Balancer

Every git-push PaaS advertises "zero-downtime deploys," and almost all of them mean the same thing: a rolling update. New pods come up, old pods drain, and for a window that can run anywhere from a few seconds to a few minutes, both versions answer requests side by side. That's fine for a stateless API absorbing a routine deploy. It's not what a tenant means when they ask for an instant, fully-reversible cutover — one command forward, one command back, no in-between state where two code versions are both live. On a Cluster API-managed fleet, that's a different primitive: two parallel MachineDeployments and a Service whose selector you flip. Here's what actually has to be built to make that flip safe.

The Rolling-Update Ceiling​

A MachineDeployment is Cluster API's analogue to a core Kubernetes Deployment: you don't touch MachineSets directly, you edit the MachineDeployment spec and its controller reconciles the difference by rolling out two MachineSets — the old one and the new one — scaling the new set up and the old set down according to maxSurge and maxUnavailable (both default to 1, expressible as an absolute count or a percentage). That's the same shape as a Deployment's rolling update, just one layer down at the machine level instead of the pod level.

The mechanism is genuinely good at what it's for: gradual, low-risk replacement where a few nodes running the old version and a few running the new version simultaneously is an acceptable, temporary state. It is structurally incapable of two things a tenant sometimes explicitly wants:

• An atomic cutover. There's no moment where 100% of traffic moves from old to new in one step — surge and drain happen incrementally, machine by machine, node by node.
• An instant, full rollback. Reversing a rolling update means running the rollout in reverse: scale the old MachineSet back up, scale the new one back down. That's the same gradual process, just pointed the other way — not a single command that restores the previous state in one action.

For a tenant running a stateful migration, a major schema change gated behind a flag, or just a deploy they want to be able to abort in one motion at 2am, "gradual and reversible-but-slow" isn't the same guarantee as "atomic and instantly-reversible." That's the gap blue-green closes.

The Blue-Green Primitive on Cluster API​

Blue-green doesn't replace the rolling MachineDeployment — it wraps two of them. Instead of one MachineDeployment mutating in place, a tenant's deploy pipeline provisions two, labeled by color, each fully scaled and fully healthy on its own:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: tenant-api-blue
  labels:
    app: tenant-api
    color: blue
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: tenant-api
        color: blue

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: tenant-api-green
  labels:
    app: tenant-api
    color: green
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: tenant-api
        color: green

One Service sits in front of both, and its selector — not a load balancer's config, not a DNS record, not an ingress rule — decides which color is live:

apiVersion: v1
kind: Service
metadata:
  name: tenant-api
spec:
  selector:
    app: tenant-api
    color: blue
  ports:
    - port: 443
      targetPort: 8080

A deploy means: build the new version, roll it onto the idle color's MachineDeployment (green, if blue is currently live), let it come up fully and pass health checks while it receives zero production traffic, then cut over with a single field change:

kubectl patch service tenant-api -p '{"spec":{"selector":{"color":"green"}}}'

That one kubectl patch — or the equivalent field update through whatever control plane sits in front of it — is the entire cutover. No load balancer restart, because the load balancer (however it's implemented — a cloud LB, an ingress controller, kube-proxy's iptables/IPVS rules) is just following the Service's Endpoints/EndpointSlice objects, which update the instant the selector changes. The mechanism is identical to the pattern Argo Rollouts formalizes with its activeService/previewService fields — an active Service pointed at the stable revision, a preview Service pointed at the new one for pre-cutover testing, and a promotion step that's really just the active Service's selector being rewritten to match the previewed revision. Cluster API doesn't need Argo Rollouts to get this; the primitive is two label-matched compute pools and one Service object, which is infrastructure a Cluster-API-backed platform already owns at the MachineDeployment layer.

Gating the Flip​

The value of blue-green collapses if "idle color is healthy" is asserted instead of verified. Before the Service selector moves, the platform needs two gates, not one:

1. Readiness at the pod level. Standard Kubernetes readiness probes on the new color's pods — the MachineDeployment's scale-up already blocks nodes from being marked Ready until they join the cluster, but pod-level readiness on the workload itself is a separate, additional check that has to pass before the color is eligible for cutover at all.
2. A pre-promotion check against the preview path. This is the piece a plain rolling update never needs and blue-green can't skip: hit the idle color directly — bypassing the Service, through a second, unadvertised preview Service or a direct pod IP — with a smoke test or a short synthetic-traffic burst, and gate promotion on that succeeding. This is exactly what Argo Rollouts' prePromotionAnalysis step formalizes: run analysis against the preview stack, and only flip activeService if it passes.

Skipping step 2 and cutting over on pod-readiness alone is the single most common way teams turn "blue-green" into "instant outage" — a pod can be Ready (process started, port open, health endpoint returns 200) while still being wrong in a way only real traffic surfaces, like a bad database connection string that only fails on the first actual query.

The Part Everyone Hand-Waves: In-Flight Requests​

Here's the detail that separates a working blue-green implementation from a demo that only looks like one: the Service selector flip is not instantaneous at the data-plane level, and it doesn't wait for in-flight requests to finish on the color being cut away from.

When a Service's selector changes, Kubernetes' EndpointSlice controller has to notice the change, rebuild the slice's endpoint list, write it, and propagate it out to every kube-proxy watching that Service before traffic actually stops routing to the old color's pods. That propagation isn't instant and isn't guaranteed to complete uniformly — different kube-proxy instances on different nodes pick up the update at slightly different times, which means for a short window after the kubectl patch returns, some fraction of new connections can still land on the color you just declared inactive. In the reverse direction — old pods terminating — kube-proxy can remove an endpoint from rotation before the pod itself has finished draining its in-flight requests, which is the same class of race that causes dropped connections during any pod termination, blue-green or not.

The concrete mitigation, not a hand-wave: don't tear down or scale down the outgoing color the instant the selector flips. Keep it running, and put a short preStop sleep on its pods (5–15 seconds is the range that covers most observed kube-proxy propagation windows) so the container keeps serving any request that was already routed to it before its endpoint fully drops out of rotation:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]

That sleep isn't padding — it's the mechanism that turns "the selector changed" into "the selector change actually finished propagating everywhere before anything gets killed." Skip it, and the failure mode isn't hypothetical: a handful of in-flight requests during every single cutover get connection-reset, which is exactly the outcome blue-green was supposed to eliminate.

Rollback and the Double-Footprint Cost​

The payoff for surviving the propagation window correctly is that rollback becomes genuinely trivial: flip the selector back.

kubectl patch service tenant-api -p '{"spec":{"selector":{"color":"blue"}}}'

That only works, though, if the outgoing color's MachineDeployment is still scaled up and warm — not torn down the moment the new color goes live. The honest tradeoff a blue-green deploy carries that a rolling update never does: for the length of the rollback window a platform decides to hold (minutes for a routine deploy, longer for a tenant doing a risky migration), both colors are fully provisioned and billed simultaneously. A tenant running three CX22 nodes (2 vCPU / 4GB, roughly €4.35/month each on Hetzner) at steady state pays for six nodes during that window — a real, if temporary, doubling of the fleet's node-hours for that tenant's workload, not a rounding error to wave off.

The design decision a Cluster-API-backed platform actually has to make explicit is how long to hold that doubled footprint before scaling the outgoing MachineDeployment to zero: too short, and a bug that only shows up after a few minutes of real traffic has already lost its instant-rollback window; too long, and every deploy is paying for idle standby capacity that a rolling update never provisions in the first place. A tenant-configurable rollback TTL — scale the outgoing color to zero automatically N minutes after a successful cutover, unless rollback is triggered first — is the concrete knob that makes this tradeoff a choice instead of a default nobody set on purpose.

Where This Sits Next to Render, Railway, and Fly.io​

None of the three most-compared managed PaaS platforms make blue-green the default, and that's informative rather than a gap. Render and Railway both run rolling deploys as their standard mechanism — the same gradual, surge-and-drain model MachineDeployments implement natively — which is the right default for the common case and the reason schema changes on either platform have to stay migration-safe across old and new code running at once. Fly.io is the outlier: it exposes an explicit blue-green strategy alongside rolling as a deploy-time choice, acknowledging that some deploys want the atomic-cutover guarantee and some don't, rather than picking one strategy for every deploy on the platform.

That's the right shape for a Cluster-API-backed platform to copy: rolling as the default deploy path (cheap, simple, correct for routine changes), blue-green as an opt-in strategy a tenant requests for a specific deploy — a risky migration, a major version bump, anything where "instant, verified, reversible" is worth temporarily doubling the node count. The primitive underneath either choice is the same MachineDeployment the platform already manages; blue-green just adds a second one, a Service selector instead of an in-place mutation, and the pre-promotion gate and drain-aware teardown that make the flip actually safe instead of merely fast.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.