Dora NodaHere's a failure that only shows up on bare-metal control planes: you run a 3-node KubeadmControlPlane with maxSurge: 0 because there's no spare Hetzner machine sitting idle to surge into, you push a Kubernetes 1.36 upgrade, and for the full duration of the rollout — not a blip, the whole rollout — anywhere from one-third to two-thirds of your control-plane replicas are still on the old version at any given moment, and requests your load balancer routes to one of them for the CRD your platform shipped last release come back 404. Kubernetes v1.36 ships a fix for exactly this, and it's been sitting in alpha since 2023 waiting for the one thing that made it actually useful for CRDs: Mixed Version Proxy graduated to Beta, enabled by default, in the May 2026 v1.36 "Haru" release.
Why maxSurge:0 turns a blip into a window
Cluster API's KubeadmControlPlaneController has two ways to roll a control plane forward when you bump spec.version. The default, maxSurge: 1, provisions a brand-new machine on the new version before it removes an old one — for the few minutes it takes the new node to join, your control plane briefly runs N+1 replicas, and the load balancer has a same-version majority to fall back on. That's the "blip" every hosted-Kubernetes vendor's upgrade docs assume.
maxSurge: 0 doesn't get that luxury. It's the setting Cluster API's own docs call out for exactly this shape of infrastructure: bare-metal, or any resource pool with no headroom to provision an extra control-plane node mid-upgrade. With it, the controller scales down an out-of-date machine first, then scales up the replacement — meaning at every step of a 3-replica rollout, the control plane runs at 2 live replicas, one new and one old, for as long as it takes the replacement to provision, join, and pass health checks. On owned Hetzner hardware, "as long as it takes" is minutes per node, times however many control-plane replicas you run. That's not a blip window measured in seconds while a new pod joins a Deployment — it's the entire rollout, and every request your API-facing load balancer routes during it has better-than-even odds of hitting the old node.
Kubernetes' own version skew policy says this is supposed to be safe: in an HA cluster, all kube-apiserver instances must sit within one minor version of each other, so a v1.35/v1.36 mix mid-upgrade is explicitly a supported configuration, not an edge case you're meant to route around. Supported didn't mean consequence-free, though — until Mixed Version Proxy, "supported" only covered the built-in API groups every apiserver ships with. Anything your platform adds itself was on its own.
What actually breaks without it — and what changes with it
The specific failure is version skew hitting resources the older apiserver has never heard of. Say your PaaS defines an App or Route CRD, or your control plane exposes an aggregated API for its own resource scheduler. During a v1.35→v1.36 rollout, a request for that resource can land on either apiserver behind the load balancer — pure chance, whichever one the LB's round-robin picked. Land on the v1.36 node, no problem. Land on the still-v1.35 node before it's replaced, and previously that node had no way to know the resource existed at all: 404 Not Found, indistinguishable from the resource genuinely not existing. Kubernetes' own alpha-era writeup on this flagged the downstream damage plainly — controllers reading a spurious 404 as "this doesn't exist" have mistakenly garbage-collected live objects and blocked namespace deletions that were waiting on a resource the requesting apiserver simply couldn't see.
Mixed Version Proxy closes that gap by making the apiservers cooperate instead of guessing. Each kube-apiserver maintains a peer-discovery cache of what its siblings can serve; when a request comes in for a resource the local server doesn't recognize, it looks up a capable peer and transparently forwards the request there — tagging it with an x-kubernetes-peer-proxied header so you can see it happened — instead of answering with a 404 it can't back up. If no peer can serve it either, you get a 503 (service temporarily unavailable), which is at least an honest answer instead of a wrong one.
The Beta graduation is what makes this relevant to a platform running its own CRDs, and it's the detail worth sitting with: the alpha implementation (v1.28) determined peer capability via the StorageVersion API, which only covers built-in Kubernetes resources — it had nothing to say about CRDs or aggregated APIs, which is exactly the category a self-hosted PaaS's own control-plane extensions fall into. Beta replaces that mechanism with Aggregated Discovery, where peer apiservers exchange full discovery documents dynamically — CRDs included. That's the actual news here: alpha protected Deployments and ConfigMaps during an upgrade; Beta protects the custom resource type your platform invented.
Before/after, concretely
| Without Mixed Version Proxy | With Mixed Version Proxy (Beta) | |
|---|---|---|
| Request for a built-in resource the local apiserver doesn't yet know | 404 Not Found | Transparently proxied to a peer that knows it |
| Request for your own CRD during a mixed-version window | 404 Not Found (StorageVersion API didn't cover CRDs even in alpha) | Transparently proxied via Aggregated Discovery |
kubectl get --raw /apis mid-upgrade | Shows only what the hit apiserver locally knows — incomplete, version-dependent | Merged, deterministically-sorted discovery document across all peers |
| Failure mode when no peer can serve it | Same wrong 404 | Honest 503, distinguishable from "doesn't exist" |
Duration of exposure on a maxSurge:0 fleet | Full rollout — every replacement step | Same duration, but requests no longer silently fail during it |
The window doesn't get shorter — maxSurge:0 still means minutes-per-node of mixed versions, and nothing about MVP changes that arithmetic. What changes is what happens to a request that lands in the window: instead of a wrong answer masquerading as "not found," it gets routed to whichever peer can actually serve it.
What Beta doesn't hand you for free
This is the part worth testing before you lean on it in production, because none of it is automatic just because the feature gate defaults on.
The proxy has to be told who its peers are, and kubeadm doesn't do this for you. UnknownVersionInteroperabilityProxy being enabled by default in Beta only turns the mechanism on. Actually authenticating peer connections requires --peer-ca-file pointed at your cluster's apiserver CA bundle, plus --proxy-client-cert-file, --proxy-client-key-file, and --requestheader-client-ca-file set correctly on every apiserver instance. Stock kubeadm init/kubeadm upgrade doesn't wire these flags up on its own — if you're rolling your own KubeadmControlPlane template through Cluster API, this is a kubeadmConfigSpec.clusterConfiguration.apiServer.extraArgs entry you have to add yourself, and it's easy to assume "Beta, on by default" means "nothing to configure" when it doesn't.
It's Beta, not GA. The feature only reached Beta this cycle by replacing its entire discovery mechanism (StorageVersion API → Aggregated Discovery) — that's a bigger internal change than a typical alpha-to-beta bump, and it's exactly the kind of change that benefits from a canary run before you trust it on a fleet serving live tenant traffic through its own upgrade. Upgrade one workload cluster first, not the whole fleet at once, and check for the x-kubernetes-peer-proxied header showing up in your apiserver access logs to confirm proxying is actually happening rather than assuming the flags took.
Verify your own CRDs specifically, not just built-ins. The whole reason Beta matters more than alpha is CRD coverage — so the thing to actually check mid-upgrade isn't kubectl get pods (any built-in resource proxies fine even in older versions), it's a request for your platform's own custom resource, issued against the older apiserver specifically, confirming it comes back proxied rather than 404.
The fleet this was built for
None of this is a hyperscaler's problem in the same way. A managed control plane's operator can always provision a throwaway extra node to surge through an upgrade — that's the entire reason maxSurge: 1 is the default and why most upgrade guidance assumes it's available. It's specifically the fleets that can't do that — bare metal, capacity-capped resource pools, anywhere maxSurge: 0 is the honest setting rather than a workaround — where the exposure window is long enough, and the custom-resource surface area large enough, for this to matter operationally instead of theoretically.
That's the shape of infrastructure Bex.co runs on: a self-hosted, AI-native Render alternative built on Cluster API–managed fleets on owned Hetzner machines, not a hyperscaler's elastic control plane. A rolling upgrade on owned hardware was always going to mean a real mixed-version window instead of a blip masked by spare capacity — Mixed Version Proxy Beta is the piece of upstream Kubernetes that makes that window survivable for the CRDs a platform like this defines itself, provided the peer-auth flags actually get set.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
Sources
- Kubernetes v1.36: Mixed Version Proxy Graduates to Beta
- Mixed Version Proxy — Kubernetes docs
- Kubernetes 1.28: A New (alpha) Mechanism For Safer Cluster Upgrades
- Kubernetes v1.36: "Haru" release announcement
- Kubernetes Version Skew Policy
- Mixed Version Proxy (Unknown Version Interoperability Proxy) — KEP-4020
- Upgrading management and workload clusters — The Cluster API Book
- KubeadmControlPlane rollout strategy and MaxSurge behavior discussion
