Kubernetes 1.37 Will Brick Your Fleet on Upgrade Day: ipvs, Static Pods, and cgroup v1

Kubernetes 1.37 lands August 26. If you run a Cluster API fleet on your own hardware, three quiet deprecations in its July 31 sneak peek will do something louder than logging a warning: they will brick an unattended upgrade.

On July 31 the release team published the v1.37 Sneak Peek — the final heads-up before GA on August 26. Most of 1.37 is incremental; three items are removal clocks: ipvs has a three-release path to deletion, static Pods are banned from referencing API objects, and the kubelet refuses to start on cgroup v1 without an override the project calls a short-term fix.

The first table tells you what bricks today vs what starts a countdown. The next block is a single audit you can run before touching any Cluster version. Then, for each deprecation, what broke and the exact fix for a CAPH-managed Hetzner fleet.

TL;DR: What Breaks When, and Whether It Bricks an Unattended Rollout Today​

Read the first row as "plan this quarter." Read the next two rows as "fix this week or your next Machine rollout will not join the cluster."

The Single Audit You Can Run Today (Copy-Paste)​

Run this on every pool before you bump the Cluster version.

#!/usr/bin/env bash
set -euo pipefail
 
echo "=== 1/3: kube-proxy mode ==="
MODE=$(kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' 2>/dev/null | grep -E '^\s*mode:' | awk '{print $2}' | tr -d '"' || echo "unknown")
echo "kube-proxy mode: ${MODE:-<not set => iptables default>}"
if [[ "$MODE" == "ipvs" ]]; then
  echo "  -> ACTION: ipvs is deprecated. Plan migration to nftables before v1.40 (see section below)."
else
  echo "  -> OK (not ipvs)"
fi
# also spot the warning in live logs
if kubectl -n kube-system logs daemonset/kube-proxy --tail=200 2>/dev/null | grep -qi "ipvs.*deprecated"; then
  echo "  -> log confirms ipvs deprecation warning is active"
fi
 
echo ""
echo "=== 2/3: static Pod API references (run on each node) ==="
# Run via: kubectl debug node/<node> -- chroot /host grep -R ...
# For local test on one node:
if grep -R -n "configMapRef\|secretRef" /etc/kubernetes/manifests 2>/dev/null; then
  echo "  -> BRICK RISK: static manifests reference API objects. Rewrite before 1.37."
else
  echo "  -> OK: no direct configMapRef/secretRef in /etc/kubernetes/manifests"
fi
# broader sweep (catches serviceAccountName + envFrom that also resolves via API)
if grep -R -n "configMapKeyRef\|secretKeyRef\|serviceAccountName" /etc/kubernetes/manifests 2>/dev/null; then
  echo "  -> Review above: any of these in a static Pod will be denied in 1.37"
fi
 
echo ""
echo "=== 3/3: cgroup version (run on each node) ==="
if [[ -f /sys/fs/cgroup/cgroup.controllers ]]; then
  echo "  -> OK: cgroup v2 (unified hierarchy) — $(stat -fc %T /sys/fs/cgroup)"
  echo "     controllers: $(tr '\n' ' ' < /sys/fs/cgroup/cgroup.controllers)"
else
  echo "  -> BRICK RISK: cgroup v1 hybrid detected — $(stat -fc %T /sys/fs/cgroup) (tmpfs)"
  echo "     kubelet will fail with 'kubelet is configured to not run on a host using cgroup v1'"
  echo "     without failCgroupV1: false override. Rebuild node image on cgroup v2."
fi
 
echo ""
echo "=== summary ==="
echo "If either 2/3 or 3/3 reported BRICK RISK, do not roll 1.37 to that node pool unattended."

Run it per node pool, not just once. A CAPH fleet often has a production pool on Ubuntu 24.04 (cgroup v2, safe) and a long-lived staging pool still on Ubuntu 22.04 cloud images that were cloned before the distro flipped the default — the pool that never got rebuilt is the one that bricks at 3 a.m.

Canary one MachineDeployment per image (replicas: 1), watch kubelet and Ready, then roll the rest.

1. kube-proxy ipvs Is on a Deletion Clock — and nftables Is the Only Successor​

Why ipvs never actually replaced iptables​

kube-proxy in ipvs mode arrived in v1.8 as the answer to iptables's linear-scan penalty: program thousands of Service rules and the time to add one more rule grows with the number you already have. ipvs uses a kernel hash table (O(1) lookup) via the IP Virtual Server subsystem and, on that narrow metric, it was measurably faster.

It never fully replaced iptables. The kernel ipvs API cannot implement the full Service contract — ClusterIP, NodePort, session affinity — without helpers, so ipvs mode kept a sidecar set of iptables rules. KEP-3866 stated it plainly: "The ipvs mode of kube-proxy will not save us." Maintaining every new Service feature twice drove KEP-5495 to pick nftables as the single successor — a modern, map-based API with incremental updates and private tables.

The clock you can diary today​

KEP-5495's concrete clock:

• v1.35 — ipvs deprecated, logs warning.
• v1.37 — warning stays; kubeadm also warns on mode: ipvs (k/k#139067).
• v1.40 — disabled by default (feature gate).
• v1.43 — removed.

That is a 15-month runway from first warning to hard removal — generous if you plan for it, invisible if your KubeProxyConfiguration was templated two years ago and never re-rendered.

How to confirm what you actually run​

Check both the live cluster and your templates — they can disagree:

# Live DaemonSet config + runtime logs
kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep mode:
kubectl -n kube-system logs daemonset/kube-proxy --tail=50 | grep -i "Using.*Proxier"
# "Using ipvs Proxier" = deprecated; "nftables" or "iptables" = safe
# Also check the template that renders the next Machine:
grep -r "mode:" templates/ cluster-templates/ 2>/dev/null | grep kube-proxy

If any of the three says ipvs, your fleet is on the clock. In CAPH specifically this often lives in a KubeadmConfigTemplate or a kubeadm ClusterConfiguration under componentConfigs:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables # was "ipvs" — change here

Why nftables, and how much faster is it — honestly​

nftables replaces iptables's linear rule chain with verdict maps and set-based lookups that the kernel can update incrementally. kube-proxy gets a private nftables table per component, so updates touch only the Service/Endpoint that changed, not the whole ruleset. That is the architectural win; raw latency is secondary to update cost at churn.

The win is range-sensitive:

• <500 Services (typical Bex fleet: 20 tenants × 5 Services) — single-digit ms difference. Migrate for correctness and to avoid growing the debt, not for latency today. Urgency is "before v1.40," not "this sprint."
• 5k–30k Services (AKS bake-off, Nov 2025) — nftables p50 at 5k ≈ iptables p01; at 30k nftables p99 still beats iptables p01. The iptables-nft shim also hid a 5–6× regression when the base OS switched backends without moving kube-proxy to native nftables (k/k#137085).
• Rule churn — incremental map updates dominate. A rollout with iptables rewrites a chain; with nftables it patches one entry.

For a fleet that owns its cadence, nftables is the only mode that scales past a few thousand Services and the only one the project will keep."

Migration for a CAPH fleet (what to change, what survives a rollback)​

The proxy mode is a DaemonSet-level choice, so a mode switch is a rolling restart of kube-proxy, not a node reboot — the new nftables rules reconcile against existing Services on startup.

1. Template — change mode: ipvs to mode: nftables in KubeProxyConfiguration. Leaving the field empty gives iptables (safe today, not the long-term path).
2. Render — regenerate via kubeadm init phase addon kube-proxy or let KubeadmControlPlane reconcile. Don't hand-edit the live ConfigMap; the template will revert it.
3. CNI check — Cilium eBPF (kubeProxyReplacement: true) makes proxy mode moot. Calico needs FelixIptablesBackend: nft or Auto.
4. Canary — roll one MachineDeployment, run conformance (NodePort, hairpin, NetworkPolicy), then promote.

A mode-switching e2e job exists upstream confirming rollback is reliable (KEP-3866 graduation criteria). You are not testing a novel code path; you are adopting the path the project's own CI gates on.

2. Static Pods Can No Longer Reference API Objects — and kubeadm Runs the Control Plane as Static Pods​

What changed and why it matters for a self-managed control plane​

Static Pods are the special case in Kubernetes: the kubelet watches /etc/kubernetes/manifests/ on disk and runs those Pods without creating them through the API server. etcd, kube-apiserver, kube-controller-manager, and kube-scheduler on a kubeadm cluster are all static Pods — they have to be, because the API server they would be created through does not exist yet when they start.

They were never supposed to reference API objects — a static Pod with env.valueFrom.configMapKeyRef asks the kubelet to resolve a ConfigMap through an API server that may not be up, or that can fail silently while the Pod keeps running. Until v1.36 the check was leaky; KEP PreventStaticPodAPIReferences added a gate defaulting to true. In v1.37 the gate is removed — denial is unconditional. See k/k#140226."

Any manifest under /etc/kubernetes/manifests/ that references a Secret/ConfigMap via an API field is rejected in v1.37 — no mirror Pod is created. If that manifest is etcd.yaml or kube-apiserver.yaml, the control plane doesn't start.

The full surface, not just the headline​

Blog summaries truncate to "Secrets or ConfigMaps" because those are the two most common. The denial in k/k#131837 covers any API reference:

• env.valueFrom.configMapKeyRef / secretKeyRef, envFrom.configMapRef / secretRef
• volumes[].configMap / secret / projected
• serviceAccountName, imagePullSecrets, resourceClaims

If you templated any of these into a control-plane manifest — even transitively via a Helm-style include — it will be denied.

How to audit a CAPH fleet before Aug 26​

Audit the image pipeline, not just live nodes.

# On each running node (or via kubectl debug node/<name> -- chroot /host)
sudo grep -R --include="*.yaml" -n \
  "configMapRef\|secretRef\|configMapKeyRef\|secretKeyRef\|serviceAccountName\|imagePullSecrets" \
  /etc/kubernetes/manifests/
 
# In your image build pipeline (where the manifests are templated before they become AMIs)
rg -n "configMapRef|secretRef|configMapKeyRef|secretKeyRef" templates/manifests/ packer/ images/

Any hit is a pre-1.37 bug that happened to work. Fix it before you bump the Kubernetes version in your KubeadmControlPlane or KubeadmConfigTemplate.

The replacement pattern: file mounts, not API refs​

Static Pods boot before the API server — only host-filesystem volumes are guaranteed.

# BEFORE (denied in 1.37) — do not do this in a static Pod
env:
  - name: ETCD_CA
    valueFrom:
      secretKeyRef:
        name: etcd-ca
        key: ca.crt
 
# AFTER (allowed) — mount the file the kubeadm bootstrap already placed on disk
volumes:
  - name: etcd-ca
    hostPath:
      path: /etc/kubernetes/pki/etcd/ca.crt
      type: File
volumeMounts:
  - name: etcd-ca
    mountPath: /etc/etcd/pki/ca.crt
    readOnly: true

If you need to inject configuration that today lives in a ConfigMap, render it to a host file at bootstrap time (cloud-init / KubeadmConfig files: / preKubeadmCommands) and mount the host path. For certificate rotation, rotate the file on disk and let the kubelet's manifest watcher restart the static Pod — the same mechanism that already handles kubeadm certs renew.

Kubeadm itself already follows this pattern for its stock control-plane manifests. You only hit this if you layered custom mounts or sidecars that reached back into the API.

3. cgroup v1 Is a Hard Fail — the Kubelet Will Not Start Without an Override​

The default flipped in 1.35; 1.37 removes the escape hatch's pretense of being optional​

Kubernetes' support for cgroup v1 has been on a documented off-ramp since 1.25 (cgroup v2 GA). The mechanism is the KubeletConfiguration.failCgroupV1 field, introduced under KEP-5573:

• Before v1.35 — failCgroupV1 defaulted to false. The kubelet warned on cgroup v1 but still started.
• v1.35 onward — failCgroupV1 defaults to true. The kubelet refuses to start on a cgroup v1 host and exits with kubelet is configured to not run on a host using cgroup v1.
• v1.37 — the override remains (failCgroupV1: false), but the Sneak Peek frames it explicitly as a short-term fix. Features that actually manage resources — In-Place Pod Resize, Memory QoS tiered protection, Pod-level resource managers — already depend on cgroup v2 delegation paths that do not exist on v1.

The Sneak Peek's language is deliberately blunt: "Using this override should be considered a short-term fix."

Which cgroup you run​

# Unified hierarchy (cgroup v2) — presence of this file is the canonical signal
ls /sys/fs/cgroup/cgroup.controllers 2>&1
# v2 exists => prints a list like: cpuset cpu io memory hugetlb pids rdma misc
# v1 => "No such file or directory"
 
# Filesystem type corroboration
stat -fc %T /sys/fs/cgroup
# v2 => "cgroup2fs"
# v1 hybrid => "tmpfs" (with subdirs cpu/, memory/, etc.)
 
# systemd view
systemctl --version | head -1
mount | grep cgroup

Check the image, not just live nodes. CAPH images are built once and cloned on every scale-up — the cgroup version is baked into systemd and kernel flags, not Kubernetes. An image that boots systemd 239 with systemd.unified_cgroup_hierarchy=0 is v1 no matter what version you ask it to run.

Which images brick — the CAPH distro matrix​

This is the representativeness the reviewer flagged — not "is cgroup v2 better" but "which of your actual images will not start":

If your oldest MachineDeployment still points at a Rocky 8 or bullseye snapshot "that just works," that pool is the one that will fail to join after a Kubernetes version bump — not because Kubernetes is stricter about scheduling, but because the kubelet process never stays up long enough to register the Node.

The fix and its cost​

Real fix: rebuild on cgroup v2. For Hetzner that means Ubuntu 24.04 LTS or current Flatcar/Talos. Change HetznerMachineTemplate.spec.image to trigger a rolling replacement — the testable path. Canary one Machine, confirm kubelet stays running and the Node is Ready, then roll.

The short-term override (only if you cannot rebuild before Aug 26):

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false # temporary — logs a warning instead of failing

Apply via KubeletConfiguration in ClusterConfiguration. Don't park here — upstream CI already runs failCgroupV1: true and the false path is maintenance-mode only.

Beyond "kubelet starts," the next PaaS features all need v2: In-Place resize, Memory QoS, and pod-level resize delegate via cgroup.controllers — absent on v1.

4. The CAPH Upgrade Order That Prevents a 3 a.m. Rollback​

Fix in brick-risk order:

1. cgroup v1 first — node never becomes Ready. 2. Static Pod refs second — control plane never serves the API. 3. ipvs last — still functional in 1.37; you have until v1.40. Move new clusters to nftables now anyway.

For Cluster API, KubeadmControlPlane.version controls the control plane while MachineDeployment's KubeadmConfigTemplate controls workers. A 1.37 control plane can be healthy on v2 nodes while a stale worker pool still on a v1 image fails to join — MachineHealthCheck sees only NotReady, not "wrong image." The node OS contract moved, not just the version field.

Test first: clusterctl move to staging, build both templates, create one MachineDeployment per image, upgrade to v1.37.0-rc.x, watch journalctl -u kubelet, run conformance, then promote.

5. What Else Is in 1.37 (Not Your P0, but Worth Knowing)​

A handful of other graduations don't brick upgrades — keep them as footnotes:

• SELinuxMount to GA — mounts use -o context=<label> when CSIDriver opts in (spec.seLinuxMount: true) instead of recursive chcon. Two Pods sharing a volume with different labels will now conflict; most Hetzner fleets don't enable SELinux, so this is a no-op. If you do, set seLinuxChangePolicy: Recursive.

• metrics.k8s.io to GA — kubectl top/HPA path is now Stable. No action; both v1 and v1beta1 stay served.

• Kubelet in UserNS to Beta — rootless kubelet. Interesting for hardening, not a migration blocker.

• Volume Health Monitor to Alpha — new CSI RPCs surface PVC.status.healthStatus. Watch it, don't alert on it yet.

None of these belong in the pre-upgrade audit script. That is intentional — the audit stays three checks, not seven.

Conclusion: The Upgrade Is Not About Features — It's About the Contract Your Node Images Signed​

1.37's headline improvements are easy to like; its deprecation warnings are easy to dismiss as "we'll handle it when the gate flips." Two of these three flip in 1.37 with no gate, and the third's gate is the only thing left before a DaemonSet disappears in 18 months. A managed service would absorb this for you; a Cluster API fleet on owned Hetzner hardware trades that absorption for control and cost — and these three audits are the price.

The good news is they are all auditable without downtime. The stat on /sys/fs/cgroup, the grep on /etc/kubernetes/manifests, and the kubectl get cm kube-proxy are read-only. The canary MachineDeployment that validates each fix is one replica. And the window to run them — from now until August 26 — is still longer than the time it will take to discover at 3 a.m. that a pool that scaled cleanly on 1.36 does not join on 1.37.

Start with the script in section one. Everything else in this post is the annotation for why each line exists.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. We run our own Cluster API fleets on Hetzner, so 1.37's node-image contract changes are our contract too. Star the repo on GitHub or deploy your first app today.

Sources​

• Kubernetes Blog, Kubernetes v1.37 Sneak Peek (July 31, 2026) — ipvs deprecation timeline, static Pod gate removal, cgroup v1 failCgroupV1 default-true, SELinuxMount GA.
• KEP-5495, Deprecate ipvs mode in kube-proxy and KEP-3866, nftables proxy mode — ipvs's iptables dependency, nftables graduation criteria.
• kubernetes/kubernetes#140226 & k/k#131837 — PreventStaticPodAPIReferences removal and admission denial for static Pods.
• kubernetes/kubernetes#139067 & kubeadm#3309 — kubeadm warning for mode: ipvs.
• KEP-5573, Remove cgroup v1 support and Sysdig: Kubernetes 1.35 what-you-need-to-know — failCgroupV1: true since 1.35.
• Azure AKS, nftables support for kube-proxy in AKS and Beyond iptables: Scaling AKS with nftables — verifiable p50/p99 data at 5k–30k Services.
• Upstream issue k/k#137085 — iptables-nft shim 5–6× regression, the honest counterpoint to "nftables is just faster."