Kubernetes 1.37 Will Reject Your Static Pods: the kubeadm Audit to Run Before You Upgrade

On August 26, 2026, a quiet kubelet upgrade will start rejecting pods that have been running fine for years. No deprecation warning at runtime, no graceful fallback — the kubelet simply refuses to admit any static pod that references a Secret or a ConfigMap, and the flag that used to let you opt out is gone. If you run kubeadm or Cluster API, here is your 30-second audit — run it on every control-plane node before you upgrade:

grep -rnE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|kind: ConfigMap|kind: Secret' /etc/kubernetes/manifests/ && echo "ACTION REQUIRED" || echo "clean"

If that prints ACTION REQUIRED, one of your control-plane manifests is borrowing configuration through the API server it is supposed to boot. Kubernetes 1.37 "Garhwal" closes that loophole for good. The rest of this post is why the loophole existed, exactly what gets rejected, and the file-mount patterns that replace it.

Why static pods were never supposed to read the API​

Static pods are the bootstrap paradox of Kubernetes, resolved by fiat. The kubelet watches a directory on disk — /etc/kubernetes/manifests/ on kubeadm machines — and starts whatever YAML it finds there without talking to anyone. That is how the control plane starts without a control plane: kubelet launches kube-apiserver, etcd, kube-scheduler, and kube-controller-manager as static pods, and only then does an API server exist for anything else to register against.

Because a static pod is never created through the API server, it was never meant to consume API objects. A Secret or ConfigMap reference inside a static pod manifest asks the kubelet to resolve something through a server that may not exist yet — on a fresh boot, the very apiserver that would serve that Secret is the pod still starting. The v1.37 sneak peek states the rationale plainly: static pods "aren't created through the API server," so reading API resources directly was never intended.

So why did it ever work? A bug. The kubelet admitted the references, created a mirror pod object for visibility, and in many clusters the timing lined up: by the time the static pod needed the value, the API server was up and the reference resolved. It looked like a feature. Underneath, failure was silent in the worst way — as the 2025 Last Week in Kubernetes note describing the original gate put it, mirror-pod reconciliation for static pods referencing API objects would fail while the pod itself kept running. The node silently ran a container whose declared configuration never reconciled. That is the exact class of "works until it doesn't" that a strict admission check exists to kill.

The soft enforcement arrived first: a PreventStaticPodAPIReferences feature gate that denied admission for offending static pods when enabled. Kubernetes 1.37 finishes the job — references are strictly prohibited and the gate itself is removed (kubernetes/kubernetes#140226). There is no --feature-gates=PreventStaticPodAPIReferences=false escape hatch anymore. As the Garhwal release roundups summarize it: the logic is that static pods are not created through the API server, so they should not consume API objects — full stop.

What exactly gets rejected in v1.37​

The rule is broader than the two field names in the headline. Any static-pod manifest that reaches into API state is denied admission. Concretely, grep for all of these:

The kubelet rejects the pod at admission — it never starts, on every boot, until the manifest is fixed. Note what is not affected: regular pods created through the API server keep working exactly as before. Your tenants' envFrom: configMapRef is fine. Only the files in the kubelet's static-pod directory (--pod-manifest-path, default /etc/kubernetes/manifests/) are subject to the ban.

Who actually has these references? Three usual suspects. First, hand-customized control-plane manifests — someone added a secretRef to inject an etcd password or a cloud-provider credential into kube-controller-manager instead of mounting a file. Second, kubeadm patches and Cluster API bootstrap customizations that template API objects into static manifests. Third, monitoring or backup sidecars dropped into the manifests directory because "the kubelet runs whatever is here" felt like a convenient daemon mechanism. All three break identically on 1.37.

The fleet-wide audit: find every offender before August 26​

On one node the check is the one-liner from the top of this post. Across a fleet, run it everywhere the kubelet serves static manifests — which means every control-plane node, not just one:

# Per-node check (control-plane nodes only)
grep -rnE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|configMap:|secret:' \
  /etc/kubernetes/manifests/ && echo "ACTION REQUIRED on $(hostname)" || echo "clean on $(hostname)"

For a Cluster API fleet, do not SSH node to node. Bake the check into your pre-upgrade job: a DaemonSet constrained to control-plane nodes that mounts /etc/kubernetes/manifests read-only and reports hits, or a single clusterctl-era Ansible loop over the management cluster's Machine inventory. Two details people miss:

1. Check the rendered manifests, not just your templates. Kubeadm merges ClusterConfiguration, patches, and --config overlays at kubeadm init / upgrade time. The file the kubelet reads is /etc/kubernetes/manifests/kube-apiserver.yaml on disk — audit that, because a clean template plus a stale patch directory can still render a dirty manifest.
2. Watch the kubelet log signal on a canary node. Upgrade one control-plane node first and look for admission-denied events for mirror pods referencing API objects. A static pod that fails admission leaves the old container gone and nothing listening — on an apiserver node that reads as "the API went away after kubelet restart," which is a confusing symptom if you are not expecting an admission rejection. Knowing the signal ahead of time turns a mystery outage into a five-minute manifest fix.

Managed-Kubernetes users never see any of this — there is no manifests directory to audit because the provider owns the control plane. That is the standing trade of self-hosting: you own the bootstrap, so you own the bootstrap's breaking changes. The price is one grep; the reward is everything else in this blog's archive.

The fix: file mounts, not API refs​

The replacement principle fits in one sentence: a pod that boots before the API server must get its configuration from the filesystem, not from the API. In practice that means hostPath volumes and files on disk:

# Before (rejected on 1.37): credential via API reference
env:
- name: ETCD_PASSWORD
  valueFrom:
    secretKeyRef:
      name: etcd-creds
      key: password

# After (works everywhere): credential via file mount
volumeMounts:
- name: etcd-creds-file
  mountPath: /etc/etcd/creds
  readOnly: true
volumes:
- name: etcd-creds-file
  hostPath:
    path: /etc/kubernetes/etcd-creds
    type: DirectoryOrCreate

The application then reads /etc/kubernetes/etcd-creds/password, or the manifest passes --flag-file / --config pointing at the mounted path. For certificates and keys — the most common thing people smuggled through Secret volumes — nothing changes structurally: kubeadm already mounts /etc/kubernetes/pki into control-plane static pods via hostPath. Put the credential next to the certs and mount it the same way.

If you generate control-plane manifests with kubeadm, the supported seam is extraVolumes in ClusterConfiguration, not hand-editing rendered YAML:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  extraVolumes:
  - name: tenant-creds
    hostPath: /etc/kubernetes/tenant-creds
    mountPath: /etc/tenant-creds
    readOnly: true
    pathType: DirectoryOrCreate
controllerManager:
  extraVolumes:
  - name: tenant-creds
    hostPath: /etc/kubernetes/tenant-creds
    mountPath: /etc/tenant-creds
    readOnly: true
    pathType: DirectoryOrCreate

Distribute the files with your existing machine-bootstrap mechanism — cloud-init, Cluster API bootstrap data, or your image build — so the content exists before the kubelet first reads the manifests directory. Ordering matters: file on disk first, manifest referencing it second, kubelet restart last. And keep the Secret/ConfigMap objects around for your regular workloads if they consume the same data; the ban touches only static pods, so a Secret can simultaneously feed a Deployment (fine) while no longer being referenced by a manifest in /etc/kubernetes/manifests/ (required).

One migration trap: projected volumes that combine a ServiceAccount token with a ConfigMap or Secret source. Removing just the config/secret projection while keeping the token projection is legal, but re-verify the resulting volume — a half-edited projection is the most common way a "fixed" manifest still fails admission.

Upgrade sequencing for a fleet you own​

With the audit clean and the file mounts in place, the rollout is deliberately boring:

1. Inventory. List every control-plane node and confirm the audit one-liner is clean on each, against rendered manifests on disk.
2. Canary. Upgrade one control-plane node to 1.37, watch kubelet logs for admission rejections, confirm apiserver/etcd/scheduler/controller-manager static pods all reach Ready.
3. Roll. Upgrade remaining control-plane nodes one at a time, etcd quorum respected throughout.
4. Workers later. Worker nodes run no static control-plane pods in a standard kubeadm layout, but any node with a custom --pod-manifest-path gets the same audit.

Do this before you need it — the GA date was August 26, 2026, and every patch cycle after that bakes the behavior in deeper. The broader lesson generalizes beyond this one gate: anything in /etc/kubernetes/manifests/ is a promise to the kubelet, not to the API server, and promises to the kubelet must be satisfiable with the API server down. Audit that directory against every Kubernetes minor's "kubelet" section, not just the API-removal tables most upgrade guides emphasize.

Self-hosting the control plane means owning quirks like this one — and getting a platform that answers to you in return. Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.