Running Your First StorageVersionMigration: What Kubernetes v1.37's GA Migration API Changes in Your Cluster API Upgrade Runbook

You upgraded kube-apiserver to v1.37 last weekend. The rollout went clean, every node rejoined, and the dashboards are green. But down in etcd, thousands of your objects are still serialized in API versions you thought you left behind two releases ago — because upgrading the apiserver changes what Kubernetes serves, not what it has stored.

Kubernetes v1.37 "Garhwal", released August 26, 2026, finally gives you a built-in, declarative way to fix that: the StorageVersionMigration API (storagemigration.k8s.io/v1) graduated to stable and is now enabled by default on every cluster. No more hand-rolled kubectl get loops, no more deploying a separate migrator component. Here is the whole thing in 30 seconds — the rest of this post is the runbook around it.

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: secrets-migration
spec:
  resource:
    group: ""
    resource: secrets

kubectl --context workload-1 apply -f migrate-secrets.yaml
kubectl --context workload-1 wait --for=condition=Succeeded storageversionmigration/secrets-migration --timeout=300s

That is a complete storage migration: one declarative object, one wait. Now let's walk through when to run it, what to migrate before the apiserver bump versus after, how to watch it per resource, and where it slots into a Cluster API fleet's upgrade runbook.

Why upgrading kube-apiserver doesn't rewrite etcd​

Every object in etcd is serialized in a specific storage version — the schema representation the apiserver used when the object was last written. When you upgrade the apiserver, new writes use the new storage version, but existing objects sit untouched until something mutates them. Kubernetes has always required an active re-write to move a stored object to the latest storage version; the upgrade itself rewrites nothing.

This bites in two familiar places. The first is CRD version promotion. Say your platform CRD crontabs.example.com has served v1beta1, v1beta2, and v1 over its lifetime, and you now want v1 to be the only storage version so you can eventually drop the betas. Flipping storage: true to v1 only affects new writes — every existing object is still encoded as v1beta1 or v1beta2 in etcd.

Until every object is rewritten, the old versions are load-bearing whether your manifests admit it or not: the apiserver will not let you remove a version from the CRD while it is still listed in .status.storedVersions.

The second place is encryption at rest. When you enable encryption or rotate keys, objects already in etcd stay encrypted under the old key (or unencrypted) until they pass through the apiserver again. A key rotation without a follow-up rewrite is a rotation on paper only.

Historically, both cases meant the same drudgery: manual read-and-replace scripts, or deploying the out-of-tree kube-storage-version-migrator from kubernetes-sigs and babysitting it. As the upstream GA announcement puts it, those approaches were tedious, error-prone, and difficult to monitor — which is exactly why most teams silently skipped the migration step and let stale storage versions accumulate across upgrades.

What v1.37 GA actually changes​

After a beta run that started in v1.35 with the StorageVersionMigrator feature gate off by default, v1.37 graduates both the storagemigration.k8s.io/v1 API and its in-tree controller to stable and turns them on everywhere. The controller lives in the control plane, watches for StorageVersionMigration objects, and migrates every stored object of the named resource to the API's current storage version by issuing no-op rewrites that force the apiserver to re-encode each object (invoking the conversion webhook for anything still stored at an older version).

Three properties make this a runbook-grade primitive rather than a nicer script. First, it is declarative: the migration is a standard API object you can kubectl apply, commit to git, and re-run idempotently. Second, it is observable: the controller reports Running and Succeeded conditions in the object's .status, so kubectl wait --for=condition=Succeeded becomes your completion gate.

Third, it covers built-in and custom resources through the same shape — only group and resource change between migrating secrets and migrating your own CRDs. The one structural requirement is that the resource must have integer resource versions, which every native resource and CRD satisfies; aggregated APIs that don't will fail the migration rather than silently half-migrate.

Note what GA does not do: it migrates to the current storage version, so the object takes no version field — you don't pick a target, you declare "bring this resource up to date" and the controller resolves what up to date means.

The walkthrough: audit, migrate, verify​

Every command below runs against one workload cluster, addressed explicitly with --context workload-1. That flag is doing real work: in a Cluster API fleet your default kubeconfig context is usually the management cluster, and a migration object applied without --context would migrate the wrong etcd behind the wrong apiserver. Repeat each step per workload cluster; Section 5 turns the repetition into a loop.

Step 0: Audit stored versions before the bump (BEFORE-phase)​

Before touching the apiserver version, find out what is stale. This audit answers the runbook's central question — what migrates before the bump versus after:

kubectl --context workload-1 get crds -o custom-columns=NAME:.metadata.name,STORED:.status.storedVersions

Sample output:

NAME                      STORED
crontabs.example.com      [v1beta1,v1beta2,v1]
tenants.platform.io       [v1]

The rule is simple. Before the apiserver bump, migrate anything the new release stops serving (check the release notes for removed API versions — an object the new apiserver can't serve is an object you can't read back) plus any encryption re-key you have pending. After the bump, migrate resources whose preferred storage version the new release changed. Example A below is a before-phase migration; Example B is shown as an after-phase migration, though the identical manifest pattern serves before-phase CRD cleanups — only the timing differs.

Example A: Re-encrypt Secrets after a key rotation (BEFORE-phase)​

You rotated your encryption config from key1 to key2. New Secrets encrypt under key2; everything already stored still sits under key1. Write migrate-secrets.yaml:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: secrets-migration
spec:
  resource:
    group: ""
    resource: secrets

Apply it and gate on completion:

kubectl --context workload-1 apply -f migrate-secrets.yaml
kubectl --context workload-1 wait --for=condition=Succeeded storageversionmigration/secrets-migration --timeout=300s

A note on the manifest: there is deliberately no version field. Both the upstream GA announcement's example and the official task guide's examples set only group and resource — the controller always targets the API's current storage version, so there is nothing to choose.

This migration is a same-storage-version rewrite: the storage schema for Secrets didn't change, only the encryption key did. That means .status.storedVersions can't prove anything here — the proof is in the stored bytes. From a control-plane host with etcd credentials, read a Secret's raw value:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret [...] | hexdump -C

([...] stands for your endpoint and TLS flags.) Before the migration the dump shows the k8s:enc:aescbc:v1:key1 prefix; after Succeeded=True, it shows k8s:enc:aescbc:v1:key2. If you skip this byte-level check on a re-key migration, you have a green condition and an unproven rotation.

Example B: Collapse a CRD's version pile-up (AFTER-phase)​

crontabs.example.com carries three stored versions and you have already flipped storage: true to v1 (with a conversion webhook in place for the betas). Write migrate-crd-pileup.yaml:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: crd-pileup-migration
spec:
  resource:
    group: example.com
    resource: crontabs

Apply, wait, and prove the collapse with the exact before/after check:

kubectl --context workload-1 apply -f migrate-crd-pileup.yaml
kubectl --context workload-1 wait --for=condition=Succeeded storageversionmigration/crd-pileup-migration --timeout=300s
kubectl --context workload-1 get crd crontabs.example.com -o jsonpath='{.status.storedVersions}'

Before the migration that last command prints ["v1beta1","v1beta2","v1"]; after a successful migration it prints ["v1"]. Only now is it safe to remove the beta versions from the CRD's served list — the apiserver enforces this ordering by refusing to drop a version that is still recorded as stored.

Watching status per resource​

Between apply and Succeeded, inspect progress exactly as you would any controller-owned object:

kubectl --context workload-1 get storageversionmigration/secrets-migration -o yaml

While running you will see Running: True with reason StorageVersionMigrationInProgress; on completion the conditions settle to:

status:
  conditions:
  - type: Running
    status: "False"
    reason: StorageVersionMigrationInProgress
  - type: Succeeded
    status: "True"
    reason: StorageVersionMigrationSucceeded

One resource, one object, one status — which is also why fleet-scale migration is a matrix of small, individually observable jobs rather than one opaque "migrate everything" blast.

The failure to rehearse: CRD edited mid-migration​

Here is the gotcha worth practicing on a dev cluster before it finds you in production. If the CRD itself is updated while a migration is running, the migration can report Succeeded=True while .status.storedVersions still lists the old versions — the controller migrated what it saw, but the goalposts moved mid-run. Your jsonpath re-check from Example B catches it:

kubectl --context workload-1 get crd crontabs.example.com -o jsonpath='{.status.storedVersions}'

If that still prints ["v1beta1","v1beta2","v1"] after success, don't argue with it — retry by deleting and recreating the migration object:

kubectl --context workload-1 delete storageversionmigration/crd-pileup-migration
kubectl --context workload-1 apply -f migrate-crd-pileup.yaml
kubectl --context workload-1 wait --for=condition=Succeeded storageversionmigration/crd-pileup-migration --timeout=300s
kubectl --context workload-1 get crd crontabs.example.com -o jsonpath='{.status.storedVersions}'

The re-check should now print ["v1"]. The lesson for the runbook: Succeeded means the migrator finished its pass, and the storedVersions check means the goal was actually reached. Gate promotions on the second, not just the first.

Where it fits in a Cluster API fleet's upgrade runbook​

On a CAPH-provisioned Hetzner fleet, the topology is what makes this a fleet problem: one management cluster plus N workload clusters, each workload cluster with its own etcd and apiserver. A StorageVersionMigration object only migrates the cluster it is applied to, so the unit of work is per workload cluster, per resource — an N-clusters × M-resources matrix of small declarative jobs.

The full upgrade sequence, with the walkthrough's examples tagged to their phases, is:

1. SVM-before — §Example A runs here: migrate anything the target release stops serving, plus pending encryption re-keys, on every workload cluster.
2. KCP roll — bump KubeadmControlPlane.spec.version; Cluster API rolls the control plane one node at a time. This is the apiserver bump.
3. SVM-after — §Example B runs here: migrate resources whose preferred storage version the new release changed, on every workload cluster.
4. MD roll — bump the MachineDeployment version and roll the workers. One minor version at a time, control plane first, as always.

Across the fleet, the per-cluster repetition is a loop, not a copy-paste session — remembering that your default context is the management cluster, which is why every command carries its target explicitly:

for ctx in workload-1 workload-2 workload-3; do
  kubectl --context "$ctx" apply -f migrate-crd-pileup.yaml
done
for ctx in workload-1 workload-2 workload-3; do
  kubectl --context "$ctx" wait --for=condition=Succeeded storageversionmigration/crd-pileup-migration --timeout=300s
done

Because migrations are plain manifests, CRD authors and platform teams can bundle a StorageVersionMigration object in the same repo — even the same file — as the CRD upgrade that needs it. But ordering matters: apply the CRD change first and the migration after, enforced with SyncWave or dependsOn ordering in your GitOps tooling, never as an unordered co-sync. An unordered bundle re-invites the mid-migration-edit failure from the previous section and earns you the same false Succeeded.

What GA still doesn't cover​

Three honest limits before you rewrite the runbook in ink. First, nothing is automatic: flipping a storage version or rotating a key creates no migration by itself. The API being GA means the mechanism is stable and always on — the decision of what to migrate and when is still yours, which is why the before/after split above is a policy your team writes down, not a default you inherit.

Second, aggregated API servers are out of scope. The migrator requires integer resource versions; an aggregated API that doesn't provide them fails the migration. If your fleet serves extension apiservers, those keep their old manual rewrite procedures until they meet the requirement.

Third, mind the blast radius on large tenant control planes. A migration lists and rewrites every object of the resource — on a control plane with hundreds of thousands of objects that is real apiserver and etcd load. Run one resource at a time, run it off-peak for tenant-facing clusters, and treat the per-resource status as your pacing signal: don't start the next migration until the previous one reads Succeeded and your storedVersions or byte-level proof confirms it.

Storage hygiene becomes a line item​

The unglamorous truth is that most clusters have never run a single storage migration — the old tooling made it a chore, so stale versions compounded silently across upgrades until a CRD cleanup or a key rotation turned into archaeology. v1.37 doesn't make the decision for you, but it finally makes the mechanism boring: a ten-line manifest, a kubectl wait, and a proof check. Put the before/after split in your upgrade runbook next to kubeadm upgrade plan, loop it across the fleet's contexts, and the next apiserver bump leaves nothing behind in etcd.

Running this kind of fleet-wide upgrade hygiene by hand on every release is exactly the toil a self-hosted PaaS should absorb into its controllers. Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.