Dora NodaFlipping storage: true to a new CRD version rewrites exactly nothing.
That is the sentence I wish someone had taped to my monitor before my first CustomResourceDefinition upgrade. You edit the CRD, you move the storage: true flag from v1alpha1 to v1, you apply, and kubectl reports success. Every new object from that moment on is persisted in etcd as v1. But every object you already had — every route, every deploy record, every tenant blob your PaaS control plane stored last month — sits in etcd exactly as it was: serialized as v1alpha1, untouched, aging.
Kubernetes converts objects on read, so everything looks fine, right up until the day you remove v1alpha1 from the CRD and those unreadable rows turn your routine upgrade into an outage.
Kubernetes 1.37, released August 2026 under the name Garhwal, finally makes the fix boring. The StorageVersionMigration API (storagemigration.k8s.io/v1) has graduated to general availability and is enabled by default: a declarative, in-tree controller that rewrites every stored object to the current storage version on demand. Here is the runbook — inventory, migrate, observe, verify, and only then delete the old version — followed by what actually changed in 1.37, how to abort and roll back, and why this matters more than it looks for anyone whose CRDs hold tenant state.
The runbook: four steps between you and deleting that old version
Our running example is a PaaS-typed object: routes.example.com, the CRD behind per-service domains and TLS state. It shipped as v1alpha1, grew up, and now v1 is the storage version. The same four steps apply to your deploy records, tenant objects, or anything else gathering dust in etcd.
Step 1 — Inventory what etcd actually holds. The CRD's .status.storedVersions lists every version ever persisted, and it is the only honest answer to "are we clean yet":
kubectl get crd routes.example.com -o jsonpath='{.status.storedVersions}'If that prints [v1alpha1 v1], you have objects in both encodings and you are not allowed to drop v1alpha1 yet. This check is the whole game; everything below exists to make this command eventually print [v1].
Step 2 — Stage the migration. Create one declarative object describing the resource to rewrite:
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
name: routes-migration
spec:
resource:
group: example.com
resource: routesApply it with kubectl apply -f routes-migration.yaml. That is the entire trigger — no daemon to deploy, no script to babysit. On 1.37+ the built-in StorageVersionMigrator controller in kube-controller-manager picks it up and starts rewriting stored objects to the current storage version. Because the migration is itself a standard Kubernetes API object, you can ship it in the same manifest as the updated CRD, so the upgrade and its storage rewrite travel together instead of living in separate runbooks.
Step 3 — Observe progress. The controller reports into the object's status:
kubectl get storageversionmigration routes-migration -o yamlA finished migration reports a Succeeded condition set to true:
status:
conditions:
- type: Running
status: "False"
reason: StorageVersionMigrationInProgress
- type: Succeeded
status: "True"
reason: StorageVersionMigrationSucceededPoll this, alert on it, gate your rollout on it — it is machine-readable, which is precisely what the old kubectl get | kubectl replace loops never were.
Step 4 — Verify, then delete the old version. Re-run the step 1 command. Only when .status.storedVersions shows exactly [v1] — no v1alpha1 anywhere — is it safe to remove the old version from the CRD's spec.versions list and drop conversion support for it. If the list still contains the old version after a reported success, the CRD was updated mid-migration; re-run the migration and check again. Order matters here and it is non-negotiable: migrate first, confirm storedVersions, then delete. The reverse order is the outage.
Why lazy rewriting bites
Kubernetes serves reads by converting whatever is in etcd to the requested version on the fly, which hides the skew completely. Your monitoring is green, your controllers are happy, and meanwhile a growing fraction of your durable state is pinned to a schema you plan to delete.
The failure mode is binary: the moment the old version leaves spec.versions, any object still serialized in it becomes unreadable — not degraded, unreadable. For a PaaS control plane whose CRDs are the tenant record (which routes exist, which deploys are live, which tenant owns what), that is not a cosmetic API cleanup gone wrong. It is data you can no longer serve, caused by an upgrade step that reported success weeks earlier.
The same mechanism explains a second, quieter hazard worth one line: objects rewritten late also pick up late-applied protections, so encryption-at-rest and key-rotation rollouts leave pre-existing objects unencrypted or under old keys until something rewrites them — and a storage migration is that something.
What 1.37 actually changed
None of the four steps above is a new idea. Operators have done this migration for years with hand-rolled kubectl get / kubectl replace loops — tedious, error-prone, and unobservable — or by deploying the out-of-tree kube-storage-version-migrator from kubernetes-sigs as a separate component with its own lifecycle to manage. What 1.37 changes is that the capability stops being a sidecar and becomes part of the control plane.
Concretely: the work tracked as KEP-4192, led by SIG API Machinery, moved the migrator in-tree. The StorageVersionMigration API is now storagemigration.k8s.io/v1 — stable, no more alpha/beta qualifier roulette across upgrades — and both the API and the controller ship enabled by default on every 1.37 cluster.
Per the 1.37 release announcement and the dedicated GA post, CRD authors can now trigger migrations as part of a CRD upgrade instead of managing the migration separately. If you run a self-hosted fleet, that removes an entire component — deployment, RBAC, version skew policy — from your management cluster's bill of materials. And because the migration object is declarative, GitOps-managed fleets get the rewrite audited and versioned like any other manifest, rather than as an imperative job somebody ran from a laptop at 2 a.m.
Rolling back: aborting and reverting without panic
A migration that can only go forward is a migration you will postpone. The in-tree design gives you three concrete escape hatches, and they belong in your runbook next to the happy path.
First, deleting the StorageVersionMigration object aborts an in-flight migration. The controller stops rewriting; objects already rewritten stay rewritten (they are valid in the new storage version), and objects not yet touched stay exactly as they were. There is no half-written state to repair because each rewrite is an ordinary API write of a fully converted object.
Second, keep the old version served: true until Succeeded is true. Serving is what keeps reads working during the rewrite window; storage is what new writes use. The failure teams actually hit is conflating the two — unserving the old version to "finish the cleanup" while the migration is still running, which breaks reads of not-yet-migrated objects. The rule: the old version stays served until storedVersions collapses, full stop.
Third, know what Failed means and what the revert path is. A migration fails fast on mechanical errors — a misnamed group or resource in spec.resource is the classic one — and the status tells you so instead of silently doing nothing. If the outcome is wrong rather than errored (migrated under a CRD revision you no longer want), the revert is: restore the previous CRD revision with the old storage version, re-run a migration back to it, and only then roll your controller binaries. Nothing about the migration burns bridges as long as the old version still exists in the CRD — which is exactly why step 4's ordering is load-bearing.
One honest limitation: the migrator rewrites bytes, it does not reconcile intent. If a conversion webhook or schema change drops a field between versions, the rewrite faithfully persists the converted shape, dropped field included. Review what the version bump removes before you migrate ten thousand tenant objects through it, not after.
Fleet-scale evidence this is a real problem, not theory
Cluster API's own v1beta1 to v1beta2 contract transition — with conversion webhooks bridging the two versions and clusterctl's CRD migration logic deciding when old storage could go — put this exact skew problem in front of every CAPI operator at fleet scale. The 1.37 GA graduates the general solution the whole ecosystem was previously hand-rolling per project.
The boring future is the point
The best infrastructure changes are the ones that delete runbooks, and this one deletes a real one: no more migrator deployment to version-pin, no more kubectl loops in a wiki page, no more "did anyone check storedVersions" postmortem action items. Inventory, migrate, observe, verify, delete — five verbs, one API object, on by default.
Make the migration part of every CRD version promotion, not just the major ones. Minor-looking version bumps are exactly how skew accumulates unnoticed: each one strands a few more objects until the eventual cleanup becomes the risky big-bang migration you were trying to avoid.
If your platform stores tenant state in CRDs and your upgrade checklist does not yet include a storedVersions gate, add it this week, before your next version bump makes the question urgent.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
