CSI Changed Block Tracking Hits Beta: Incremental Tenant-Volume Backups Without the Full-Copy Tax

On September 14, the Kubernetes project published the post that moves CSI changed block tracking from experiment to plan-around-it primitive: “Kubernetes Changed Block Tracking API – Beta Differences”, by Prasad Ghangal of Veeam Kasten. The headline is short — CBT graduated to beta with the March 2026 v1.0.0 release of the external-snapshot-metadata project — but the consequence for anyone running stateful tenants is large: backups can finally scale with how much data changed instead of how much data exists.

Here is the core math up front, because everything else in this post is commentary on it. Take a typical 100 GiB tenant volume — a Postgres primary, a build-cache disk, an agent-sandbox scratch volume — snapshotted nightly:

That is the whole pitch in one table: the backup window and the bytes pushed to object storage stop growing with the volume and start growing with the churn. And the honest footnote belongs right here too: past roughly 50% churn per interval, the transfer saving narrows toward nothing — you still skip re-reading unchanged blocks and keep snapshot-chained recovery, but a white-hot volume is nearly a full copy by another name. The lever then is shortening the snapshot interval, not the primitive. CBT is not magic; it is accounting. It just happens to be accounting most platforms have never had.

What CBT actually gives you: two RPCs and a sidecar​

Changed block tracking answers one question a backup tool has historically had to answer by brute force: which blocks differ between snapshot A and snapshot B? Without it, every “incremental” is either vendor-proprietary (VMware’s change IDs, cloud-provider snapshot diff APIs) or a full re-read with client-side dedup doing the cleanup. KEP-3314 standardizes the question as a CSI SnapshotMetadata gRPC service with two streaming calls:

• GetMetadataAllocated streams the allocated block ranges of a snapshot — the first full pass, which still only copies blocks that hold data rather than the whole provisioned volume.
• GetMetadataDelta streams the block ranges that changed between two snapshots, keyed off CSI snapshot handles — every later pass, which copies only churn.

Three components make that work: the SnapshotMetadata gRPC service implemented by the CSI driver, the SnapshotMetadataService CRD that advertises a driver’s metadata endpoint to the cluster, and the external-snapshot-metadata sidecar that fronts the driver’s service. A client — the reference snapshot-metadata-lister example, or Velero, or your own mover — discovers the endpoint through the CRD and streams metadata, then reads only the blocks the metadata names.

One scope limit matters enormously and is stated plainly in the beta post: CBT currently applies to block volumes only. File volumes and network file shares have no changed-list tracking in this feature. If your tenant data lives on NFS-style shares, beta changes nothing for you yet.

What beta actually changed, and the upgrade gotcha​

If you adopted the September 2025 alpha, the beta delta is small but sharp: the SnapshotMetadataService CRD graduated from v1alpha1 to cbt.storage.k8s.io/v1beta1, and the release removed v1alpha1 instead of serving both versions. The schema is unchanged, but there is no automatic conversion, so upgrading is a manual three-step: re-apply the CRD definition shipped with v1.0.0, move your manifests to apiVersion: cbt.storage.k8s.io/v1beta1, and update any client or controller code that talks to the CRD. One-time pain, and then the API surface is stable enough to build a product against.

The compatibility floor is equally concrete:

• Minimum Kubernetes version: 1.33
• CSI spec: 1.10 or newer
• Sidecar image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0

Note what that floor implies: the beta does not require a bleeding-edge cluster. A fleet tracking supported releases is likely already past 1.33, which means the adoption question is almost never “can my apiserver do this” — it is “can my storage do this.” That is the next section, and it is where most plans will stall.

The driver gate: your CSI driver decides, not your Kubernetes version​

CBT is a contract between the backup tool and the storage system, and Kubernetes merely hosts the handshake. A driver has to support volume snapshots and ship the external-snapshot-metadata sidecar and implement the metadata service before a single delta flows. So the serious adoption question is a driver matrix, not a version number.

The good news first: Ceph-CSI’s RBD driver has a full CBT implementation, with the snapshot-metadata sidecar integrated through the ceph-csi-operator, and the hostpath driver serves as the working reference (there is even a community CBT-to-S3 mover demo that walks the whole flow). The same community matrix marks AWS EBS CSI as not supporting it, and vSphere travels its own long-standing changeID-based path that Velero already consumes. The beta post’s “what’s next” section is admirably direct about this: the rest of the beta cycle is about wider driver adoption and operational feedback before GA.

Now the Hetzner consequence, stated without flinching: the hcloud-csi driver’s published architecture wires the attacher, provisioner, and resizer sidecars — no snapshotter in the chain — and Hetzner Cloud volumes are RWO block devices without a snapshot primitive for CBT to key off. A fleet whose tenant data sits directly on hcloud-volumes gets nothing from this beta today. The Hetzner fleet that does get CBT is the one running a self-managed storage layer that implements the contract — Ceph/RBD on owned machines being the obvious candidate — with the metadata sidecar deployed next to the controller plugin. That is a real architecture decision with real operating cost, which is exactly why the primitive being standard matters: you adopt Ceph’s CBT support once and every KEP-3314 consumer works with it, instead of marrying each backup vendor’s proprietary integration.

The consumer side: Velero already speaks it​

A standard API with no clients is a whitepaper. CBT has clients. Velero’s CSI snapshot data movement supports a velero-block data mover path that talks to the SnapshotMetadataService to retrieve changed blocks since the previous snapshot and backs up only those, letting unchanged blocks share data with parent snapshots. Velero 1.18 added visible incremental sizes for data-mover backups, so you can watch the table from this post’s opening materialize per backup instead of trusting it. And the stack is being hardened in production right now: an August 2026 Velero fix corrected generic-CSI changeID retrieval that had been silently degrading every CBT incremental to a full backup on non-vSphere drivers — precisely the failure mode (quietly paying the full-copy tax while believing you weren’t) that a beta period exists to shake out.

Kasten’s authorship of the beta announcement is its own signal: the backup vendors are building on the standard API rather than around it. When the company whose product lives or dies on restore reliability documents your beta, the beta is past the “will anyone adopt this” phase and into “how fast do drivers follow.”

What you still have to build before you promise it​

Here is the part the announcement doesn't cover, because a primitive is not a product. CBT hands you which blocks changed. Everything below still has to be built, operated, and tested before “incremental backups included” appears in anything a tenant reads:

1. Snapshot scheduling. CBT diffs snapshots; something has to take them on a cadence matched to your recovery-point objective. Hourly snapshots bound both data loss and per-interval churn (remember the 50% crossover), but each snapshot is retained state on the storage backend with its own cost and lifecycle.
2. Retention and chain policy. Deltas key off snapshot handles, so your retention policy is your recovery topology: which fulls you keep, how long delta chains run before a fresh full, and what happens to restores when an intermediate snapshot ages out. Decide this on paper before the first backup runs.
3. The restore runbook, rehearsed. Incremental backups restore by replaying a chain, which is strictly more moving parts than restoring one full copy. Cross-node restore, full-chain verification, time-to-restore under a tenant breathing down your neck — none of that ships in the sidecar. Schedule the fire drill before you need it.
4. Driver-qualification testing. Beta means the API is stable, not that every driver’s implementation is correct. The Velero changeID fix above is the template: verify on your driver that an incremental actually transferred incrementally (Velero 1.18’s size reporting exists for this) rather than assuming the standard guarantees the behavior.

The adopt-vs-wait rule for a small fleet follows directly. Adopt the design now if your storage already speaks the contract (Ceph/RBD) or you are choosing storage this quarter — beta API stability plus Velero support is enough to build against, and designing snapshot cadence around CBT today beats retrofitting it later. Wait on the promise if your volumes come from a driver without snapshot support: track the driver’s roadmap, keep your full-copy backups, and revisit when the sidecar lands. Either way, stop sizing future backup windows against full-copy math. The industry just agreed on what the delta API looks like, and full copies are now a fallback position, not the plan.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Prasad Ghangal (Veeam Kasten), “Kubernetes Changed Block Tracking API – Beta Differences”, Kubernetes Blog, September 14, 2026
• “Spotlight on SIG Storage”, Kubernetes Blog, June 15, 2026 (CBT beta in v1.36; Volume Group Snapshots GA)
• KEP-3314: CSI Changed Block Tracking
• kubernetes-csi/external-snapshot-metadata (v1.0.0 beta release; snapshot-metadata-lister example client)
• Ceph-CSI RBD deploy docs: snapshot metadata and ceph-csi-operator RBD snapshot metadata
• Velero: CSI Snapshot Data Movement (velero-block incremental via SnapshotMetadataService), Block Data Mover design, CHANGELOG-1.18 (incremental-size observability), and the August 2026 generic-CSI changeID fix
• hcloud-csi architecture docs (sidecar chain: attacher, provisioner, resizer)