Your Fleet Is Paying Rent on Disks Nobody Mounts: Reaping Orphaned PVCs With Kubernetes v1.37's Last-Used Timestamp

Somewhere in your fleet right now, a disk is billing by the hour for a database that no longer exists. The tenant deleted the app months ago — or scaled the StatefulSet down, or uninstalled the Helm release — and the PersistentVolumeClaim stayed behind, exactly as designed. Kubernetes does not garbage-collect PVCs when pods go away, because deleting data by surprise is worse than keeping it. The result is the quietest line item in self-hosted storage: volumes nobody mounts, everybody pays for, and nobody can confidently delete.

Put a number on it. Hetzner Cloud Volumes have billed €0.0572 per GB-month since the April 2026 price adjustment, hourly with a monthly cap, until the volume is deleted. A single forgotten 100 GB tenant volume therefore costs about €69 a year — every year — for storing bytes no pod will ever read again. Fifty such orphans across a multi-tenant fleet is roughly €3,400 a year of pure waste, before counting the operational drag of a storage inventory nobody trusts.

And this is the cheap substrate: the same orphan on DigitalOcean or Vultr standard tiers runs about $0.10 per GB-month, roughly twice as much. The waste is systemic, not anecdotal — Flexera's 2026 State of the Cloud report puts estimated wasted cloud spend at 29%, the first increase in five years, with idle and orphaned resources named among the drivers.

The reason orphans survive is not laziness. Until now, answering "is anything actually using this volume?" meant cross-referencing pods, PVCs, and PVs across namespaces and time — custom scripts, brittle inventory jobs, or a human squinting at kubectl get pvc output. On September 21, 2026, Kubernetes shipped a first-class answer: v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature to beta, enabled by default, so every PVC now carries an Unused condition with a timestamp recording when it went idle. This post turns that timestamp into a deletion policy you can defend to your tenants: here is the whole pipeline up front, with each stage's concrete parameter, and the rest of the post justifies every number in it.

The policy at a glance: discover candidates idle more than 30 days via the Unused condition; exclude protected namespaces and the production stateful tier outright; resolve a human owner from labels; cross-check StatefulSets, CronJobs, Helm releases, and GitOps state; snapshot before touching anything; annotate the PVC and wait a 14-day review window with tenant notice; then verify the PV reclaim policy and delete. Seven stages, two clocks, zero guesses.

What v1.37 actually ships​

The mechanism is small, which is why it is trustworthy. The PVC protection controller — the same controller that already watches pods to enforce storage object in use protection — now maintains an Unused condition on every PVC's status. No new CRD, no sidecar, no inventory service to run. The feature graduated from alpha in v1.36, where it required explicitly enabling the gate, to beta in v1.37, where it is on unless you opt out, with full end-to-end test coverage. The design lives in KEP-5541, and GA is planned in a future release pending feedback.

The condition has exactly two states, and the semantics are deliberate:

Three details matter for everything that follows. First, terminated pods don't count: a pod in Succeeded or Failed phase will not keep a PVC marked as in use, so finished batch jobs release their claims. Second, pending pods do count — even an unschedulable pod with an impossible node selector holds the PVC as used, because the reference expresses intent. Third, for shared claims the condition flips to True only after the last non-terminated pod lets go.

Like every Kubernetes condition, Unused carries a standard lastTransitionTime. When the condition flips from False to True, that field records the moment the PVC went idle — an "unused since" timestamp you get for free. Inspect one claim with:

kubectl get pvc app-data -n production \
  -o jsonpath='{.status.conditions[?(@.type=="Unused")]}' | jq .

{
  "lastTransitionTime": "2026-08-10T14:22:00Z",
  "message": "No pods are currently referencing this PVC",
  "reason": "NoPodsUsingPVC",
  "status": "True",
  "type": "Unused"
}

And the fleet-wide discovery query — the mouth of the whole pipeline — lists every PVC idle more than 30 days:

kubectl get pvc -A -o json | jq -r '
  .items[]
  | select(.status.conditions[]?
      | select(.type == "Unused" and .status == "True"))
  | (.status.conditions[] | select(.type == "Unused")) as $condition
  | select((now - ($condition.lastTransitionTime | fromdateiso8601)) > (30 * 86400))
  | "\(.metadata.namespace)/\(.metadata.name)\t\($condition.lastTransitionTime)"
'

Treat this output as a review queue, never a deletion list. The condition answers whether a non-terminal pod currently references the claim. It says nothing about whether the organization still needs the data — and closing that gap is what the other six stages are for.

Stage by stage: the policy in detail​

Stage 1 — Discover (threshold: idle more than 30 days). Run the query above on a schedule — a weekly CronJob writing to your inventory is enough — and record each candidate's namespace, name, size, StorageClass, and idle-since timestamp. Thirty days is deliberately longer than any legitimate short-idle pattern: deploy freezes, monthly batch jobs, and tenant vacations all fit inside it. Export the signal into your existing inventory or metrics pipeline alongside provisioned GiB so FinOps can see idle capacity by namespace, team, and age bucket.

Stage 2 — Exclude (protected namespaces and the production stateful tier). Some volumes must never enter the pipeline no matter how long they sit idle. Exclude system namespaces, backup staging areas, and — explicitly, by StorageClass or namespace label — the production tenant-Postgres tier. A primary database volume that shows Unused=True is either a failover artifact or a monitoring blind spot, and neither should be resolved by a reaper. The exclusion list is the cheapest insurance in the policy; keep it in version control next to the discovery job.

Stage 3 — Resolve an owner (from labels, not memory). Every candidate needs a human: a tenant contact, a team channel, an on-call rotation. Resolve it from PVC labels or annotations (tenant-id, team, cost-center) and flag candidates with no owner as a separate work item — a high "claims without owners" count is itself a hygiene metric worth tracking. If your platform doesn't require owner labels at provisioning time, this stage will hurt the first month and then fix your provisioning defaults forever.

Stage 4 — Cross-check the workload managers (StatefulSets, CronJobs, Helm, GitOps). This is where most naive reapers die. Check whether a scaled-down StatefulSet still owns the claim via its volumeClaimTemplates (scaling back up reattaches the same PVCs), whether a CronJob mounts it on a schedule longer than your discovery window, whether a Helm release that created it is still installed, and whether the GitOps repo still declares the workload.

Note the existing alternative here: StatefulSets have supported persistentVolumeClaimRetentionPolicy since beta in v1.27 and GA in v1.32, with whenDeleted and whenScaled settable to Delete — but both default to Retain, so the fleet-wide default still orphans volumes unless every workload opts in. The Unused condition is the backstop for everything that didn't.

Stage 5 — Snapshot before touching anything. If your CSI driver supports snapshots, take one and verify it before the candidate proceeds. The snapshot is what converts "we deleted your data" from an incident into a restore ticket. Record the snapshot handle on the PVC annotation from the next stage. For drivers without snapshot support — check yours before you need it — fall back to a documented backup job and keep the candidate waiting until the backup verifies.

Stage 6 — Annotate, notify, wait (review window: 14 days). Mark the PVC as a cleanup candidate where every tool can see it:

kubectl annotate pvc app-data -n staging \
  reaper.example.com/candidate-since=2026-09-25 \
  reaper.example.com/snapshot=snap-abc123

Notify the owner from stage 3 with the idle-since date, the snapshot handle, and the deletion date. Then wait the full 14 days. The two clocks serve different purposes: the 30-day discovery threshold filters noise, while the 14-day review window gives a human time to object. An admission policy can enforce the discipline mechanically — refuse PVC deletion in scope unless the candidate annotation is present and old enough.

Stage 7 — Verify the reclaim policy, then delete. Before deleting the claim, inspect what happens to the volume underneath:

kubectl get pv "$PV_NAME" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}{"\n"}'

This is the branch the whole pipeline funnels into. With Retain, deleting the PVC releases the PV for manual recovery — a mistaken deletion is embarrassing but survivable. With Delete, the provisioner destroys the backing volume — on Hetzner, that storage stops billing immediately and the bytes are gone. Never let stage 7 run on a Delete-policy volume without the stage 5 snapshot verified.

Then delete, confirm the PV and backing volume are actually released, and log the recovered GiB against the candidate record. Track restore requests after deletion as your policy's error rate: a rising restore rate means the grace windows are too aggressive.

The four traps each stage exists to catch​

Each stage above exists because a real workload pattern looks orphaned while being anything but. Learn these four and the pipeline stops feeling bureaucratic.

The CronJob that mounts twenty minutes a day. A nightly job's PVC reads Unused=True for 23-plus hours out of every 24. The 30-day discovery threshold plus the stage 4 CronJob cross-check exist precisely for this shape. Any reaper that acts on "currently unused" without an age gate will eventually eat a monthly-reporting volume.

The StatefulSet scaled from 3 to 1. The PVCs for the missing replicas sit idle, and scaling back up reattaches them — the data is part of the workload's live state, not debris. This is also the case for setting whenScaled: Delete deliberately on workloads where scaled-away replicas genuinely are disposable, rather than letting the reaper guess.

The completed job whose output is under retention. Terminated pods don't hold the Unused condition, so a finished job's claim flips to idle immediately — while legal, compliance, or analytics retention may require the output for months. The controller cannot know your retention schedule. Stage 4's backup-and-retention check is the only thing standing between the timestamp and a compliance incident.

The upgrade that resets every clock. The Unused condition is born when the v1.37 controller first writes it, which means lastTransitionTime cannot predate your upgrade — a volume idle since March and a volume idle since yesterday get indistinguishable timestamps on upgrade day. Never run stage 7 inside the first grace window after upgrading; start the 30-day clock at the upgrade date and treat the first discovery run as a census, not a hit list. The same caution applies to any cluster where the feature gate was off and is newly enabled.

One orphan, end to end​

Walk one concrete candidate through the pipeline. The weekly discovery job flags tenant-acme/orders-db-data, a 100 GB tenant-Postgres volume on the standard StorageClass, Unused=True since August 2 — 41 days idle, over the 30-day threshold. At Hetzner rates that is €5.72 a month, €68.64 a year, for a volume nothing mounts.

Stage 2 passes it through: it's in a tenant namespace, not the protected production tier (the tenant migrated to a dedicated instance in July; this was the old shared-tier copy). Stage 3 resolves the owner from the tenant-id: acme label to the tenant's contact plus the platform team. Stage 4 finds no StatefulSet, CronJob, Helm release, or GitOps manifest referencing it — the owning release was uninstalled in July, and the default Retain policy kept the claim, exactly as designed.

Stage 5 takes a CSI snapshot and verifies it. Stage 6 annotates the PVC, notifies the tenant with a deletion date 14 days out, and nobody objects. Stage 7 checks the PV: reclaim policy Delete, so without the snapshot this would be the point of no return — with it, deletion proceeds, the backing volume is released, and the fleet stops paying €5.72 a month for bytes nobody reads.

One volume is a rounding error. But the discovery query that found it found all of them: if the census turns up fifty Acme-shaped orphans, the same pipeline recovers roughly €3,400 a year of spend plus an inventory the team can finally trust. That is the real deliverable — not the deletion, but the confidence that every remaining volume is there on purpose.

Monday morning checklist​

If you run a Cluster-API-managed fleet on Hetzner or any other owned substrate, here is the week-one sequence. Upgrade a non-production cluster to v1.37 and confirm the Unused condition appears on its PVCs. Export the condition and lastTransitionTime into your inventory so idle capacity becomes a dashboard instead of a rumor. Write down the exclusion list and the two clocks — 30 days to candidacy, 14 days of review — and start the first clock on upgrade day, not before. Run the discovery query as a census, resolve owners for whatever it finds, and schedule the first real review for 44 days out. Then set persistentVolumeClaimRetentionPolicy deliberately on new StatefulSets so fewer orphans are born while the reaper clears the backlog.

Kubernetes spent years making accidental deletion hard. With a beta timestamp and a seven-stage policy, deliberate deletion can finally be easy — and auditable, and reversible, and cheap enough to run every week instead of never.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Storage hygiene like this is built into the platform's fleet operations. Star the repo on GitHub or deploy your first app today.