Stop Passing Service Account Tokens to CSI Drivers in Volume Context: What the v1.35 Token-in-Secrets Fix Means for Your Storage Layer

Every CSI driver that authenticates as a workload — to pull secrets from a vault, to mount a cloud file share — receives its service account token inside volume_context, a gRPC map the ecosystem's log sanitizer does not treat as sensitive. That one design detail has already produced two CVEs' worth of tokens in driver logs. Kubernetes 1.35 shipped the fix: an opt-in flag that moves token delivery into the CSI spec's secrets field, and 1.36 took it GA. Here is the mechanism, the two CVEs, and the exact rollout sequence for a fleet running its own storage drivers.

Run this first, before reading further — it tells you whether your fleet is exposed:

# Which installed drivers request service account tokens at all?
kubectl get csidriver -o json | \
  jq -r '.items[] | select(.spec.tokenRequests != null) | .metadata.name'
 
# Is any of them currently receiving tokens via volume_context (the leaky path)?
kubectl get csidriver -o json | \
  jq -r '.items[] | select(.spec.tokenRequests != null) |
    "\(.metadata.name) serviceAccountTokenInSecrets=\(.spec.serviceAccountTokenInSecrets // false)"'
 
# Have tokens already landed in a driver's logs? (check each driver that uses TokenRequests)
kubectl logs -l app=secrets-store-csi-driver -c secrets-store | \
  grep -c "csi.storage.k8s.io/serviceAccount.tokens"

If the first command prints driver names and the second prints false, your tokens travel the log-friendly path. The rest of this post explains why that path exists, what it cost two drivers, and how to migrate off it without breaking mounts.

Why workload-identity tokens ride in volume_context​

CSI drivers sometimes need to act as the workload: the Secrets Store CSI Driver exchanges a token for vault secrets, a file-share driver presents one to the storage backend. The TokenRequests feature exists for exactly this — a driver declares tokenRequests (audience, expiration) in its CSIDriver object, and kubelet mints bound service account tokens and hands them to the driver inside the volume attributes map, under the key csi.storage.k8s.io/serviceAccount.tokens. That map travels as volume_context in the driver's gRPC requests.

It works. It was also never designed for secrets. The protosanitizer utility from csi-lib-utils — the standard tool CSI drivers use to scrub sensitive fields before logging gRPC traffic — does not treat volume_context as sensitive. So any driver that logs requests at a verbose level prints its service account tokens in plaintext. Every driver that wanted to avoid this had to hand-roll its own sanitization, which meant protection was inconsistent by construction: each driver either remembered to redact that one key or it didn't.

The CSI specification already has a field designed for this: secrets in NodePublishVolumeRequest, which protosanitizer redacts automatically. The only reason tokens weren't there from the start is compatibility — every driver that reads TokenRequests today expects them in volume_context, and moving them unilaterally would break every one of those drivers on upgrade day.

Two CVEs, one shape​

CVE-2023-2878 — Secrets Store CSI Driver. Versions before 1.3.3 logged service account tokens whenever TokenRequests was configured and the driver ran at log verbosity 2 or higher. Rated MEDIUM (CVSS 6.5), with a scope-changed confidentiality impact: anyone who could read driver logs could lift a token and exchange it with an external cloud provider for the vault secrets the token was minted to fetch. The irony wrote itself — the driver whose entire job is keeping secrets out of the wrong hands was printing workload identity into its own logs.

CVE-2024-3744 — Azure File CSI Driver. A year later, the same shape in a different driver: TokenRequests configured plus -v at 2 or above, tokens in the logs, MEDIUM 6.5, same scope-changed vector. The interim mitigation was to pin the driver to log level 0 or 1 — effective, but a posture that evaporates the next time someone raises verbosity to debug a mount failure at 3 a.m.

Two drivers, two years, identical root cause. That repetition is what makes this a platform problem rather than two driver bugs: as long as the delivery field is one the sanitizer ignores, every driver that adopts TokenRequests re-rolls the same dice. Patching drivers one CVE at a time converges only if no new driver ever makes the same mistake — a bet the ecosystem already lost twice.

The fix: v1.35 beta, v1.36 GA​

KEP-5538, implemented in kubernetes/kubernetes#134826 and announced on the Kubernetes blog on January 7, 2026, adds one opt-in field to the CSIDriver spec:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: example-csi-driver
spec:
  tokenRequests:
    - audience: "example.com"
      expirationSeconds: 3600
  # New in 1.35. Defaults to false: existing behavior, tokens in volume_context.
  serviceAccountTokenInSecrets: true

The semantics are deliberately boring. With the field false (the default), tokens land in volume_context exactly as before. With it true, tokens land only in the secrets field of NodePublishVolumeRequest, under the same csi.storage.k8s.io/serviceAccount.tokens key — and protosanitizer redacts that field with no driver-specific code.

The CSIServiceAccountTokenSecrets feature gate shipped beta and enabled by default on both kubelet and kube-apiserver. SIG Storage started at beta rather than alpha precisely because the default-false field makes the gate a no-op for existing drivers: turning it on changes nothing until a driver opts in. Kubernetes 1.36 then graduated the API to stable.

One honest caveat before the runbook: GA standardizes the platform side, not driver adoption. Whether your drivers honor the flag depends on each driver's own release — some have shipped fallback support and documented the flip, others haven't. Check each driver's changelog before relying on the flag in production; "the cluster is on 1.36" is necessary but not sufficient.

The rollout sequence (and the gotcha that breaks mounts)​

Driver authors and cluster operators share one ordering rule: fallback logic first, flag flip second, with a completed rollout between them. The driver-side fallback is a few lines — check secrets first, fall back to volume_context:

const serviceAccountTokenKey = "csi.storage.k8s.io/serviceAccount.tokens"
 
func getServiceAccountTokens(req *csi.NodePublishVolumeRequest) (string, error) {
    // Check secrets field first (new behavior when driver opts in)
    if tokens, ok := req.Secrets[serviceAccountTokenKey]; ok {
        return tokens, nil
    }
    // Fall back to volume context (existing behavior)
    if tokens, ok := req.VolumeContext[serviceAccountTokenKey]; ok {
        return tokens, nil
    }
    return "", fmt.Errorf("service account tokens not found")
}

That snippet is backward compatible and safe to ship in any driver version, even on clusters older than 1.35 — which is why SIG Storage asked authors to ship it early and backport it to maintenance branches. Once it is deployed, the cluster-side order is:

1. Upgrade kube-apiserver to 1.35 or later.
2. Upgrade kubelets to 1.35 or later on all nodes.
3. Confirm the driver version with fallback logic is deployed.
4. Let the driver's DaemonSet rollout complete across all nodes.
5. Only now, set serviceAccountTokenInSecrets: true on the CSIDriver object.

Step 5 has a trap that has bitten combined manifests: if your driver DaemonSet and CSIDriver object live in the same manifest or Helm chart, you need two separate updates. Ship the fallback driver first, wait for the rollout, then flip the flag in a second change. Flip the flag while old driver pods are still running anywhere and volume mounts fail on those nodes — the old pods only look in volume_context, and the tokens are no longer there. The failure mode is not subtle (mounts error), but it is exactly the kind of thing a single helm upgrade that bumps both objects at once will produce, on a Friday, on the node that happened to roll last.

Note the asymmetry the sequence protects: new drivers tolerate old clusters (fallback covers them), but old drivers cannot tolerate the flipped flag. Every step before step 5 exists to eliminate old-driver pods from the fleet before the flag makes them unable to mount.

Audit your own fleet​

For a self-hosted fleet — Cluster API-managed nodes on owned hardware, running whatever CSI provisioners back tenant Postgres and build caches — the work is an audit, not an upgrade. The upgrade only enables the option; the exposure closes per driver, when that driver's fallback is deployed and its flag is flipped.

Walk every driver the first command in this post printed and answer three questions:

• Does it set tokenRequests? If yes, its tokens currently travel in volume_context unless the flag is already true. Don't assume only the two CVE names matter — any driver can declare TokenRequests, including storage drivers you installed for entirely un-secret reasons that later gained workload-identity features.
• Has its fallback shipped? Check the driver's release notes for secrets-field support and confirm the deployed DaemonSet is at least that version on every node — recall that one lagging node is what turns a flag flip into mount failures.
• Are tokens already in your log pipeline? The grep from the top of the post checks live driver logs, but if you ship logs to central storage, search there too. A token that was logged last month is a credential that lived outside its intended audience for a month, whatever the flag says today.

Then treat the answer as tenant-isolation surface, not driver trivia. On a multi-tenant platform, the service account token a CSI driver mints for one tenant's volume is that tenant's workload identity — and volume_context put it one verbose log level away from anyone with log access. "Where do workload-identity tokens travel" deserves the same review rigor as "who can read etcd," because when the answer was "a map the sanitizer ignores," the blast radius of log access quietly included vault access.

The rule going forward fits in one sentence: tokens travel in the secrets field, or you are one -v flag away from a leak. Audit the drivers, ship the fallback, flip the flags — in that order.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.