Stop Copying Registry Passwords Into Every Namespace: Kubernetes 1.34 ServiceAccount Image Pulls in Practice

The most over-shared credential in a multi-tenant Kubernetes cluster is the one that unlocks every tenant's container images. Today it usually lives as a long-lived imagePullSecret, copied into namespace after namespace, readable by anything with Secret read access, and rotated approximately never because rotation means touching every namespace at once. One self-hosted-registry project puts the arithmetic bluntly: 200 namespaces times 3 Harbor projects equals 600 Secrets to provision and remember.

Kubernetes 1.34 offers a way out. Service Account Token Integration for Kubelet Credential Providers — KEP-4412 — graduated to beta and is enabled by default, letting the kubelet mint a short-lived, pod-bound ServiceAccount token at pull time and hand it to a credential-provider plugin that exchanges it for registry credentials. No static password ever lands in a tenant namespace. This post is the migration walkthrough a self-hosted, multi-tenant git-push PaaS needs before deleting a single secret: what the registry must support, how to bind ServiceAccounts to pull identity, how rotation and caching behave, which failure modes to test, and why the whole exercise is worthless unless the token audience and repository scope stay tenant-specific.

The whole migration in one paragraph: upgrade kubelets to 1.34 or later so the KubeletServiceAccountTokenForCredentialProviders gate is on; confirm your registry (or a broker in front of it) can validate a Kubernetes-issued OIDC token and map the workload's identity to per-repository pull rights; configure tokenAttributes on your credential provider with a dedicated audience, a lifetime-matched cacheType, and the annotation keys carrying each tenant's identity pointer; grant kubelets the RBAC right to mint tokens for that audience; prove every failure mode in the table below fails closed; and only then remove the static secrets — keeping one shared role or robot account across tenants anywhere in this chain re-creates the exact blast radius you set out to eliminate.

What 1.34 actually changed, and the pull flow it enables​

The timeline matters because kubelet support and provider support did not arrive together. The feature entered alpha in 1.33, graduated to beta in 1.34 with the gate on by default, and remains beta in the current documentation. The beta renamed the gate (alpha's ServiceAccountTokenForKubeletCredentialProviders became KubeletServiceAccountTokenForCredentialProviders) and made cacheType a required field — a breaking change for alpha pilots. And "beta in Kubernetes" does not mean "supported by your provider": AWS notes the ECR credential provider only added full support, including fallback to the node role, in v1.35, so EKS clusters need 1.35 or later end to end.

The flow itself, per the beta announcement, runs like this. When an image is not present locally, the kubelet checks its credential cache, mints a ServiceAccount token bound to the pod's ServiceAccount with the configured audience, and passes that token — plus any configured ServiceAccount annotations — to the credential-provider plugin over stdin. The plugin exchanges the token for registry credentials and returns them; the kubelet caches them per its cacheType strategy, pulls the image, and records the ServiceAccount coordinates (namespace, name, UID) alongside the cached image. When the image is already cached, the kubelet verifies the requesting pod's ServiceAccount matches the recorded coordinates exactly, and re-pulls under the new identity when it does not.

One constraint shapes everything downstream: the kubelet cannot send ServiceAccount tokens to registries directly. The plugin must transform the token into the username-and-password-shaped credentials registries expect. There is no bearer-token channel in CredentialProviderResponse, which is why every working provider — ECR, GCR, ACR, and the community Harbor bridges — returns Basic-auth-shaped credentials.

What your registry must support​

The registry side has three jobs: validate the Kubernetes-issued token, map the workload identity to pull rights, and return short-lived credentials the kubelet can use. How those jobs get done splits cleanly between cloud and self-hosted registries.

Two details from these implementations deserve emphasis. First, AWS documents two-layer enforcement explicitly: the team IAM role scopes who the pod authenticates as, and the repository policy decides what that identity can access — neither layer alone suffices. Second, the Harbor bridge is alpha software, verified end-to-end on kind (Kubernetes v1.37) plus Harbor 2.x, bridging the gap until upstream Harbor lands native OIDC trust-policy support (goharbor/harbor#17520). If your registry is neither cloud-backed nor Harbor-with-a-bridge, that row of the table is code you have to write: a plugin plus a broker validating tokens and minting scoped credentials. KEP-4412 gives you the kubelet half; the registry half is still yours.

Binding ServiceAccounts to pull identity​

The kubelet half is configured in CredentialProviderConfig. A minimal tokenAttributes block looks like this:

providers:
  - name: my-registry-provider
    matchImages:
      - "registry.example.com/*"
    defaultCacheDuration: "10m"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    tokenAttributes:
      serviceAccountTokenAudience: "registry.example.com"
      cacheType: "ServiceAccount"
      requireServiceAccount: true
      requiredServiceAccountAnnotationKeys:
        - "registry.example.com/pull-identity"

Each field is load-bearing. serviceAccountTokenAudience sets the aud claim kubelet mints into the token, and exactly one audience is allowed per provider entry — supporting two audiences means configuring the provider twice under two executable names (a symlink suffices). requireServiceAccount: true keeps pods without a ServiceAccount away from the plugin; setting it false exists for static pods. requiredServiceAccountAnnotationKeys names the annotations the plugin reads as identity pointers — the ECR role ARN, a Harbor project binding — and if any listed key is absent, the kubelet skips the plugin and returns an error, so treat that list as an admission contract: your tenant-provisioning controller must stamp those annotations, or deploys fail at the pull.

The audience also needs an RBAC grant, or nothing works. Since 1.33 the kubelet may only mint tokens for audiences it is authorized for, via the synthetic request-serviceaccounts-token-audience verb:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubelet-registry-audience
rules:
  - verbs: ["request-serviceaccounts-token-audience"]
    apiGroups: [""]
    resources: ["registry.example.com"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-registry-audience
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubelet-registry-audience
subjects:
  - kind: Group
    name: system:nodes
    apiGroup: rbac.authorization.k8s.io

AWS's walkthrough stresses the ordering: this RBAC must exist before nodes join with the new configuration. Without it the kubelet cannot project tokens, every image pull fails, and nodes never become Ready. That is your first failure-mode test: apply the config without the grant in staging and watch the cluster refuse to schedule.

Rotation, caching, and revocation​

Projected ServiceAccount tokens live about an hour by default, and the credentials derived from them can be equally short-lived. That ephemerality is the security win — a leaked pull credential dies within the hour instead of living in etcd until someone remembers to rotate it — but it puts real pressure on two behaviors: caching and long pulls.

Caching is governed by cacheType, and the choice is a lifetime question, not a performance tweak. Token caches returned credentials per token: correct when the registry credential dies with the input token. ServiceAccount caches per ServiceAccount identity: correct only when the returned credential is valid for every pod sharing that ServiceAccount. Pick ServiceAccount for credentials that are actually token-bound and the kubelet will happily serve a dead credential from cache; pick Token for SA-wide credentials and you multiply provider invocations for no benefit. The documentation calls Token the most conservative option — start there unless your provider contract says otherwise in writing.

Two harder limits come straight from the KEP's caveats. The kubelet hands credentials to the runtime once, when a pull starts; nothing refreshes them mid-pull. So an image pull that runs longer than the credential lifetime can fail partway through — test your largest tenant image over your slowest node link, not a cached nginx over the datacenter LAN. And streaming-style registries that fetch image content lazily after the container starts — GKE image streaming is the named example — are incompatible outright: content fetched after the pull returns has no valid credential left.

Revocation works through identity, not credential expiry. With image-pull credential verification in play, the kubelet records each pulled image's ServiceAccount coordinates, and deleting plus recreating a ServiceAccount changes its UID, invalidating every cached image authorization tied to the old identity. That is a useful incident lever — delete and re-create the SA and previously pulled images become inaccessible until re-pulled — but rehearse it before you need it, because it also re-pulls everything.

Failure modes to test before you delete a single secret​

Every row below is a test to run in staging with the static secrets still in place. Each must fail closed — denied pull, clear error — never open.

Row 9 is the trap most likely to catch a self-built broker: the kubelet half can be perfect while the provider maps every ServiceAccount token to the same admin-level registry credential. Functionally everything pulls; from a security standpoint you have re-implemented the shared secret with extra steps. The AWS sample walkthrough's verification matrix — team A to team B denied, team B to team A denied, shared and baseline images allowed — is the shape of the test to copy.

Row 5 deserves a capacity note. Per-SA image-cache isolation means a popular base image gets pulled once per ServiceAccount per node rather than once per node. On a densely packed multi-tenant node with dozens of tenant ServiceAccounts, that is a real multiplier on registry egress and pull latency during rollouts. Measure it with your tenant density before promising faster deploys.

The scoping rule: audience and repository scope stay tenant-specific​

This is the point the TODO spec ends on, and it is worth stating as a rule rather than a recommendation: removing the shared static secret only reduces blast radius if nothing in the new chain is secretly shared. Two specific sharing points matter.

First, the audience. The aud claim binds a minted token to one intended recipient. A single platform-wide audience that every broker trusts for every repository turns the token into a bearer pass for the whole registry — any workload's token validates anywhere. Scope audiences per trust boundary (per registry at minimum; per project or tenant where your broker supports it) and enforce the audience check on the registry side, the way the Harbor bridge's trustPolicy.audience match does. Minting a narrow audience that nobody verifies is theater.

Second, the repository scope. Issuing a per-workload token answers "who is pulling" but says nothing about "what may it pull" — that decision lives in the repository policy (ECR), the robot's project grants (Harbor), or your broker's mapping table. If that mapping grants every tenant identity pull rights on every repository, you have per-pod authentication with cluster-wide authorization, and a compromised pod in tenant A reads tenant B's images exactly as before. The fix is the two-layer pattern from the ECR walkthrough: a distinct pull identity per tenant (IAM role, robot account, broker mapping entry) and a repository-side policy that denies cross-tenant reads. Verify both layers independently — swap the identity and confirm denial, then widen the policy and confirm the identity alone does not save you.

Rollout plan for a git-push PaaS​

Ship it the way you would any credential-system migration: narrow, observable, and reversible until the evidence says otherwise.

1. Pick one pilot tenant namespace. Stand up the provider, audience, and repository scoping for that tenant only. Keep its imagePullSecret in place — the provider path is additive.
2. Force the provider path and watch. Reschedule a pod and confirm in provider logs, broker audit trails, and policy evaluations that the pull went through the plugin. Then schedule a second tenant's pod against the same image on the same node and confirm it re-pulls under its own SA.
3. Run the failure-mode table. All nine rows, in staging first, then against the pilot tenant in production. Row 8 and row 9 are the release gates.
4. Expand tenant by tenant. Each new tenant gets its own pull identity and repository grants before its first provider-path pull. Add a provisioning check: no identity annotation, no namespace.
5. Delete secrets last, and watch pull metrics. Remove the pilot tenant's imagePullSecret, monitor pull latency and error rates across a full rollout cycle, then repeat per tenant. Document the provider's fallback — ECR falls back to the node role when the annotation is absent, which suits system pods but is a hole if a tenant can strip its own annotation.

The end state is worth the caution: tenant namespaces that contain zero registry credentials, pull rights that expire within the hour, and a compromised pod with nothing to exfiltrate. That is a meaningfully smaller blast radius than 600 static Secrets — as long as the audience and the repository scope stayed tenant-specific all the way down.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.