Kubernetes 1.37 Made Pod Certificates GA: The 4-Test Signer Harness to Run Before You Kill Static Credentials

Static service-account tokens are the worst kind of credential: long-lived, bearer-style, and scattered across every pod filesystem on your fleet. Kubernetes 1.37 — Garhwal, released August 26, 2026 — finally gives you a native way out. Pod Certificates and Cluster Trust Bundles went stable in certificates.k8s.io/v1, enabled by default, with kubelet-issued PodCertificateRequest objects, short-lived X.509 certs, and projected trust anchors. No feature gates. No beta API dance.

But here is the part the release notes undersell: Kubernetes ships the machinery, not the policy. There is still no built-in signer in core — you install a third-party controller, and that controller decides who gets a certificate. If you swap static tokens for pod certs without proving your signer policy first, you have traded a known-weak credential for an untested identity system. Run the harness below first.

Here is the whole harness in one table. Four tests, each with a pass criterion. If all four go green per tenant signer, you can start retiring static credentials. Anything red means stop.

The rest of this post works the table row by row, with the YAML and commands to run each test on a Cluster API fleet.

How the flow actually works​

The architecture has three actors: your app, kubelet, and the signer controller. The feature blog walks the sequence chronologically, and it is worth internalizing because every test below pokes one step of it:

1. The pod is scheduled. Kubelet finds the podCertificate and clusterTrustBundle projected volume sources in its spec.
2. Per podCertificate source, kubelet generates a private key per keyType, then creates a PodCertificateRequest addressed to the signerName in the source. The request carries podName, podUID, serviceAccountName, nodeName, and maxExpirationSeconds.
3. The signer controller watches PodCertificateRequest objects for its signer name, decides whether to issue, and on approval fills status.certificateChain plus status.beginRefreshAt — the timestamp telling kubelet when to start refreshing.
4. Kubelet writes the key and chain to the container filesystem. Per clusterTrustBundle source, it collects every matching ClusterTrustBundle by signer name and label selector, unifies the PEM anchors, stably reorders them so apps cannot depend on ordering, and writes the bundle file.
5. Your app starts, reads key, chain, and trust anchors from disk, and serves or dials mTLS. From here on, kubelet keeps both file sets fresh: bundle files track ClusterTrustBundle changes, and each certificate refreshes past beginRefreshAt. Your app must notice via inotify or polling — kubelet will not restart it for you.

A minimal pod spec shows both halves side by side:

volumes:
  - name: workload-cert
    projected:
      sources:
        - podCertificate:
            signerName: example.com/my-tenant-signer
            keyType: ECDSA_P256
            maxExpirationSeconds: 86400  # 24h default if omitted
            credentialBundlePath: workload.bundle.pem
            # single file: key + chain together, subscribe to one path
        - clusterTrustBundle:
            signerName: example.com/my-tenant-signer
            labelSelector:
              matchLabels:
                trust.example.com/managed: "true"
            path: trust-bundle.pem

Two details in that YAML carry the whole security model. signerName is the policy boundary — whoever controls that signer controls identity for every pod that names it. And credentialBundlePath versus split key/cert files is the reliability boundary, which Test 4 covers. Keep both in mind as we test.

Test 1: signer authorization — prove the wrong pod gets nothing​

Pod certificates are pluggable by design. JWT service-account tokens come in one flavor with standardized claims; X.509 certs vary by purpose, extensions, and signer, so Kubernetes built common kubelet machinery plus a signer interface where many signers can coexist in one cluster. Flexibility is the point — and the risk. Your test job is to prove each signer says no correctly.

Three checks make up this test. First, RBAC. The reference docs require permissions on both the PodCertificateRequest type and the specific signer name, so create two tenant signers — say example.com/team-a and example.com/team-b — and assert a pod naming signer A cannot be fulfilled by controller B, and a controller scoped to B cannot read or approve A's requests. Run it cross-namespace too: a pod in team-b requesting example.com/team-a must stay pending or be explicitly denied, never issued.

Second, node isolation. The kube-apiserver's built-in node-restriction admission plugin enforces that one compromised node cannot request certificates for pods not scheduled to it. Test it the adversarial way: craft a PodCertificateRequest with a nodeName that does not match the requesting node's identity and confirm the apiserver rejects it. On a Cluster API fleet this matters more than on managed Kubernetes, because your node identity pool is yours to misconfigure — a sloppy system:nodes group binding can quietly widen this.

Third, and most commonly skipped, unverifiedUserAnnotations. Kubelet copies the pod author's userAnnotations verbatim into spec.unverifiedUserAnnotations on the request, unvalidated by design — validating them is the signer's job.

Demo signers treat them as convenience config (CN, SANs, duration hints). Production signers must treat them as attacker input. Your negative test: submit a request whose annotations ask for a broader identity than the pod owns — a different service account's SPIFFE ID, a wildcard DNS SAN, a longer lifetime — and confirm the signer clamps or denies it.

The cross-cluster mTLS demo README states this explicitly, and community signers like RafPe/pod-certificate-signer document the same contract. If your signer honors annotations blindly, Test 1 fails and you do not proceed.

Pass criterion: cross-tenant issuance denied, wrong-node request rejected, annotation-escalation clamped. Log all three denials — auditors will ask.

Test 2: trust-bundle selection — prove each workload trusts exactly its CA​

Issuance is half the mTLS story. The other half is the trust anchor: the CA bundle your app uses to verify the peer. ClusterTrustBundle objects carry those anchors, published by the signer controller — current and previous CAs across rotations — and kubelet projects the matching set into each pod by signerName plus label selector.

The failure mode here is leakage or staleness. In a two-signer cluster, mount signer A's bundle into workload A and signer B's into workload B, then dump both files and diff. Workload A must contain exactly A's anchors, nothing from B. Label selectors are the usual leak vector: an overly broad selector like "all bundles" silently merges trust, so test the selector you actually ship, not the demo's permissive one. Kubelet helpfully unifies and stably reorders certificates across all matching bundles, which means a mis-selected bundle does not even look wrong — the file is still valid PEM, just trusting too much.

Then rotate. Publish a new CA through the signer, keep the old bundle briefly alongside the new one (the standard overlap pattern the signer README documents), and watch both workloads' trust-bundle.pem paths. Kubelet updates projected bundle files in place as ClusterTrustBundle contents change — no pod restart.

Your app must pick up the change with inotify or a poll loop. Assert the new anchor appears within your file-sync window and the peer presenting a new-CA cert verifies while old-CA certs still verify during overlap. Then remove the old CA and assert old certs stop verifying.

If your app reads the bundle once at startup — the classic GOWEB_MTLS_CLIENT_CA-at-boot mistake — this test catches it before production does.

On a multi-tenant PaaS, run Test 2 per tenant signer, not once globally. Bundle selectors are per-workload configuration, and one tenant's correct selector does not prove another's.

Test 3: lifetime and refresh — prove 1 hour, 24 hours, and the cap​

This is the test the TODO spec names most directly, and the one most teams under-test by checking a single lifetime. Do not. The API surface has four numbers and your harness should touch all of them: the apiserver rejects anything under 3600 seconds (1 hour); maxExpirationSeconds defaults to 24 hours when omitted; kubernetes.io signers never issue longer than 24 hours; other signers may go up to 91 days. Those bounds come from KEP-4317 and the v1.37 docs, and they are the sensitivity range the audit gate demanded.

Run three issuance cases against your real signer. Request ~1 hour and confirm openssl x509 -enddate lands near it, then watch status.beginRefreshAt arrive and kubelet re-issue past that timestamp with up to five minutes of randomized jitter.

Request the default by omitting maxExpirationSeconds and confirm ~24 hours. Request the cap — 24 hours against a core-style signer, or your documented max up to 91 days against a third-party signer — and confirm the signer clamps rather than honors an over-long ask.

Each case asserts two things: the enddate matches the clamped request, and a refresh actually fires at beginRefreshAt plus jitter rather than silently expiring.

Why the range matters for a PaaS: 24-hour certs are the steady state the Kubernetes authors intend — core signers will eventually include a SPIFFE client-identity signer filling today's service-account-JWT role and a server-TLS signer for Service DNS names. Short-lived certs near the 1-hour floor stress your refresh path the way an outage will: jitter, apiserver load, signer downtime. The 91-day ceiling tests the opposite temptation — an operator "temporarily" lengthening lifetimes to dodge refresh bugs, which re-creates the static-credential problem with extra steps. Your harness should make the refresh path so boring at 1 hour that nobody reaches for 91 days.

Record beginRefreshAt versus actual re-issuance timestamps across at least ten refreshes. A signer that sets the field but drifts, or a kubelet that refreshes late under node pressure, shows up here first.

Test 4: reload behavior — prove rotation never drops a connection​

Automatic rotation is built in; handling it is your app's job. The feature blog is blunt about this: applications must properly handle rotation, and the simplest way is kubelet's single-file credential bundle. When key and chain live in one file, the app subscribes to one path's inotify events, re-reads, and re-configures TLS. When they live in separate files, the app must manage the race of reading mid-rotation — key from the new generation, chain from the old — and most apps do not.

Your test is a loop, not a snapshot. With a short lifetime from Test 3 (1 hour, or shorter if your signer allows the 1-hour floor with early beginRefreshAt), run a client-server pair doing continuous mTLS through the projected files, reloading on inotify, and count TLS failures across three full rotations. Pass is zero failures with the single-file bundle. Then, if you ship split files anywhere, repeat and watch the torn-read errors appear — that delta is the artifact that justifies standardizing on credentialBundlePath fleet-wide.

Also test the trust side mid-connection: rotate the CA during active connections (from Test 2's rotation) and confirm long-lived connections either re-verify cleanly or recycle without 500s, depending on your stack's session-resumption behavior. Go's TLS library, for example, needs the reloaded config actually swapped into the listener — the Tinycert project's spiffefsd helper and its SPIFFE client/server example exist precisely because this wiring is easy to get almost right. "Almost" drops connections at 3 a.m. when the 24-hour refresh fires.

The rollout checklist for a Cluster API fleet​

With four green tests per signer, rollout is mechanical. Install a real signer — Tinycert is the documented starting point with its tinycert-service DNS-SAN signer and tinycert-spiffe namespace-and-service-account signer, explicitly a toy to build on rather than a production answer. Evaluate production signers the way RafPe's documents its contract: native PodCertificateRequest handling with no sidecars, per-workload identity fields, CA hot-reload without restarts, and trust bundles published across rotations. Wire ValidatingAdmissionPolicy around signer selection if tenants can edit pod specs directly.

Then gate per tenant: run Tests 1–4 against that tenant's signer name and selectors, in a staging workload cluster managed by the same Cluster API management cluster that owns production. Only then stop mounting static service-account tokens for that tenant's workloads — and keep the projected SA token volume available during a transition window where both credential types verify, so rollback is a config flip rather than an incident. The prize is concrete: no more bearer tokens with indefinite lifetimes sitting on every node disk, replaced by 24-hour X.509 certs with signer-scoped trust that refresh themselves.

Longer term, expect core Kubernetes to ship the two obvious built-in signers — server TLS for Service DNS names and SPIFFE client identity — and treat that as the moment this harness becomes a conformance check rather than a custom build.

The fastest way to feel this working is to run it where you own the machines. Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.