Skip to main content

SOPS with Age vs Sealed Secrets: What GitOps Secrets Cost to Rotate on a Cluster API Fleet

12 min readDora NodaDora Noda
Share
On this page

In February 2026, the most boring part of Bitnami Sealed Secrets broke. Not the crypto — the /v1/rotate endpoint, the housekeeping API teams call when they are being responsible. A logic flaw let an attacker submit a victim's namespace-scoped SealedSecret with one injected annotation and get back a re-encrypted secret scoped cluster-wide, ready to unseal anywhere (CVE-2026-22728, fixed in v0.36.0).

Nobody lost a secret by committing plaintext. They were bitten by the rotation path — the part of a GitOps secrets tool you evaluate last and pay for first.

That is the right lens for this whole comparison. If you run a Cluster API fleet that declares tenants in Git — Flux-reconciled Clusters, MachineDeployments, app manifests — and each tenant needs a Postgres password, a TLS key, and a provider token without committing plaintext, you have two Kubernetes-native answers. SOPS with age: envelope-encrypt the files, commit the ciphertext, let Flux's kustomize-controller decrypt at apply time. Bitnami Sealed Secrets: encrypt through the controller's RSA public key with kubeseal, commit a SealedSecret CRD, let the controller decrypt it in-cluster.

Both work. They send you very different bills at rotation, audit, and disaster-recovery time.

Here is the verdict up front, for a fleet of N tenants on owned hardware with no cloud KMS to lean on:

DimensionSOPS + ageSealed Secrets
What Git holdsEncrypted YAML files, keys in cleartextSealedSecret CRDs, fully opaque blobs
Rotate one tenant's DB passwordEdit 1 file, re-encrypt, commitRe-seal 1 secret with kubeseal, commit
Rotate the master key at N=50One find … | xargs sops updatekeys loop, offline, no cluster access neededController auto-renews every 30 days; then kubeseal --re-encrypt per secret, each call needing a live controller
Audit (git log)Ciphertext diffs — you see that a secret changed, not whatStructured CRD diffs plus scope annotations — you see what kind of object changed and how widely it can unseal
Management cluster diesRestore one age private key from offline backup into one Secret; Flux re-decrypts everythingWithout a backup of the controller's sealing key, every SealedSecret is unrecoverable — re-seal all N from plaintext originals
Blast-radius controlRecipient lists per file (.sops.yaml rules)Strict / namespace-wide / cluster-wide scopes per secret

The short version: SOPS wins on bulk, offline, machine-generated tenant files; Sealed Secrets wins where every tenant is already a Kubernetes resource with per-secret scope needs. The rest of this post is the evidence — including the fleet-scale math behind the table.

The setup both tools assume

Picture the fleet this TODO-list item was written for: a self-hosted PaaS on Cluster API, Flux reconciling a Git repo that declares everything — Clusters, MachineDeployments, per-tenant app manifests — running on owned Hetzner machines. Tenant onboarding is a generator: it mints a Postgres role and password, a TLS keypair, and a provider token, and writes them into the repo. The one hard rule is that Git never holds plaintext, because Git history is forever and repo access is wider than production access.

The owned-hardware constraint matters more than it looks. On AWS you would reach for KMS-backed envelope encryption without thinking. On your own machines there is no KMS, which is exactly why this comparison is SOPS with age rather than SOPS with cloud KMS: age (FiloSottile/age, v1.3.1) is a file-encryption tool with no server, no account, and no network — an X25519 keypair you generate with age-keygen and back up like any other root credential.

SOPS itself (getsops/sops, v3.13.x, a CNCF Sandbox project with ~22k stars) is the envelope format around it: it encrypts each YAML value with a per-file data key, wraps that data key for every age recipient listed in .sops.yaml, and leaves the YAML keys in cleartext so Kustomize and Flux can still parse the file's structure. Both halves run fully offline, which is the whole point on hardware you own.

The Flux half is native, not a plugin. Flux's kustomize-controller decrypts SOPS-encrypted manifests at apply time using an age private key you install once as a Secret named sops-age (key age.agekey) in flux-system — this is documented Flux behavior, not a sidecar. Sealed Secrets is equally native-shaped but inverted: instead of the sync engine decrypting, a controller in the cluster holds an RSA private key, and kubeseal encrypts secrets against the matching public certificate, which anyone can fetch without cluster credentials. That inversion — decrypt-at-sync versus decrypt-by-controller — is the root of every row in the verdict table.

How each one actually works

With SOPS, the developer loop is file-shaped. You write a normal Kubernetes Secret manifest, run sops -e -i tenants/acme/postgres.yaml, and commit the result: same keys, ciphertext values, plus an unencrypted sops metadata block recording which age recipients can open it. Flux pulls the repo, kustomize-controller sees the sops block, decrypts with the in-cluster age key, and applies a plain Secret.

Nobody outside the cluster ever holds anything but ciphertext, and anyone with the age public key can encrypt a new tenant file without being able to read existing ones. That asymmetry is what makes generator-driven tenant onboarding safe to run in CI.

With Sealed Secrets, the loop is API-shaped. You write the same Secret manifest, run it through kubeseal (pointed at the controller, or with a fetched certificate for offline use), and commit a SealedSecret CRD: the name, namespace, and encrypted payload bound together by the controller's RSA key. The controller watches SealedSecrets, decrypts them, and writes plain Secrets.

The binding is the feature: a strict-scope SealedSecret (the default) only unseals into the exact name and namespace it was sealed for, so a leaked blob cannot be retargeted at another tenant's namespace. Namespace-wide and cluster-wide scopes relax that binding when you genuinely want one sealed value reused across tenants.

One operational detail that surprises teams: the Sealed Secrets controller renews its sealing key automatically every 30 days, keeping old keys around so previously sealed secrets still unseal. Key renewal is not secret rotation — the project README says so explicitly — but it does mean the "latest key" is a moving target, and kubeseal --fetch-cert output from last quarter seals against a key that is no longer newest. SOPS has no equivalent background motion: recipients change only when you edit .sops.yaml.

Rotation: the bill the title promised

There are two different rotations and they cost opposite amounts under each tool. Level one is rotating a credential: tenant Acme's Postgres password leaks, you mint a new one. Under both tools this is one object, one command, one commit — SOPS re-encrypts one file, Sealed Secrets re-seals one CRD. Call it a tie, seconds each, and the only real difference is muscle memory.

Level two is rotating the master key, and this is where fleet scale bites. Say the age private key or the sealing key must be replaced — an engineer leaves, a backup is suspected exposed — across 50 tenants with 3 secrets each, 150 encrypted objects.

Under SOPS the runbook is a loop you run on a laptop, against no cluster at all:

bash
age-keygen -o age-new-key.txt
# add the new public key to .sops.yaml, then:
find tenants -name '*.enc.yaml' -print0 | xargs -0 -n1 sops updatekeys -y
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=age-new-key.txt --dry-run=client -o yaml | kubectl apply -f -
git add -A && git commit -m "rotate age recipient" && git push

Cost units: 150 files × one updatekeys invocation each (seconds per file, parallelizable), one commit, one in-cluster Secret update. The old private key must be present during the loop — updatekeys decrypts the data key with an old recipient and re-wraps it for the new list — but nothing talks to Kubernetes until the final Secret update, and Flux reconciles the re-encrypted files on its next pull with zero workload restarts if the plaintext values are unchanged. You can rehearse the entire rotation on a plane.

Under Sealed Secrets the runbook needs a live controller for every object. The controller's own key renewal is automatic, but moving existing SealedSecrets onto the new key is per-secret work with kubeseal --re-encrypt, which calls back into the cluster:

bash
kubeseal --fetch-cert --controller-name sealed-secrets \
  --controller-namespace kube-system > new-cert.pem
for f in tenants/*/postgres-sealed.yaml; do
  kubeseal --re-encrypt --cert new-cert.pem -f "$f" -w "$f"
done
git add -A && git commit -m "re-encrypt to latest sealing key" && git push

Cost units: 150 files × one controller round-trip each, serially gated on API availability, and the rotation cannot even start if the management cluster is down. The redeeming feature is that old keys are retained, so un-rotated secrets keep working — stragglers degrade gracefully rather than breaking. Still, at fleet scale, "needs a live cluster and touches every object through it" versus "a local loop plus one Secret update" is the single largest operational gap between the two tools, and it grows linearly with tenant count.

Then there is the caution the February CVE stapled to this topic. CVE-2026-22728 was a scope-widening flaw in Sealed Secrets' /v1/rotate HTTP endpoint: the handler derived the new secret's sealing scope from untrusted annotations on the submitted object, so injecting sealedsecrets.bitnami.com/cluster-wide: "true" into a victim's namespace-scoped secret returned a cluster-wide blob decryptable anywhere (GHSA-465p-v42x-3fmj, fixed in v0.36.0 — upgrade if you run anything older). The lesson generalizes beyond one bug: a rotation path that accepts objects and re-emits them with new trust properties is privileged parsing code, and a controller-based design has that code reachable over HTTP while a file-based design keeps it on the operator's laptop. Prefer strict scope everywhere, treat the rotate endpoint as sensitive, and count this as a point for the architecture with fewer network-reachable key-handling surfaces.

Audit and recovery: what git log shows, and what a dead cluster costs

On audit, the tools trade visibility for structure. A SOPS commit diff shows that tenants/acme/postgres.yaml changed and roughly how many values rotated, but every value is an opaque blob — you cannot tell a password rotation from a username change, and git blame tells you when ciphertext changed, never why. What you gain is structural honesty: keys stay in cleartext, so reviewers see exactly which fields exist and which rotated. A SealedSecret diff is similarly opaque on values but richer on metadata: the CRD shows scope annotations, template labels, and which controller key generation sealed it, so policy checks ("no cluster-wide secrets in tenant namespaces") can run as admission control or CI lint against committed files.

If auditability-to-a-reviewer matters more than auditability-to-a-machine, Sealed Secrets' object model reads better; if you want git log to answer "which tenants rotated this quarter," both need a commit-message convention, because neither puts answers in diffs.

Recovery is where the designs diverge hardest, so run the thought experiment both ways. Your management cluster is gone — etcd lost, nodes unreachable, the thing that reconciles everything is a pile of rented metal.

Under SOPS, the recovery bill is: fetch the age private key from offline backup (the USB stick, the vault envelope, whatever your root-credential ceremony is), rebuild any cluster, kubectl create secret generic sops-age --from-file=age.agekey=... in flux-system, point Flux at the same repo. Every one of the 150 encrypted files decrypts exactly as before, because decryption never depended on any cluster-held state. One key, one Secret, total recovery.

Under Sealed Secrets, the recovery bill depends entirely on whether you backed up the controller's sealing key — the sealed-secrets-key Secret in the controller namespace, which lives in the etcd you just lost. If you backed it up (Velero, a sealed copy in another vault, a printed QR code in a safe — pick your ceremony and actually rehearse it), restore it into the rebuilt controller and all 150 SealedSecrets unseal as before. If you did not, every SealedSecret in Git is cryptographically unopenable, permanently, and your recovery is re-sealing all 150 from plaintext originals you hopefully still have somewhere that is not the dead cluster. This is the failure mode teams discover once: SOPS degrades to "restore one static key," Sealed Secrets degrades to "re-run onboarding for every tenant."

Either is survivable if drilled; only one of them is survivable if forgotten.

A smaller 2025 footnote: Bitnami retired its classic GitHub-Pages Helm hosting, so bitnami-labs.github.io chart URLs 404 and pinned sealed-secrets installs broke (charts moved to OCI; images continue on docker.io/bitnami). Not a strike against the encryption design — but the controller path inherits a supply chain (chart repo, registry, controller deployment) the file path does not have.

Verdict: bulk files go SOPS, CRD-native tenants go Sealed Secrets

Decide by the shape of your tenant lifecycle, not by brand loyalty:

  • Tenant onboarding is a generator writing bulk files — dozens of near-identical credential files minted by automation, rotated in sweeps, on hardware with no KMS. Use SOPS with age. File-level encryption matches file-shaped workflows, recipient rotation is an offline loop, and recovery is one backed-up key. This is the PaaS case: the platform manufactures the secrets, so the file-shaped tool wins.
  • Every tenant is already a Kubernetes resource and operators hand-craft secrets per namespace, with real variance in who may unseal what. Use Sealed Secrets. Per-secret scopes express "this blob opens only in this namespace" as committed, reviewable API objects, and the controller model means developers seal without ever touching a private key. This is the platform-team case: humans manage the secrets, so the API-object tool wins.
  • Either way, write down the two ceremonies on day one: the master-key rotation runbook (with its fleet-scale unit cost: files × commands, and whether a live cluster is required) and the cluster-death recovery drill (where the age key or sealing-key backup lives, and when you last rehearsed restoring it). Both tools survive these events when drilled and embarrass you when not — the difference is only how steep the undrilled bill is.

The honest third option is a secrets store rather than secrets in Git: External Secrets Operator syncing from Vault removes rotation-from-Git entirely, at the price of operating the store itself — a real dependency on owned hardware. Until a fleet is ready for that, encrypted-in-Git is the pragmatic default. Just remember February's CVE the next time a rotation endpoint looks boring: the tool you pick is the rotation path you marry.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex