Rotate the Database Password Without Shipping a Rebuild: A No-Redeploy Secrets Playbook for Git-Push Platforms

Somewhere in your incident runbook there is a line that reads "rotate the database password." On most self-hosted git-push platforms, executing that line also means rebuilding every app container, re-running the deploy pipeline, and holding your breath through a rollout — all to change one string. The password rotation is not the hard part. The redeploy it drags along is.

That coupling is a platform design choice, not a law of physics. This post is the playbook for breaking it: project the database credential into the pod as a mounted file instead of a baked-in environment variable, let a reload sidecar notice the change, and turn rotation into a control-plane API call instead of a git push. Here is the whole thing up front, then the mechanics, the gotchas, and a worked example you can copy.

The playbook, up front​

Four steps, in this order, every time:

1. Create overlap on the database side first. The new credential must be valid before any app process reads it. For Postgres that means a second role or a staged flip (see the gotcha section) — never a bare password swap with readers mid-flight.
2. Write the new value to the secret-of-record exactly once. Vault, your platform's secrets store, or a Kubernetes Secret — one write, through the control plane. No commit, no build.
3. Let projection propagate it. The pod mounts the secret as files, not env vars, so the kubelet syncs the new value onto disk within about a minute. The deploy pipeline stays idle.
4. Recycle connections, not containers. A reload sidecar watches the projected file and — on change — drains the app's connection pool or signals a config reload. Same pods, same image, new password.

The ordering rule behind all four steps fits in one sentence: the new credential must be valid before any app reads it, and the old credential must stay valid until every app has stopped reading it. Every rotation outage I have seen is a violation of one half of that sentence.

Why env vars force redeploys and mounted files don't​

A Unix process receives its environment once, at exec, as a block of copied memory. Nothing the platform does later — no dashboard edit, no API call — can reach into a running process and rewrite that block. Kubernetes is explicit about the consequence: environment variables sourced from a ConfigMap or Secret are resolved when the container starts, and updating the source object never touches running pods.

Mounted secrets play by different rules. When a pod mounts a Secret as a volume, the kubelet periodically syncs the volume's contents from the API server and swaps the new files in atomically. Update the Secret object and every pod mounting it sees the new value on disk, typically within about a minute, with no restart and no new image.

The subPath row is the trap. Mounting mountPath: /app/config/db-password with subPath: password gives you a tidy single-file path — and permanently opts that file out of kubelet updates, because a subPath bind-mount points at one inode the atomic swap never replaces. The Kubernetes docs call this out, but it still bites: if your rotation playbook "works on staging, never in prod," check for subPath first. Always mount the whole directory.

Three propagation paths, compared​

Projection gets the new value onto disk; something still has to make the app use it. Three patterns cover nearly every setup, and the reload sidecar is the one that asks least of your application code:

• (a) Mounted volume + reload sidecar (this playbook's default). A lightweight sidecar watches the projected password file — inotify or a checksum loop — and on change either hits the app's local reload endpoint (POST localhost:8080/-/reload-pools) or sends SIGHUP through a shared PID namespace. No app-code change beyond exposing a recycle hook; no pod restart at all.
• (b) External Secrets Operator pull. When Vault (or OpenBao, AWS SM, …) is the secret-of-record, an ExternalSecret with a refreshInterval re-fetches on schedule and lands the value in a plain Kubernetes Secret — from which the same volume projection takes over. End-to-end latency is refresh interval plus kubelet sync. ESO's VaultDynamicSecret goes further with short-lived database leases, where rotation becomes lease expiry rather than an event.
• (c) Secrets Store CSI auto-rotation. The CSI driver mounts the vault secret directly as a volume and re-fetches on republish. Real, upstream, and the shortest path from vault to pod — but auto-rotation is still alpha, off by default (--enable-secret-rotation), with a default 2-minute poll floor. Adopt it deliberately, not by accident.

One honest middle ground: Stakater Reloader watches Secrets and triggers a rolling restart when they change. That is a restart without a rebuild — no new image, no pipeline run — and for apps that genuinely cannot reload config it beats a full redeploy. But it still bounces every pod, so treat it as the fallback, not the playbook.

What "control-plane operation" means on a git-push PaaS​

Today's single-box UX makes the deploy pipeline the only delivery vehicle for config. Coolify injects environment variables at deploy time, so an edited value waits for the next build; Dokploy models the flow as save-environment then redeploy — two separate API calls, the second one mandatory. Rotating a database password through either dashboard is therefore, mechanically, a redeploy with a password edit attached.

The decoupled UX inverts that: a Rotate database password action — dashboard button, API call, or CLI — that generates the password, applies the database-side overlap, writes the secret-of-record, and returns. The build queue is never touched. Within a minute or two the sidecars fire, pools recycle, and rotation is done with zero builds enqueued.

If you build or operate the platform, three primitives make this UX possible: secret versioning (so N services rotate independently and roll back cleanly), a privileged rotation endpoint (audited, rate-limited — it writes to the database, not just the store), and a per-service reload hook contract (SIGHUP, HTTP endpoint, or pooler reload — the platform needs to know which one each service speaks).

The database-side gotcha: Postgres has one password per role​

Here is the happy-path killer. ALTER USER app WITH PASSWORD 'new' flips the credential atomically on the database side: from that instant, any new connection attempt with the old password fails. Established connections survive — Postgres authenticates at connect time, not per query — but pools churn, pods reschedule, and poolers re-authenticate, so "existing connections are fine" degrades into a rolling brownout exactly as fast as your infrastructure recycles connections.

With a single password there is no ordering that is safe by itself: flip the database first and old-password readers break on next connect; distribute first and new-password readers break until the flip. You need overlap, and there are three honest ways to get it:

1. Two-role leapfrog (safest, recommended). Keep roles app_a and app_b with identical grants. Rotation means: set the password on the idle role, flip the secret-of-record to it, wait out propagation plus a connection-drain window, then retire the old role's password. Overlap is structural — at no instant is a valid reader holding an invalid credential. The cost is grant hygiene: both roles must stay identically privileged, which is a migration-discipline problem, not a downtime problem.
2. Staged single-password (smallest window, nonzero risk). Flip the secret-of-record, let the sidecars drain pools onto the new password immediately — connections fail fast for seconds, not minutes — then flip the database password inside that window. This compresses the outage to seconds but cannot eliminate it; use it only with low-churn pools and a maintenance-tolerant workload, and never call it zero-downtime.
3. Dynamic short-lived credentials (best long-term, most machinery). The Vault/OpenBao database engine mints per-lease credentials via ESO's VaultDynamicSecret; rotation is lease expiry, and the app re-authenticates on lease boundaries. No shared password exists long enough to leak. The price is running the vault, the engine, and lease-aware connection handling — worth it past a handful of databases, overkill for one Postgres next to the app.

One pooler note, since most production Postgres sits behind one: PgBouncer re-reads its configuration — including the auth file — on RELOAD (SIGHUP), while pooled server connections persist undisturbed. So the pooler fits the same playbook: the sidecar (or the rotation job) rewrites the auth file and SIGHUPs PgBouncer, and new client authentications use the new password with zero dropped server connections.

Worked example: ExternalSecret, projected volume, reload sidecar​

Everything above, as manifests. First, an ExternalSecret pulls the password from Vault into a plain Secret on a one-minute refresh:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
name: app-db-password
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: app-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/app-db
        property: password

The Deployment mounts the whole directory (no subPath — see above) and runs the file-watcher sidecar next to the app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/app:1.4.2
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: db-pass
              mountPath: /var/run/secrets/db
              readOnly: true
        - name: reload-sidecar
          image: registry.example.com/db-reload:0.3.0
          args: ["--watch", "/var/run/secrets/db/password",
                 "--on-change", "curl -sf -X POST localhost:8080/-/reload-pools"]
          volumeMounts:
            - name: db-pass
              mountPath: /var/run/secrets/db
              readOnly: true
      volumes:
        - name: db-pass
          secret:
            secretName: app-db

The app reads /var/run/secrets/db/password at startup and re-reads it inside its /-/reload-pools handler, which drains each pool so new connections authenticate with the fresh value. If your app speaks signals instead of HTTP (nginx-style SIGHUP, PgBouncer RELOAD), swap the curl for a signal via shareProcessNamespace: true — same shape, different last mile.

The rotation runbook is then five commands, zero builds:

# 1. Overlap: mint the new password and set it on the idle role (two-role leapfrog)
NEW=$(openssl rand -base64 24)
psql "$ADMIN_DSN" -c "ALTER USER app_b WITH PASSWORD '$NEW'"
# 2. Publish: write it to the secret-of-record (control-plane write, no commit)
vault kv put secret/data/prod/app-db password="$NEW" active_role=app_b
# 3. Wait: ESO refresh (<=1m) + kubelet sync (~1m); watch sidecars fire
kubectl logs deploy/app -c reload-sidecar --since=3m | grep reloaded
# 4. Drain: confirm no connections remain on the retired role, then retire it
psql "$ADMIN_DSN" -c "SELECT count(*) FROM pg_stat_activity WHERE usename='app_a'"
# 5. Retire: only after step 4 reads zero
psql "$ADMIN_DSN" -c "ALTER USER app_a WITH PASSWORD '$(openssl rand -base64 32)'"

Honest limits​

• Some rotations still deserve a redeploy. A secret schema change (renamed key, new field) needs app code that reads the new shape — that is a code deploy wearing a rotation costume. The first migration to projection also costs one deploy. The playbook eliminates the redeploy-per-rotation tax, not deploys.
• Rotation completes in minutes, not seconds. ESO refresh plus kubelet sync puts the new value live in roughly one to three minutes. For a suspected breach, do not wait for the poll cycle: flip the database side immediately, terminate attacker sessions (pg_terminate_backend), and accept the brief blip while readers converge. Scheduled rotation optimizes for zero downtime; incident rotation optimizes for containment.
• Projection is not encryption. Mounted secrets still land in etcd, and etcd is plaintext base64 until you enable encryption at rest with a KMS provider. Our survey of self-hosted PaaS secrets storage covers the at-rest half of this story; this playbook covers the rotation half. You need both.
• Reloader remains the pragmatic fallback. If a service cannot expose a recycle hook, a Reloader-driven rolling restart still beats a pipeline rebuild — no image build, no registry push, seconds instead of minutes. Adopt the sidecar where you can; annotate Reloader where you can't.

Conclusion​

The redeploy-per-rotation tax survives because env vars are the default secret delivery on git-push platforms, and env vars can only be delivered by starting new containers. Move the credential to a projected file, give each service a reload hook, and rotation collapses to what it always should have been: one database write, one secret write, and a few drained connection pools. No build queue, no rollout roulette, no 2 a.m. deploy to change a string.

Start with your highest-churn credential — usually the primary app database — and the two-role leapfrog: it needs no new infrastructure, just grant discipline and a sidecar. The vault, the dynamic leases, and the CSI driver can come later. What matters first is breaking the idea that changing a secret means shipping software.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.