Self-hosted Bex installations store operator-critical state outside tenant application disks. This guide is the entry point for isolated recovery rehearsals using the scripts shipped in the Bex repository. It does not perform a live cutover, rotate credentials, or claim an RPO/RTO.
Tenant service disks are a different model — see Persistent disks. Tenant Postgres and Key Value application data also have product-level guides: PostgreSQL and Key Value.
Source map (pin these when you refresh the guide):
| Artifact | Location (Bex @ 0eb2036e2) |
|---|---|
| Shared guards | scripts/lib/restore.sh |
| etcd extract/apply | scripts/restore-etcd.sh |
| OpenBao throwaway | scripts/restore-openbao.sh |
| Key Value RDB | scripts/restore-keyvalue.sh |
| Postgres CNPG archive | scripts/restore-postgres.sh |
| Offline script tests | scripts/restore.test.sh |
Run every command from a checkout of bex-co/bex with the scripts above. The CMS does not embed or execute those scripts.
What each store protects
| Store | Backup artifact (typical) | Rehearsal script | Isolated target rule |
|---|---|---|---|
| Control-plane etcd | Encrypted etcd snapshot object | restore-etcd.sh | Local throwaway etcd; optional apply only to restore-* kube contexts |
| OpenBao | Encrypted Raft snapshot | restore-openbao.sh | New restore-* namespace only |
| Platform / auth Postgres | CNPG object-store base backup + WAL | restore-postgres.sh | New restore-* namespace; source archive stays read-only |
| Paid Key Value | Encrypted RDB object | restore-keyvalue.sh | New restore-* namespace + empty PVC |
| Service disks | Product snapshots | (product APIs) | Not covered by these scripts |
A configured backup schedule or paid plan is not proof that a usable
snapshot exists. Before a drill, list the object store (or CNPG recovery
status) and pick a concrete archive identity (latest only after you have
confirmed objects are present and decryptable).
Shared preparation
- Tools. The restore helpers require
bash,kubectl(where noted), digest-pinned container images for credential-bearing helpers, and AWS CLI compatible credentials for the backup object store (TF_STATE_*aliases are accepted byrestore_load_dotenv). - Throwaway naming. Mutating targets must be namespaces or kube contexts
whose names start with
restore-. Live namespaces are refused. - Confirmation. Real mutations require
--confirmmatching the target name (orAPPLY-restore-…for etcd apply). ReviewDRY_RUN=1output first. - Encryption keys. Encrypted platform backups need the retained
ageprivate key generation that can decrypt the selected object. OpenBao additionally needs the original unseal key set and a root token after transport decryption — see the OpenBao drill. - Postgres readers. Some installations override the archive reader credentials per ObjectStore; others fall back to pre-migration defaults. Do not assume every cluster already uses write-only writers.
- DRY_RUN side effects.
DRY_RUN=1is not always offline: depending on the script it may talk to Kubernetes, list/decrypt object-store objects, or plan Docker/kube resources. Never source production.envfrom CI or from this documentation site. UseRESTORE_SKIP_DOTENV=1with explicit env vars in disposable shells when experimenting.
Drill order
Work through one store at a time in a disposable environment:
- etcd extraction and optional apply
- OpenBao throwaway restore
- Key Value RDB restore
- Postgres archive restore
Success for a rehearsal means: the throwaway target starts, the documented verification query or path succeeds, and you tear the target down. It does not mean application traffic, DNS, or credential rotation were updated.
Cleanup
Prefer --teardown-on-success or the explicit --teardown restore-NAME --confirm restore-NAME forms after you capture verification evidence. Leave no
restore-* namespaces behind in shared clusters.