---
id: platform/recovery
title: Platform recovery rehearsals
description: Prepare isolated drills that restore Bex platform state into throwaway targets using the upstream restore scripts.
keywords: [bex, recovery, backup, etcd, openbao, postgres, key value, self-hosted]
sidebar_position: 90
last_updated: 2026-09-23
---

# Platform recovery rehearsals

Self-hosted Bex installations store operator-critical state outside tenant
application disks. This guide is the entry point for **isolated recovery
rehearsals** using the scripts shipped in the Bex repository. It does not
perform a live cutover, rotate credentials, or claim an RPO/RTO.

Tenant service disks are a different model — see
[Persistent disks](./persistent-disks.md). Tenant Postgres and Key Value
application data also have product-level guides:
[PostgreSQL](./postgres.md) and [Key Value](./key-value.md).

**Source map (pin these when you refresh the guide):**

| Artifact | Location (Bex @ `0eb2036e2`) |
| --- | --- |
| Shared guards | `scripts/lib/restore.sh` |
| etcd extract/apply | `scripts/restore-etcd.sh` |
| OpenBao throwaway | `scripts/restore-openbao.sh` |
| Key Value RDB | `scripts/restore-keyvalue.sh` |
| Postgres CNPG archive | `scripts/restore-postgres.sh` |
| Offline script tests | `scripts/restore.test.sh` |

Run every command from a checkout of [bex-co/bex](https://github.com/bex-co/bex)
with the scripts above. The CMS does not embed or execute those scripts.

## What each store protects

| Store | Backup artifact (typical) | Rehearsal script | Isolated target rule |
| --- | --- | --- | --- |
| Control-plane etcd | Encrypted etcd snapshot object | `restore-etcd.sh` | Local throwaway etcd; optional apply only to `restore-*` kube contexts |
| OpenBao | Encrypted Raft snapshot | `restore-openbao.sh` | New `restore-*` namespace only |
| Platform / auth Postgres | CNPG object-store base backup + WAL | `restore-postgres.sh` | New `restore-*` namespace; source archive stays read-only |
| Paid Key Value | Encrypted RDB object | `restore-keyvalue.sh` | New `restore-*` namespace + empty PVC |
| Service disks | Product snapshots | (product APIs) | Not covered by these scripts |

A configured backup **schedule** or paid plan is not proof that a usable
snapshot exists. Before a drill, list the object store (or CNPG recovery
status) and pick a concrete archive identity (`latest` only after you have
confirmed objects are present and decryptable).

## Shared preparation

1. **Tools.** The restore helpers require `bash`, `kubectl` (where noted),
   digest-pinned container images for credential-bearing helpers, and AWS CLI
   compatible credentials for the backup object store (`TF_STATE_*` aliases are
   accepted by `restore_load_dotenv`).
2. **Throwaway naming.** Mutating targets must be namespaces or kube contexts
   whose names start with `restore-`. Live namespaces are refused.
3. **Confirmation.** Real mutations require `--confirm` matching the target
   name (or `APPLY-restore-…` for etcd apply). Review `DRY_RUN=1` output first.
4. **Encryption keys.** Encrypted platform backups need the retained `age`
   private key generation that can decrypt the selected object. OpenBao
   additionally needs the **original** unseal key set and a root token after
   transport decryption — see the OpenBao drill.
5. **Postgres readers.** Some installations override the archive reader
   credentials per ObjectStore; others fall back to pre-migration defaults.
   Do not assume every cluster already uses write-only writers.
6. **DRY_RUN side effects.** `DRY_RUN=1` is not always offline: depending on
   the script it may talk to Kubernetes, list/decrypt object-store objects, or
   plan Docker/kube resources. Never source production `.env` from CI or from
   this documentation site. Use `RESTORE_SKIP_DOTENV=1` with explicit env vars
   in disposable shells when experimenting.

## Drill order

Work through one store at a time in a disposable environment:

1. [etcd extraction and optional apply](./recovery-etcd.md)
2. [OpenBao throwaway restore](./recovery-openbao.md)
3. [Key Value RDB restore](./recovery-keyvalue.md)
4. [Postgres archive restore](./recovery-postgres.md)

Success for a rehearsal means: the throwaway target starts, the documented
verification query or path succeeds, and you tear the target down. It does
**not** mean application traffic, DNS, or credential rotation were updated.

## Cleanup

Prefer `--teardown-on-success` or the explicit `--teardown restore-NAME
--confirm restore-NAME` forms after you capture verification evidence. Leave no
`restore-*` namespaces behind in shared clusters.