> Markdown version of https://agledger.ai/docs/operations/recovery/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # Recovery Recovery for a tamper-evident chain is **restore, then verify** - never rewrite. A restore is not done when the data is back; it is done when the chain re-verifies. This runbook pairs with the [backup runbook](/docs/operations/backup/). ## Restore the database `scripts/restore.sh` - from the [`agledger-ai/install`](https://github.com/agledger-ai/install) repository - takes a backup tarball, stops the application containers, drops and recreates the database, runs `pg_restore --no-owner`, and brings the stack back up: ```bash ./scripts/restore.sh /tmp/backup/backup-2026-06-09-155745.tar.gz ./scripts/restore.sh --non-interactive # skip the confirmation prompt ./scripts/restore.sh --force # see "What it refuses to do" below ``` Privileges come back with the data. `pg_restore` applies the dump's own grants: the DML the engine writes with, the `ALTER DEFAULT PRIVILEGES` that keep later migrations reachable, and the append-only `REVOKE`s on the audit chain. A least-privilege runtime role can therefore serve the restored database without any grant being reapplied by hand. Restoring onto a server that does not carry the same roles (a cross-server DR) reports the grants it could not apply and carries on with the rows intact. It prints the restore sequence and the container orchestration as services come back, ending with the completion line: ``` Container compose-agledger-worker-1 Healthy Container compose-agledger-api-1 Healthy [2026-06-09T15:58:07Z] ========================================= [2026-06-09T15:58:07Z] Restore complete. [2026-06-09T15:58:07Z] ========================================= ``` For an external database the script needs `psql` on the host, and connects as `DATABASE_URL_MIGRATE` when that is set (the owner role, for the DDL) and `DATABASE_URL` otherwise. The database it drops and restores into is the one **`DATABASE_URL`** names. `POSTGRES_DB` configures the bundled PostgreSQL container and is ignored here. The role needs `CREATEDB` and must own the database it drops, and no other session may be connected to it. ### What it refuses to do Every one of these refusals happens before the application containers are stopped, so the Server is still serving when you read the message: - **A dump it cannot read.** The archive header is checked before anything is stopped or dropped. - **Two different servers.** `DATABASE_URL` and `DATABASE_URL_MIGRATE` must name the same host and port. The database *name* is the same on nearly every install, so a migrate URL left pointing at another environment would otherwise pass a name check and drop that environment's database. - **A database that is not this install's.** If the target exists, holds tables, and has no `public.records`, the restore stops rather than dropping someone else's database that happens to share the name. An empty database is fine: that is what an interrupted restore leaves behind. `--force` overrides this one, and logs that it did. - **A target it could not inspect.** A connection or privilege failure while checking is reported as such, not as a wrong database name. - **A role that cannot rebuild the audit chain's own protection.** The schema carries the `agledger_block_audit_drop` event trigger, and `CREATE EVENT TRIGGER` is superuser-only. The restoring role must be a superuser or a member of `rds_superuser`, `cloudsqlsuperuser` or `azure_pg_admin`. This is checked before anything is stopped or dropped, because `pg_restore` does not stop when the privilege is refused: it reports the refusal among its own errors ignored on restore, puts every row back, and exits 1, so the rows return and the trigger does not. The check has three answers, not two. Answered no and could not ask are different, and neither drops the database: a connection reset during the failover a restore happens in is not permission denied. Either way the message ends "Nothing was stopped and nothing was dropped." The grant lapses on its own, and disaster recovery is exactly where it lapses, because a restore routinely runs against a rebuilt server whose roles were recreated by whatever provisioning ran. ### After the restore, before the Server starts On an external database the restore does two things after the rows are back and before anything is started. It returns the `pgboss` schema to the runtime role. `pg_restore` runs as the migrate role and so owns everything it writes, which is right for the application schema and wrong for `pgboss`: the Server installs that for itself and tunes autovacuum on its own job partitions at boot, which requires ownership. If the migrate role cannot perform the transfer (it has to be a member of the runtime role), the script says so and names the grant, then carries on. Nothing on the chain is at risk either way; the job tables would simply vacuum on the server defaults. Then it runs the same runtime-role check `install.sh` and `upgrade.sh` run. If the role in `DATABASE_URL` cannot serve, the script stops there with the missing grant named: your data is restored and intact, the API and Worker are still stopped, and finishing is `docker compose up -d --wait` once the grant is in place. **Check your credentials before you declare recovery complete.** The restore replaces `api_keys` with the copy in the backup, so the credential situation changed underneath you in two ways: - A key minted after the backup was taken no longer exists. That includes the platform key a fresh `install.sh` printed minutes ago, if you reinstalled before restoring. - Restored keys only authenticate under the `API_KEY_SECRET` that was in force when they were minted. Key hashes are an HMAC keyed on that secret, so a new one invalidates every restored key at once. `API_KEY_SECRET` belongs with your secrets, under the same custody the [backup runbook](/docs/operations/backup/) requires for the private signing keys; no backup the Server produces contains it. ```bash curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" "$AGLEDGER_API_URL/v1/auth/me" ``` A `401` there is not a data problem. The chain and every record are intact; you have no working credential. Mint a fresh platform key: ```bash docker compose exec agledger-api /nodejs/bin/node dist/scripts/init.js --non-interactive ``` After it returns, wait for the readiness gate before trusting anything: ```bash curl -s "$AGLEDGER_API_URL/readyz" ``` ```json {"status":"ready","version":"1.4.0","timestamp":"2026-08-09T22:32:14.764Z"} ``` A `ready` Server is serving. It is not yet a *verified* Server. ## Verify the restored chain - the step that ends the restore Prove the chain came back intact before you declare recovery complete. Run the connected check, then the authoritative offline verification. Both scripts ship in the [`agledger-ai/install`](https://github.com/agledger-ai/install) repository and run the tools that already live inside the Server image - no source checkout, Node.js, or pnpm on the host. The in-database check walks every per-record chain against the live database (hash, link, and position integrity - fast, no signature check): ```bash ./scripts/vault-verify.sh ``` ``` Verifying 3 record(s)... [PASS] null (0 entries) [PASS] 00000000-0000-0000-0000-000000000000 (3 entries) [PASS] 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 (1 entries) 3 record(s) checked, 0 error(s) ``` Then the authoritative proof: produce a database-independent dump and verify its Ed25519 signatures offline against the published public keys (see the [audit runbook](/docs/operations/audit/) for the full handoff). Produce the dump with the shipped wrapper - ```bash ./scripts/vault-dump.sh ./dump ``` - then verify it with any stock RFC 9052 (COSE) library against the per-row `cose_sign1` bytes. The [audit runbook](/docs/operations/audit/) gives the full walkthrough; the result on a clean restore: ``` [PASS] stock-library offline verification audit_vault entries : 9 signatures verified : 9 failures : 0 signing keys : 1 ``` Clean across records signed by the active key and any rotated-out retired key - retired keys travel in the dump's `vault_signing_keys.ndjson`, so the verifier resolves whichever key signed each entry. The restore is now complete. ### When verification reports a failure A failure is information, not a dead end. The class tells you which kind of problem you have: | Class | What it means | Restore went wrong, or real problem? | |---|---|---| | `CHAIN_POSITION_GAP` | A chain position is missing | Usually a partial/interrupted restore - re-restore from a complete backup | | `CHAIN_LINK_BROKEN` | An entry's `previous_hash` does not match the prior entry | Partial restore, or a backup taken mid-write - re-restore | | `CHAIN_GENESIS_INVALID` | The first entry does not start at genesis | Truncated restore - re-restore | | `CHAIN_HASH_MISMATCH` | A stored hash does not match `sha256(cose_sign1)` | If the backup itself verifies clean, the restore corrupted bytes - re-restore. If the backup *also* fails here, the backup faithfully captured a real tamper you must investigate | The discriminator: verify the **backup** (its NDJSON dump) independently. If the backup verifies and the restore does not, the restore is at fault - repeat it. If the backup itself fails, recovery will not paper over it; you have a real integrity problem from before the backup was taken. There is one more class, and it is not about your restore at all. The offline verifier calls it `CHAIN_UNSUPPORTED_ALGORITHM` and the Server's own on-host scan calls it `unsupported_algorithm`; they are the same finding reported by two tools. It appears when you restore onto a host whose crypto provider cannot compute the algorithm that signed the entries. The case that produces it in practice is restoring an Ed25519 chain onto a FIPS host, whose provider carries no Ed25519: the restore succeeded and the chain is intact, but this host cannot check it, and an on-host scan will report every pre-existing record broken. Dump the chain and verify it off-host to confirm, then decide whether that host is the right home for that history. The [signing-key compromise runbook](/docs/operations/key-compromise/) covers the reasoning in full. ## Signing-key recovery The `vault_signing_keys` registry is part of the database dump, so it returns with the restore - the `signing keys` line in the verification output above accounts for the active key plus any rotated-out retired keys. Records signed under a retired key still verify, because the engine resolves historical keys from the registry. Restoring the registry is not the same as restoring the ability to *sign*. New records need the private key in `VAULT_SIGNING_KEY` (and `VAULT_SIGNING_KEY_PREVIOUS`), which lives in your secret store, not the backup. Restore the database and set those env vars; on boot the engine reconciles the registry to the configured key (see the [day-2 operations runbook](/docs/operations/day-2/)). If the registry row is ever lost, the `VAULT_SIGNING_KEY` env var is the fallback the engine bootstraps from. ## What recovery cannot do You cannot rewrite history to "fix" a chain. A record that was lost before the backup shows up as a chain gap (`CHAIN_POSITION_GAP`), and that gap is the honest record of what happened - there is nothing to restore it from and nothing legitimate to paper over it with. Recovery brings back what the backup holds and proves it; it does not manufacture entries. ## Multi-Server recovery Each Server owns its own database and restores its own slice independently - there is no shared store to coordinate. Cross-Server delegation chains re-link by signature reference, not by restoring a common database: once each Server is restored and verifying, the links across them resolve through the published public keys. Restore and verify Server by Server; the federation re-forms on its own.