Recovery

Recovery for a tamper-evident chain is restore, then verify - never rewrite. A restore is not done when the data is back; it is done when the chain re-verifies. This runbook pairs with the backup runbook.

Restore the database

scripts/restore.sh - from the agledger-ai/install repository - takes a backup tarball, stops the application containers, drops and recreates the database, runs pg_restore --no-owner, and brings the stack back up:

./scripts/restore.sh /tmp/backup/backup-2026-06-09-155745.tar.gz
./scripts/restore.sh --non-interactive <tarball>   # skip the confirmation prompt
./scripts/restore.sh --force <tarball>             # see "What it refuses to do" below

Privileges come back with the data. pg_restore applies the dump's own grants: the DML the engine writes with, the ALTER DEFAULT PRIVILEGES that keep later migrations reachable, and the append-only REVOKEs on the audit chain. A least-privilege runtime role can therefore serve the restored database without any grant being reapplied by hand. Restoring onto a server that does not carry the same roles (a cross-server DR) reports the grants it could not apply and carries on with the rows intact.

It prints the restore sequence and the container orchestration as services come back, ending with the completion line:

 Container compose-agledger-worker-1  Healthy
 Container compose-agledger-api-1     Healthy

[2026-06-09T15:58:07Z] =========================================
[2026-06-09T15:58:07Z] Restore complete.
[2026-06-09T15:58:07Z] =========================================

For an external database the script needs psql on the host, and connects as DATABASE_URL_MIGRATE when that is set (the owner role, for the DDL) and DATABASE_URL otherwise. The database it drops and restores into is the one DATABASE_URL names. POSTGRES_DB configures the bundled PostgreSQL container and is ignored here. The role needs CREATEDB and must own the database it drops, and no other session may be connected to it.

What it refuses to do

Every one of these refusals happens before the application containers are stopped, so the Server is still serving when you read the message:

• A dump it cannot read. The archive header is checked before anything is stopped or dropped.

• Two different servers. DATABASE_URL and DATABASE_URL_MIGRATE must name the same host and port. The database name is the same on nearly every install, so a migrate URL left pointing at another environment would otherwise pass a name check and drop that environment's database.

• A database that is not this install's. If the target exists, holds tables, and has no public.records, the restore stops rather than dropping someone else's database that happens to share the name. An empty database is fine: that is what an interrupted restore leaves behind. --force overrides this one, and logs that it did.

• A target it could not inspect. A connection or privilege failure while checking is reported as such, not as a wrong database name.

• A role that cannot rebuild the audit chain's own protection. The schema carries the agledger_block_audit_drop event trigger, and CREATE EVENT TRIGGER is superuser-only. The restoring role must be a superuser or a member of rds_superuser, cloudsqlsuperuser or azure_pg_admin. This is checked before anything is stopped or dropped, because pg_restore does not stop when the privilege is refused: it reports the refusal among its own errors ignored on restore, puts every row back, and exits 1, so the rows return and the trigger does not. The check has three answers, not two. Answered no and could not ask are different, and neither drops the database: a connection reset during the failover a restore happens in is not permission denied. Either way the message ends "Nothing was stopped and nothing was dropped."

  The grant lapses on its own, and disaster recovery is exactly where it lapses, because a restore routinely runs against a rebuilt server whose roles were recreated by whatever provisioning ran.

After the restore, before the Server starts

On an external database the restore does two things after the rows are back and before anything is started.

It returns the pgboss schema to the runtime role. pg_restore runs as the migrate role and so owns everything it writes, which is right for the application schema and wrong for pgboss: the Server installs that for itself and tunes autovacuum on its own job partitions at boot, which requires ownership. If the migrate role cannot perform the transfer (it has to be a member of the runtime role), the script says so and names the grant, then carries on. Nothing on the chain is at risk either way; the job tables would simply vacuum on the server defaults.

Then it runs the same runtime-role check install.sh and upgrade.sh run. If the role in DATABASE_URL cannot serve, the script stops there with the missing grant named: your data is restored and intact, the API and Worker are still stopped, and finishing is docker compose up -d --wait once the grant is in place.

Check your credentials before you declare recovery complete. The restore replaces api_keys with the copy in the backup, so the credential situation changed underneath you in two ways:

• A key minted after the backup was taken no longer exists. That includes the platform key a fresh install.sh printed minutes ago, if you reinstalled before restoring.
• Restored keys only authenticate under the API_KEY_SECRET that was in force when they were minted. Key hashes are an HMAC keyed on that secret, so a new one invalidates every restored key at once. API_KEY_SECRET belongs with your secrets, under the same custody the backup runbook requires for the private signing keys; no backup the Server produces contains it.

curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" "$AGLEDGER_API_URL/v1/auth/me"

A 401 there is not a data problem. The chain and every record are intact; you have no working credential. Mint a fresh platform key:

docker compose exec agledger-api /nodejs/bin/node dist/scripts/init.js --non-interactive

After it returns, wait for the readiness gate before trusting anything:

curl -s "$AGLEDGER_API_URL/readyz"

{"status":"ready","version":"1.4.0","timestamp":"2026-08-09T22:32:14.764Z"}

A ready Server is serving. It is not yet a verified Server.

Verify the restored chain - the step that ends the restore

Prove the chain came back intact before you declare recovery complete. Run the connected check, then the authoritative offline verification. Both scripts ship in the agledger-ai/install repository and run the tools that already live inside the Server image - no source checkout, Node.js, or pnpm on the host.

The in-database check walks every per-record chain against the live database (hash, link, and position integrity - fast, no signature check):

./scripts/vault-verify.sh

Verifying 3 record(s)...

[PASS] null (0 entries)
[PASS] 00000000-0000-0000-0000-000000000000 (3 entries)
[PASS] 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 (1 entries)

3 record(s) checked, 0 error(s)

Then the authoritative proof: produce a database-independent dump and verify its Ed25519 signatures offline against the published public keys (see the audit runbook for the full handoff). Produce the dump with the shipped wrapper -

./scripts/vault-dump.sh ./dump

• then verify it with any stock RFC 9052 (COSE) library against the per-row cose_sign1 bytes. The audit runbook gives the full walkthrough; the result on a clean restore:

[PASS] stock-library offline verification
  audit_vault entries : 9
  signatures verified : 9
  failures            : 0
  signing keys        : 1

Clean across records signed by the active key and any rotated-out retired key - retired keys travel in the dump's vault_signing_keys.ndjson, so the verifier resolves whichever key signed each entry. The restore is now complete.

When verification reports a failure

A failure is information, not a dead end. The class tells you which kind of problem you have:

| Class | What it means | Restore went wrong, or real problem? | |---|---|---| | CHAIN_POSITION_GAP | A chain position is missing | Usually a partial/interrupted restore - re-restore from a complete backup | | CHAIN_LINK_BROKEN | An entry's previous_hash does not match the prior entry | Partial restore, or a backup taken mid-write - re-restore | | CHAIN_GENESIS_INVALID | The first entry does not start at genesis | Truncated restore - re-restore | | CHAIN_HASH_MISMATCH | A stored hash does not match sha256(cose_sign1) | If the backup itself verifies clean, the restore corrupted bytes - re-restore. If the backup also fails here, the backup faithfully captured a real tamper you must investigate |

The discriminator: verify the backup (its NDJSON dump) independently. If the backup verifies and the restore does not, the restore is at fault - repeat it. If the backup itself fails, recovery will not paper over it; you have a real integrity problem from before the backup was taken.

There is one more class, and it is not about your restore at all. The offline verifier calls it CHAIN_UNSUPPORTED_ALGORITHM and the Server's own on-host scan calls it unsupported_algorithm; they are the same finding reported by two tools. It appears when you restore onto a host whose crypto provider cannot compute the algorithm that signed the entries. The case that produces it in practice is restoring an Ed25519 chain onto a FIPS host, whose provider carries no Ed25519: the restore succeeded and the chain is intact, but this host cannot check it, and an on-host scan will report every pre-existing record broken. Dump the chain and verify it off-host to confirm, then decide whether that host is the right home for that history. The signing-key compromise runbook covers the reasoning in full.

Signing-key recovery

The vault_signing_keys registry is part of the database dump, so it returns with the restore - the signing keys line in the verification output above accounts for the active key plus any rotated-out retired keys. Records signed under a retired key still verify, because the engine resolves historical keys from the registry.

Restoring the registry is not the same as restoring the ability to sign. New records need the private key in VAULT_SIGNING_KEY (and VAULT_SIGNING_KEY_PREVIOUS), which lives in your secret store, not the backup. Restore the database and set those env vars; on boot the engine reconciles the registry to the configured key (see the day-2 operations runbook). If the registry row is ever lost, the VAULT_SIGNING_KEY env var is the fallback the engine bootstraps from.

What recovery cannot do

You cannot rewrite history to "fix" a chain. A record that was lost before the backup shows up as a chain gap (CHAIN_POSITION_GAP), and that gap is the honest record of what happened - there is nothing to restore it from and nothing legitimate to paper over it with. Recovery brings back what the backup holds and proves it; it does not manufacture entries.

Multi-Server recovery

Each Server owns its own database and restores its own slice independently - there is no shared store to coordinate. Cross-Server delegation chains re-link by signature reference, not by restoring a common database: once each Server is restored and verifying, the links across them resolve through the published public keys. Restore and verify Server by Server; the federation re-forms on its own.