Recovery
Recovery for a tamper-evident chain is restore, then verify - never rewrite. A restore is not done when the data is back; it is done when the chain re-verifies. This runbook pairs with the backup runbook.
Restore the database
scripts/restore.sh - from the agledger-ai/install
repository - takes a backup tarball, stops the application containers, drops and recreates the
database, runs pg_restore --no-owner, and brings the stack back up:
./scripts/restore.sh /tmp/backup/backup-2026-06-09-155745.tar.gz
./scripts/restore.sh --non-interactive <tarball> # skip the confirmation prompt
./scripts/restore.sh --force <tarball> # see "What it refuses to do" below
Privileges come back with the data. pg_restore applies the dump's own grants: the DML the engine
writes with, the ALTER DEFAULT PRIVILEGES that keep later migrations reachable, and the
append-only REVOKEs on the audit chain. A least-privilege runtime role can therefore serve the
restored database without any grant being reapplied by hand. Restoring onto a server that does not
carry the same roles (a cross-server DR) reports the grants it could not apply and carries on with
the rows intact.
It prints the restore sequence and the container orchestration as services come back, ending with the completion line:
Container compose-agledger-worker-1 Healthy
Container compose-agledger-api-1 Healthy
[2026-06-09T15:58:07Z] =========================================
[2026-06-09T15:58:07Z] Restore complete.
[2026-06-09T15:58:07Z] =========================================
For an external database the script needs psql on the host, and connects as DATABASE_URL_MIGRATE
when that is set (the owner role, for the DDL) and DATABASE_URL otherwise. The database it drops
and restores into is the one DATABASE_URL names. POSTGRES_DB configures the bundled
PostgreSQL container and is ignored here. The role needs CREATEDB and must own the database it
drops, and no other session may be connected to it.
What it refuses to do
Every one of these refusals happens before the application containers are stopped, so the Server is still serving when you read the message:
-
A dump it cannot read. The archive header is checked before anything is stopped or dropped.
-
Two different servers.
DATABASE_URLandDATABASE_URL_MIGRATEmust name the same host and port. The database name is the same on nearly every install, so a migrate URL left pointing at another environment would otherwise pass a name check and drop that environment's database. -
A database that is not this install's. If the target exists, holds tables, and has no
public.records, the restore stops rather than dropping someone else's database that happens to share the name. An empty database is fine: that is what an interrupted restore leaves behind.--forceoverrides this one, and logs that it did. -
A target it could not inspect. A connection or privilege failure while checking is reported as such, not as a wrong database name.
-
A role that cannot rebuild the audit chain's own protection. The schema carries the
agledger_block_audit_dropevent trigger, andCREATE EVENT TRIGGERis superuser-only. The restoring role must be a superuser or a member ofrds_superuser,cloudsqlsuperuserorazure_pg_admin. This is checked before anything is stopped or dropped, becausepg_restoredoes not stop when the privilege is refused: it reports the refusal among its own errors ignored on restore, puts every row back, and exits 1, so the rows return and the trigger does not. The check has three answers, not two. Answered no and could not ask are different, and neither drops the database: a connection reset during the failover a restore happens in is not permission denied. Either way the message ends "Nothing was stopped and nothing was dropped."The grant lapses on its own, and disaster recovery is exactly where it lapses, because a restore routinely runs against a rebuilt server whose roles were recreated by whatever provisioning ran.
After the restore, before the Server starts
On an external database the restore does two things after the rows are back and before anything is started.
It returns the pgboss schema to the runtime role. pg_restore runs as the migrate role and so
owns everything it writes, which is right for the application schema and wrong for pgboss: the
Server installs that for itself and tunes autovacuum on its own job partitions at boot, which
requires ownership. If the migrate role cannot perform the transfer (it has to be a member of the
runtime role), the script says so and names the grant, then carries on. Nothing on the chain is at
risk either way; the job tables would simply vacuum on the server defaults.
Then it runs the same runtime-role check install.sh and upgrade.sh run. If the role in
DATABASE_URL cannot serve, the script stops there with the missing grant named: your data is
restored and intact, the API and Worker are still stopped, and finishing is
docker compose up -d --wait once the grant is in place.
Check your credentials before you declare recovery complete. The restore replaces api_keys
with the copy in the backup, so the credential situation changed underneath you in two ways:
- A key minted after the backup was taken no longer exists. That includes the platform key a fresh
install.shprinted minutes ago, if you reinstalled before restoring. - Restored keys only authenticate under the
API_KEY_SECRETthat was in force when they were minted. Key hashes are an HMAC keyed on that secret, so a new one invalidates every restored key at once.API_KEY_SECRETbelongs with your secrets, under the same custody the backup runbook requires for the private signing keys; no backup the Server produces contains it.
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" "$AGLEDGER_API_URL/v1/auth/me"
A 401 there is not a data problem. The chain and every record are intact; you have no working
credential. Mint a fresh platform key:
docker compose exec agledger-api /nodejs/bin/node dist/scripts/init.js --non-interactive
After it returns, wait for the readiness gate before trusting anything:
curl -s "$AGLEDGER_API_URL/readyz"
{"status":"ready","version":"1.4.0","timestamp":"2026-08-09T22:32:14.764Z"}
A ready Server is serving. It is not yet a verified Server.
Verify the restored chain - the step that ends the restore
Prove the chain came back intact before you declare recovery complete. Run the connected check, then
the authoritative offline verification. Both scripts ship in the
agledger-ai/install repository and run the tools that
already live inside the Server image - no source checkout, Node.js, or pnpm on the host.
The in-database check walks every per-record chain against the live database (hash, link, and position integrity - fast, no signature check):
./scripts/vault-verify.sh
Verifying 3 record(s)...
[PASS] null (0 entries)
[PASS] 00000000-0000-0000-0000-000000000000 (3 entries)
[PASS] 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 (1 entries)
3 record(s) checked, 0 error(s)
Then the authoritative proof: produce a database-independent dump and verify its Ed25519 signatures offline against the published public keys (see the audit runbook for the full handoff). Produce the dump with the shipped wrapper -
./scripts/vault-dump.sh ./dump
- then verify it with any stock RFC 9052 (COSE) library against the per-row
cose_sign1bytes. The audit runbook gives the full walkthrough; the result on a clean restore:
[PASS] stock-library offline verification
audit_vault entries : 9
signatures verified : 9
failures : 0
signing keys : 1
Clean across records signed by the active key and any rotated-out retired key - retired keys travel
in the dump's vault_signing_keys.ndjson, so the verifier resolves whichever key signed each entry.
The restore is now complete.
When verification reports a failure
A failure is information, not a dead end. The class tells you which kind of problem you have:
| Class | What it means | Restore went wrong, or real problem? |
|---|---|---|
| CHAIN_POSITION_GAP | A chain position is missing | Usually a partial/interrupted restore - re-restore from a complete backup |
| CHAIN_LINK_BROKEN | An entry's previous_hash does not match the prior entry | Partial restore, or a backup taken mid-write - re-restore |
| CHAIN_GENESIS_INVALID | The first entry does not start at genesis | Truncated restore - re-restore |
| CHAIN_HASH_MISMATCH | A stored hash does not match sha256(cose_sign1) | If the backup itself verifies clean, the restore corrupted bytes - re-restore. If the backup also fails here, the backup faithfully captured a real tamper you must investigate |
The discriminator: verify the backup (its NDJSON dump) independently. If the backup verifies and the restore does not, the restore is at fault - repeat it. If the backup itself fails, recovery will not paper over it; you have a real integrity problem from before the backup was taken.
There is one more class, and it is not about your restore at all. The offline verifier calls it
CHAIN_UNSUPPORTED_ALGORITHM and the Server's own on-host scan calls it unsupported_algorithm;
they are the same finding reported by two tools. It appears when you restore onto a host whose
crypto provider cannot compute the algorithm that signed the entries. The case that produces it in
practice is restoring an Ed25519 chain onto a FIPS host, whose
provider carries no Ed25519: the restore succeeded and the chain is intact, but this host cannot
check it, and an on-host scan will report every pre-existing record broken. Dump the chain and
verify it off-host to confirm, then decide whether that host is the right home for that history. The
signing-key compromise runbook covers the reasoning in full.
Signing-key recovery
The vault_signing_keys registry is part of the database dump, so it returns with the restore - the
signing keys line in the verification output above accounts for the active key plus any
rotated-out retired keys. Records signed under a retired key still verify, because the engine
resolves historical keys from the registry.
Restoring the registry is not the same as restoring the ability to sign. New records need the
private key in VAULT_SIGNING_KEY (and VAULT_SIGNING_KEY_PREVIOUS), which lives in your secret
store, not the backup. Restore the database and set those env vars; on boot the engine reconciles
the registry to the configured key (see the day-2 operations runbook). If
the registry row is ever lost, the VAULT_SIGNING_KEY env var is the fallback the engine bootstraps
from.
What recovery cannot do
You cannot rewrite history to "fix" a chain. A record that was lost before the backup shows up as a
chain gap (CHAIN_POSITION_GAP), and that gap is the honest record of what happened - there is
nothing to restore it from and nothing legitimate to paper over it with. Recovery brings back what
the backup holds and proves it; it does not manufacture entries.
Multi-Server recovery
Each Server owns its own database and restores its own slice independently - there is no shared store to coordinate. Cross-Server delegation chains re-link by signature reference, not by restoring a common database: once each Server is restored and verifying, the links across them resolve through the published public keys. Restore and verify Server by Server; the federation re-forms on its own.