Recovery

Recovery for a tamper-evident chain is restore, then verify - never rewrite. A restore is not done when the data is back; it is done when the chain re-verifies. This runbook pairs with the backup runbook.

Restore the database

scripts/restore.sh - from the agledger-ai/install repository - takes a backup tarball, stops the application containers, drops and recreates the database, runs pg_restore --no-owner --no-acl, and brings the stack back up:

./scripts/restore.sh /tmp/backup/backup-2026-06-09-155745.tar.gz

It prints the restore sequence and the container orchestration as services come back, ending with the completion line:

 Container compose-agledger-worker-1  Healthy
 Container compose-agledger-api-1     Healthy

[2026-06-09T15:58:07Z] =========================================
[2026-06-09T15:58:07Z] Restore complete.
[2026-06-09T15:58:07Z] =========================================

For an external database the script uses pg_restore against your DATABASE_URL directly; the user must have CREATEDB. After it returns, wait for the readiness gate before trusting anything:

curl -s "$AGLEDGER_URL/readyz"

{"status":"ready","version":"1.0.2","timestamp":"2026-06-09T15:58:16.340Z"}

A ready Server is serving. It is not yet a verified Server.

Verify the restored chain - the step that ends the restore

Prove the chain came back intact before you declare recovery complete. Run the connected check, then the authoritative offline verification. Both scripts ship in the agledger-ai/install repository and run the tools that already live inside the Server image - no source checkout, Node.js, or pnpm on the host.

The in-database check walks every per-record chain against the live database (hash, link, and position integrity - fast, no signature check):

./scripts/vault-verify.sh

Verifying 3 record(s)...

[PASS] null (0 entries)
[PASS] 00000000-0000-0000-0000-000000000000 (3 entries)
[PASS] 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 (1 entries)

3 record(s) checked, 0 error(s)

Then the authoritative proof: produce a database-independent dump and verify its Ed25519 signatures offline against the published public keys (see the audit runbook for the full handoff). Produce the dump with the shipped wrapper -

./scripts/vault-dump.sh ./dump

[PASS] stock-library offline verification
  audit_vault entries : 9
  signatures verified : 9
  failures            : 0
  signing keys        : 1

Clean across records signed by the active key and any rotated-out retired key - retired keys travel in the dump's vault_signing_keys.ndjson, so the verifier resolves whichever key signed each entry. The restore is now complete.

When verification reports a failure

A failure is information, not a dead end. The class tells you which kind of problem you have:

| Class | What it means | Restore went wrong, or real problem? | |---|---|---| | POSITION_GAP | A chain position is missing | Usually a partial/interrupted restore - re-restore from a complete backup | | LINK_BROKEN | An entry's previous_hash does not match the prior entry | Partial restore, or a backup taken mid-write - re-restore | | GENESIS_INVALID | The first entry does not start at genesis | Truncated restore - re-restore | | HASH_MISMATCH | A stored hash does not match sha256(cose_sign1) | If the backup itself verifies clean, the restore corrupted bytes - re-restore. If the backup also fails here, the backup faithfully captured a real tamper you must investigate |

The discriminator: verify the backup (its NDJSON dump) independently. If the backup verifies and the restore does not, the restore is at fault - repeat it. If the backup itself fails, recovery will not paper over it; you have a real integrity problem from before the backup was taken.

Signing-key recovery

The vault_signing_keys registry is part of the database dump, so it returns with the restore - the signing keys line in the verification output above accounts for the active key plus any rotated-out retired keys. Records signed under a retired key still verify, because the engine resolves historical keys from the registry.

Restoring the registry is not the same as restoring the ability to sign. New records need the private key in VAULT_SIGNING_KEY (and VAULT_SIGNING_KEY_PREVIOUS), which lives in your secret store, not the backup. Restore the database and set those env vars; on boot the engine reconciles the registry to the configured key (see the day-2 operations runbook). If the registry row is ever lost, the VAULT_SIGNING_KEY env var is the fallback the engine bootstraps from.

What recovery cannot do

You cannot rewrite history to "fix" a chain. A record that was lost before the backup shows up as a chain gap (POSITION_GAP), and that gap is the honest record of what happened - there is nothing to restore it from and nothing legitimate to paper over it with. Recovery brings back what the backup holds and proves it; it does not manufacture entries.

Multi-Server recovery

Each Server owns its own database and restores its own slice independently - there is no shared store to coordinate. Cross-Server delegation chains re-link by signature reference, not by restoring a common database: once each Server is restored and verifying, the links across them resolve through the published public keys. Restore and verify Server by Server; the federation re-forms on its own.

Validated against API v1.1.0 on 2026-06-10 (Developer Edition, Docker Compose: restore → readiness gate → in-database check → stock-library offline verification, end to end).