Recovery

Recovery for a tamper-evident chain is restore, then verify - never rewrite. A restore is not done when the data is back; it is done when the chain re-verifies. This runbook pairs with the backup runbook.

Restore the database

scripts/restore.sh - from the agledger-ai/install repository - takes a backup tarball, stops the application containers, drops and recreates the database, runs pg_restore --no-owner --no-acl, and brings the stack back up:

./scripts/restore.sh /tmp/backup/backup-2026-06-09-155745.tar.gz

It prints the restore sequence and the container orchestration as services come back, ending with the completion line:

 Container compose-agledger-worker-1  Healthy
 Container compose-agledger-api-1     Healthy

[2026-06-09T15:58:07Z] =========================================
[2026-06-09T15:58:07Z] Restore complete.
[2026-06-09T15:58:07Z] =========================================

For an external database the script uses pg_restore against your DATABASE_URL directly; the user must have CREATEDB. After it returns, wait for the readiness gate before trusting anything:

curl -s "$AGLEDGER_API_URL/readyz"

{"status":"ready","version":"1.4.0","timestamp":"2026-08-09T22:32:14.764Z"}

A ready Server is serving. It is not yet a verified Server.

Verify the restored chain - the step that ends the restore

Prove the chain came back intact before you declare recovery complete. Run the connected check, then the authoritative offline verification. Both scripts ship in the agledger-ai/install repository and run the tools that already live inside the Server image - no source checkout, Node.js, or pnpm on the host.

The in-database check walks every per-record chain against the live database (hash, link, and position integrity - fast, no signature check):

./scripts/vault-verify.sh

Verifying 3 record(s)...

[PASS] null (0 entries)
[PASS] 00000000-0000-0000-0000-000000000000 (3 entries)
[PASS] 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 (1 entries)

3 record(s) checked, 0 error(s)

Then the authoritative proof: produce a database-independent dump and verify its Ed25519 signatures offline against the published public keys (see the audit runbook for the full handoff). Produce the dump with the shipped wrapper -

./scripts/vault-dump.sh ./dump

[PASS] stock-library offline verification
  audit_vault entries : 9
  signatures verified : 9
  failures            : 0
  signing keys        : 1

Clean across records signed by the active key and any rotated-out retired key - retired keys travel in the dump's vault_signing_keys.ndjson, so the verifier resolves whichever key signed each entry. The restore is now complete.

When verification reports a failure

A failure is information, not a dead end. The class tells you which kind of problem you have:

| Class | What it means | Restore went wrong, or real problem? | |---|---|---| | POSITION_GAP | A chain position is missing | Usually a partial/interrupted restore - re-restore from a complete backup | | LINK_BROKEN | An entry's previous_hash does not match the prior entry | Partial restore, or a backup taken mid-write - re-restore | | GENESIS_INVALID | The first entry does not start at genesis | Truncated restore - re-restore | | HASH_MISMATCH | A stored hash does not match sha256(cose_sign1) | If the backup itself verifies clean, the restore corrupted bytes - re-restore. If the backup also fails here, the backup faithfully captured a real tamper you must investigate |

The discriminator: verify the backup (its NDJSON dump) independently. If the backup verifies and the restore does not, the restore is at fault - repeat it. If the backup itself fails, recovery will not paper over it; you have a real integrity problem from before the backup was taken.

There is one more class, and it is not about your restore at all. unsupported_algorithm appears when you restore onto a host whose crypto provider cannot compute the algorithm that signed the entries. The case that produces it in practice is restoring an Ed25519 chain onto a FIPS host, whose provider carries no Ed25519: the restore succeeded and the chain is intact, but this host cannot check it, and an on-host scan will report every pre-existing record broken. Dump the chain and verify it off-host to confirm, then decide whether that host is the right home for that history. The signing-key compromise runbook covers the reasoning in full.

Signing-key recovery

The vault_signing_keys registry is part of the database dump, so it returns with the restore - the signing keys line in the verification output above accounts for the active key plus any rotated-out retired keys. Records signed under a retired key still verify, because the engine resolves historical keys from the registry.

Restoring the registry is not the same as restoring the ability to sign. New records need the private key in VAULT_SIGNING_KEY (and VAULT_SIGNING_KEY_PREVIOUS), which lives in your secret store, not the backup. Restore the database and set those env vars; on boot the engine reconciles the registry to the configured key (see the day-2 operations runbook). If the registry row is ever lost, the VAULT_SIGNING_KEY env var is the fallback the engine bootstraps from.

What recovery cannot do

You cannot rewrite history to "fix" a chain. A record that was lost before the backup shows up as a chain gap (POSITION_GAP), and that gap is the honest record of what happened - there is nothing to restore it from and nothing legitimate to paper over it with. Recovery brings back what the backup holds and proves it; it does not manufacture entries.

Multi-Server recovery

Each Server owns its own database and restores its own slice independently - there is no shared store to coordinate. Cross-Server delegation chains re-link by signature reference, not by restoring a common database: once each Server is restored and verifying, the links across them resolve through the published public keys. Restore and verify Server by Server; the federation re-forms on its own.

Validated against API v1.4.0 on 2026-08-09 (Developer Edition, Docker Compose). Re-run live end to end on a chain that had been through a key rotation: scripts/restore.sh on a real backup tarball ending with the "Restore complete." block, /readyz returning the body shown above, scripts/vault-verify.sh reporting 6 records checked with 0 errors, then scripts/vault-dump.sh and a stock RFC 9052 library verifying all 11 entries against both signing keys with 0 failures. v1.4.0 ships migration 004_signing_agility.sql, taking a fresh install to four migrations; like 003 it is restore-neutral for the same reason. The unsupported_algorithm paragraph was reproduced rather than reasoned about: an Ed25519 chain restored onto a FIPS host running an ES256 vault key scanned as 4 of 4 records broken with that reason and healthy: false, while a dump of the same database verified clean off-host at 11 of 11 signatures.

Validated against API v1.3.4 on 2026-08-03 (Developer Edition, Docker Compose, fresh install from the pinned v1.3.4 release). Re-run live end to end: scripts/restore.sh on a real backup tarball, ending with the "Restore complete." block; /readyz returning {"status":"ready","version":"1.3.4",...}; scripts/vault-verify.sh clean; scripts/vault-dump.sh plus the stock-library offline verification, 0 failures. Confirms that migration 003's append-only trigger on vault_checkpoints does not interfere with restore: the script drops and recreates the database, so pg_restore only inserts and never trips the DELETE/UPDATE guard.

Validated against API v1.1.0 on 2026-06-10 (Developer Edition, Docker Compose: restore → readiness gate → in-database check → stock-library offline verification, end to end).

Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 (audit-vault chain-scan detection plus opt-in verdict per-actor signatures) leaves the restore, readiness-gate, and offline-verification flow unchanged. The offline verifier's failure classes (POSITION_GAP / HASH_MISMATCH) are unaffected.