> Markdown version of https://agledger.ai/docs/operations/backup/
> Full index of this site for AI assistants: https://agledger.ai/llms.txt

# Backup

A complete, restorable backup of a Server is three separate things. Two of them are not in the
database dump, and the operator decides where each lives.

| What | Where it lives | In the DB dump? |
|---|---|---|
| The PostgreSQL database - records, the `audit_vault` chain, the signing-key registry, provisioning state | Postgres | Yes |
| The customer-held encryption keys (encrypted mode) | Wherever you keep them - the Server never stores them | No |
| The config-as-code provisioning files | Your `PROVISIONING_CONFIG_PATH` directory / source control | No |

The split is the point. A database dump without the encryption keys is opaque-but-safe - an
attacker who steals it cannot read encrypted payloads. The encryption keys without the database are
useless. Back up each on its own schedule, to its own place.

This runbook pairs with the [recovery runbook](/docs/operations/recovery/) - a backup you have never
restored is a hope, not a backup.

## Back up the database

`scripts/backup.sh` - from the [`agledger-ai/install`](https://github.com/agledger-ai/install)
repository - is the supported path. It writes a timestamped, compressed tarball and prunes to the
last `--keep N` (default 7):

```bash
./scripts/backup.sh            # keep last 7
./scripts/backup.sh --keep 30
BACKUP_DIR=/mnt/backups ./scripts/backup.sh
```

Each tarball contains the database dump and, in almost every case, the key-registry export. Under
the hood the script runs `pg_dump` in custom format against the bundled postgres (or your external
`DATABASE_URL` for Aurora/RDS):

```
db.dump                  # pg_dump -Fc — the full database, custom format
vault-public-keys.csv    # the signing-key registry, PUBLIC keys only
```

```
PostgreSQL backup complete (732K).
```

The `vault-public-keys.csv` is public-key metadata only - fingerprints, algorithms, status,
activation and retirement dates. Private signing keys are never written to the database and never
appear in a backup:

```
key_id,public_key,algorithm,status,activated_at,retired_at
c4dd3e20388b594d,MCowBQYDK2VwAyEAo95XH8DQ...,Ed25519,active,2026-06-09 15:54:50.700979+00,
```

It is there so that after a restore you can confirm the registry came back intact (see the
[recovery runbook](/docs/operations/recovery/)). A fresh install carries the one active key above;
after a key rotation the CSV lists the active *and* retired keys, because retired keys still verify
records signed before the rotation.

For an external database (Aurora, RDS, Cloud SQL), back up with your provider's snapshot mechanism
instead - `backup.sh` detects an external `DATABASE_URL` and uses `pg_dump` directly. The three-way
split above is unchanged.

`pg_dump` refuses to dump a server newer than itself, and distribution packages lag: Ubuntu 24.04
ships client 16 against the PostgreSQL 18 this product is validated on. `backup.sh` compares the two
and falls back to a matching client in a container when the host's is older, so the case that needs
your attention is a host with neither an adequate client nor docker. It names the PostgreSQL apt
repository lines when it hits that.

**A backup that cannot be read is not written.** Before reporting success, `backup.sh` checks that
`db.dump` begins with the archive header `pg_restore` expects. If it does not, the whole backup
directory is removed and the script exits non-zero naming what it found, rather than leaving a
plausible tarball whose only symptom appears during a restore, after the stack is down and the
database is dropped. The key-registry export gets the same treatment against its header row; a bad
one is dropped and the run says so, leaving a one-file tarball with the database dump intact.

## Back up the chain off-box, too

Keep a second, database-independent copy of the chain: the NDJSON dump the offline verifier
consumes. This is not a replacement for the database backup - it is the artifact you hand an auditor,
and the copy that proves intact without a running Server.

```bash
./scripts/vault-dump.sh ./chain-backup
```

The shipped `scripts/vault-dump.sh` runs the dump tool inside the Server image, so it needs no source
checkout or pnpm - only a reachable database. It prints the dumped row counts and the output
directory:

```json
{
  "outDir": "/dump",
  "orgId": null,
  "counts": { "audit_vault": 10, "vault_checkpoints": 0, "vault_signing_keys": 1, "org_admin_reads": 0, "org_admin_reads_checkpoints": 0 }
}
```

It then prints the same counts as a per-file row list:

```
[...] Vault dump complete
[...]   Directory: /path/to/chain-backup
[...]   audit_vault.ndjson: 10 rows
[...]   org_admin_reads.ndjson: 0 rows
[...]   org_admin_reads_checkpoints.ndjson: 0 rows
[...]   vault_checkpoints.ndjson: 0 rows
[...]   vault_signing_keys.ndjson: 1 rows
```

It writes five NDJSON files (see the [audit runbook](/docs/operations/audit/) for the contents)
including `vault_signing_keys.ndjson` - the public-key registry travels with the dump, so the chain
verifies offline with no further inputs. Keep it alongside the `db.dump` tarball, not instead of it:
the database backup is what you restore a running Server from; the NDJSON dump is what proves the
chain to someone who does not trust your Server.

The tool that reads this dump is [`@agledger/verify`](/docs/verify/), and it takes the directory:

```bash
npx -y @agledger/verify ./chain-backup
```

`@agledger/cli`'s `verify` subcommand is a different tool. It checks one record's audit export
(`GET /v1/records/{id}/audit-export`) and cannot open a dump directory: point it at one and it
reports `EISDIR`. Either verifier is optional: the per-row `cose_sign1` bytes verify under any
RFC 9052 library, which is the path to use when you want no AGLedger package in the loop at all.

## What a backup does not contain

- **Private signing keys.** Held in `VAULT_SIGNING_KEY` (and `VAULT_SIGNING_KEY_PREVIOUS`) - your
  secret store, your responsibility. Back these up with your secrets, not your database. Without
  them a restored Server cannot sign new records (though existing records still verify against the
  public registry).
- **Customer encryption keys.** In encrypted mode, the keys that decrypt payloads never reach the
  Server. They are not in any backup the Server can produce.
- **Provisioning YAML.** Keep `PROVISIONING_CONFIG_PATH` under source control; it is config, not
  data.