> Markdown version of https://agledger.ai/docs/operations/key-compromise/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # Signing-key compromise runbook Scenario: the Ed25519 private key in `VAULT_SIGNING_KEY` is suspected or confirmed exposed - a leaked secret store, an exfiltrated `.env`, a compromised host. What the key can and cannot do in an attacker's hands: with the key **and** database write access, an attacker can rewrite chain history and re-sign it so it verifies cleanly. With the key alone they can forge records elsewhere but cannot alter your database. External anchoring is the control that bounds the rewrite risk - if you have not enabled it, do so as part of this incident (`VAULT_ANCHOR_ENABLED` and the `VAULT_ANCHOR_*` settings in your install's `.env.example`). The sequence is contain, then scope, then prove: rotation stops future forgery, the scan establishes internal consistency, and anchors are what bound the window in which signatures cannot be trusted individually. The admin calls below require a **platform** key with the `admin:system` scope - an org-admin key gets a 403. Confirm you hold one before the incident, not during it. ## 1. Contain - rotate the key Generate a new key pair (on the Server host; nothing leaves the box): ```bash docker compose run --rm agledger-api dist/scripts/generate-signing-key.js ``` Update the environment: put the new private key in `VAULT_SIGNING_KEY` and **remove the compromised key from the environment entirely** - do not park it in `VAULT_SIGNING_KEY_PREVIOUS`. That variable is an optional bootstrap fallback for routine rotation, and historical records do not need it: verification resolves public keys from the signing-key registry in the database, which already holds the compromised key's public half. The private key has no remaining legitimate use, and keeping it resident on the host class that may have leaked it only extends the exposure. Restart the Server and worker. On boot, the engine reconciles the registry to the configured key: the compromised key is retired, the new key is activated, and every entry notarized from that moment is signed by the new key. Confirm the registry state - and register the rotation explicitly if you prefer an API-driven record of it: ```bash curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" \ "$AGLEDGER_API_URL/v1/admin/vault/signing-keys/rotate" ``` The call is idempotent. After the restart has already reconciled the registry it returns `"status": "already_active"` - that is confirmation, not failure. Verify the final state: `GET /v1/admin/vault/signing-keys` shows the new key `active` and the compromised key `retired` with a `retiredAt` timestamp. Verifiers pick up the published key set from `GET /v1/verification-keys` and `/.well-known/agledger-vault-keys.json` automatically. **Replacing the key with one of a different algorithm.** Generate with `--algorithm es256` and set `AGLEDGER_ALLOW_NON_DEFAULT_SIGNING_ALG=true` alongside it. The sequence above is otherwise identical, and the registry ends up holding an active key of the new algorithm and a retired key of the old one. Read the scan section below before doing this on a FIPS host, because it changes what your own scan reports. Rotate the surrounding credentials in the same incident: wherever the key leaked from (secret store, host, backup), assume neighbors leaked too. `API_KEY_SECRET` has its own graceful rotation path via `API_KEY_SECRET_PREVIOUS`, and the federation signing key and webhook secrets each rotate separately (see the [authentication guide](/docs/guides/authentication/) and [day-2 operations](/docs/operations/day-2/)). ## 2. Scope - establish the compromise window Rotation stops future forgery. It does not tell you whether the attacker used the key while they had it. Establish the window: from the earliest plausible exposure to the moment the rotation landed. Run a full chain scan. The scan is an asynchronous job - the POST returns `202` with a job id: ```bash curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/vault/scan" ``` ```json {"jobId":"","state":"created"} ``` Poll `GET /v1/admin/vault/scan/{jobId}` until `state` is `completed`, then read `result.healthy`. A healthy scan reads `"healthy": true` with `broken` and `signatureErrors` at 0, and proves internal consistency - every hash links, every signature verifies. It does **not** by itself rule out a rewrite-and-re-sign by the key holder; a forger with the signing key could have re-signed checkpoints too. That is what anchors are for. ### If the scan reports `unsupported_algorithm`, read this before escalating A host whose crypto provider cannot compute the algorithm a retired key used cannot check the signatures that key made. The scan fails those entries closed, and the summary looks alarming: ```json {"recordsScanned": 4, "verified": 0, "broken": 4, "signatureErrors": 4, "healthy": false, "brokenRecords": [{"recordId": "019fe8a5-...", "brokenAt": 1, "reason": "unsupported_algorithm"}]} ``` This is a host capability gap, not tamper evidence, and `unsupported_algorithm` is the reason code that says so. The case that produces it in practice is a FIPS host holding pre-FIPS history: the provider carries no Ed25519, so every entry written before the switch to ES256 is unverifiable there, while everything written after it verifies normally. The chain itself is intact, and you can prove that off-host. Dump it and verify the same bytes somewhere without the FIPS restriction: ```bash ./scripts/vault-dump.sh ./dump ``` The dump's `vault_signing_keys.ndjson` carries every historical public key, so a stock RFC 9052 (COSE) verifier resolves whichever key signed each entry and reports the chain clean. Run that before treating a scan like the one above as an incident. A genuine tamper reports `signature_invalid`, `payload_drift`, or a broken hash link, not `unsupported_algorithm`. ## 3. Prove - compare against anchors Anchors live in Object Lock storage the attacker could not modify, so they are ground truth for everything written before they landed: ```bash curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" -H "Content-Type: application/json" \ -d '{"recordId": ""}' \ "$AGLEDGER_API_URL/v1/admin/vault/anchors/verify" ``` Interpretation: - **`match: false` with `detail: "Vault anchoring is not enabled"`** - you have no anchors at all, and this is not a tamper signal. Check for it first: with anchoring disabled every record returns `match: false`, and reading that as the mismatch row below is the easiest way to escalate an incident on data that does not support it. Treat it as the "written inside the window, no covering anchor" case applied to your whole history, and go straight to the corroboration paths there. - **Anchored before the window, matches** - that history is intact. The attacker could not have rewritten it without the mismatch you are looking at right now. - **Anchor mismatch**, meaning `match: false` against a checkpoint that really was anchored - concrete evidence of tampering. Preserve the anchor object and the database state; the anchor is the version to trust. - **Written inside the window, no covering anchor** - the honest residue. Signatures from the compromised key in this span are individually indistinguishable from legitimate ones. Corroborate from outside the chain: your SIEM stream of `system_audit_log`, webhook deliveries to downstream systems, and counterparty Servers' slices of federated chains. Checkpoint metadata at `GET /v1/audit-vault/checkpoints` (an org-admin key with `compliance:read` suffices here) includes each checkpoint's `signingKeyId` - checkpoints signed by the compromised key after your rotation timestamp would themselves be suspect, which is another reason the anchor (not the database row) is the authority. ## What the product deliberately does not do There is no `compromised` key status; keys are `active` or `retired`, and retired keys continue to verify their historical records. This is correct for routine rotation - and it means the product will not flag records signed by a compromised key for you. The trust decision over the compromise window is operational and forensic, bounded by your anchors. If your compliance regime needs a machine-readable compromise marker, raise it with us - it has not yet been needed. The chain is never rewritten, including after an incident. A forged-then-detected span stays in the chain as evidence; remediation is new records and your incident report, not history edits. ## Reduce the next window - Enable [external anchoring](/docs/operations/anchoring/) to a bucket in a separately-administered account - it is the single control that bounds this entire runbook. The sweep interval (`VAULT_ANCHOR_INTERVAL_MINUTES`, default six hours) is your tamper-exposure window. - Keep `VAULT_SIGNING_KEY` in a real secret manager, not a flat `.env` on disk. - Alert on `agledger_vault_anchor_upload_total{outcome="error"}` so the anchor stream never silently stops.