Signing-key compromise runbook

Scenario: the Ed25519 private key in VAULT_SIGNING_KEY is suspected or confirmed exposed - a leaked secret store, an exfiltrated .env, a compromised host.

What the key can and cannot do in an attacker's hands: with the key and database write access, an attacker can rewrite chain history and re-sign it so it verifies cleanly. With the key alone they can forge records elsewhere but cannot alter your database. External anchoring is the control that bounds the rewrite risk - if you have not enabled it, do so as part of this incident (VAULT_ANCHOR_ENABLED and the VAULT_ANCHOR_* settings in your install's .env.example).

The sequence is contain, then scope, then prove: rotation stops future forgery, the scan establishes internal consistency, and anchors are what bound the window in which signatures cannot be trusted individually.

The admin calls below require a platform key with the admin:system scope - an org-admin key gets a 403. Confirm you hold one before the incident, not during it.

1. Contain - rotate the key

Generate a new key pair (on the Server host; nothing leaves the box):

docker compose run --rm agledger-api dist/scripts/generate-signing-key.js

Update the environment: put the new private key in VAULT_SIGNING_KEY and remove the compromised key from the environment entirely - do not park it in VAULT_SIGNING_KEY_PREVIOUS. That variable is an optional bootstrap fallback for routine rotation, and historical records do not need it: verification resolves public keys from the signing-key registry in the database, which already holds the compromised key's public half. The private key has no remaining legitimate use, and keeping it resident on the host class that may have leaked it only extends the exposure.

Restart the Server and worker. On boot, the engine reconciles the registry to the configured key: the compromised key is retired, the new key is activated, and every entry notarized from that moment is signed by the new key.

Confirm the registry state - and register the rotation explicitly if you prefer an API-driven record of it:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" \
  "$AGLEDGER_API_URL/v1/admin/vault/signing-keys/rotate"

The call is idempotent. After the restart has already reconciled the registry it returns "status": "already_active" - that is confirmation, not failure. Verify the final state: GET /v1/admin/vault/signing-keys shows the new key active and the compromised key retired with a retiredAt timestamp. Verifiers pick up the published key set from GET /v1/verification-keys and /.well-known/agledger-vault-keys.json automatically.

Replacing the key with one of a different algorithm. Generate with --algorithm es256 and set AGLEDGER_ALLOW_NON_DEFAULT_SIGNING_ALG=true alongside it. The sequence above is otherwise identical, and the registry ends up holding an active key of the new algorithm and a retired key of the old one. Read the scan section below before doing this on a FIPS host, because it changes what your own scan reports.

Rotate the surrounding credentials in the same incident: wherever the key leaked from (secret store, host, backup), assume neighbors leaked too. API_KEY_SECRET has its own graceful rotation path via API_KEY_SECRET_PREVIOUS, and the federation signing key and webhook secrets each rotate separately (see the authentication guide and day-2 operations).

2. Scope - establish the compromise window

Rotation stops future forgery. It does not tell you whether the attacker used the key while they had it. Establish the window: from the earliest plausible exposure to the moment the rotation landed.

Run a full chain scan. The scan is an asynchronous job - the POST returns 202 with a job id:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/vault/scan"

{"jobId":"<uuid>","state":"created"}

Poll GET /v1/admin/vault/scan/{jobId} until state is completed, then read result.healthy. A healthy scan reads "healthy": true with broken and signatureErrors at 0, and proves internal consistency - every hash links, every signature verifies. It does not by itself rule out a rewrite-and-re-sign by the key holder; a forger with the signing key could have re-signed checkpoints too. That is what anchors are for.

If the scan reports `unsupported_algorithm`, read this before escalating

A host whose crypto provider cannot compute the algorithm a retired key used cannot check the signatures that key made. The scan fails those entries closed, and the summary looks alarming:

{"recordsScanned": 4, "verified": 0, "broken": 4, "signatureErrors": 4, "healthy": false,
 "brokenRecords": [{"recordId": "019fe8a5-...", "brokenAt": 1, "reason": "unsupported_algorithm"}]}

This is a host capability gap, not tamper evidence, and unsupported_algorithm is the reason code that says so. The case that produces it in practice is a FIPS host holding pre-FIPS history: the provider carries no Ed25519, so every entry written before the switch to ES256 is unverifiable there, while everything written after it verifies normally.

The chain itself is intact, and you can prove that off-host. Dump it and verify the same bytes somewhere without the FIPS restriction:

./scripts/vault-dump.sh ./dump

The dump's vault_signing_keys.ndjson carries every historical public key, so a stock RFC 9052 (COSE) verifier resolves whichever key signed each entry and reports the chain clean. Run that before treating a scan like the one above as an incident. A genuine tamper reports signature_invalid, payload_drift, or a broken hash link, not unsupported_algorithm.

3. Prove - compare against anchors

Anchors live in Object Lock storage the attacker could not modify, so they are ground truth for everything written before they landed:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" -H "Content-Type: application/json" \
  -d '{"recordId": "<uuid>"}' \
  "$AGLEDGER_API_URL/v1/admin/vault/anchors/verify"

Interpretation:

match: false with detail: "Vault anchoring is not enabled" - you have no anchors at all, and this is not a tamper signal. Check for it first: with anchoring disabled every record returns match: false, and reading that as the mismatch row below is the easiest way to escalate an incident on data that does not support it. Treat it as the "written inside the window, no covering anchor" case applied to your whole history, and go straight to the corroboration paths there.
Anchored before the window, matches - that history is intact. The attacker could not have rewritten it without the mismatch you are looking at right now.
Anchor mismatch, meaning match: false against a checkpoint that really was anchored - concrete evidence of tampering. Preserve the anchor object and the database state; the anchor is the version to trust.
Written inside the window, no covering anchor - the honest residue. Signatures from the compromised key in this span are individually indistinguishable from legitimate ones. Corroborate from outside the chain: your SIEM stream of system_audit_log, webhook deliveries to downstream systems, and counterparty Servers' slices of federated chains.

Checkpoint metadata at GET /v1/audit-vault/checkpoints (an org-admin key with compliance:read suffices here) includes each checkpoint's signingKeyId - checkpoints signed by the compromised key after your rotation timestamp would themselves be suspect, which is another reason the anchor (not the database row) is the authority.

What the product deliberately does not do

There is no compromised key status; keys are active or retired, and retired keys continue to verify their historical records. This is correct for routine rotation - and it means the product will not flag records signed by a compromised key for you. The trust decision over the compromise window is operational and forensic, bounded by your anchors. If your compliance regime needs a machine-readable compromise marker, raise it with us - it has not yet been needed.

The chain is never rewritten, including after an incident. A forged-then-detected span stays in the chain as evidence; remediation is new records and your incident report, not history edits.

Reduce the next window

Enable external anchoring to a bucket in a separately-administered account - it is the single control that bounds this entire runbook. The sweep interval (VAULT_ANCHOR_INTERVAL_MINUTES, default six hours) is your tamper-exposure window.
Keep VAULT_SIGNING_KEY in a real secret manager, not a flat .env on disk.
Alert on agledger_vault_anchor_upload_total{outcome="error"} so the anchor stream never silently stops.

Validated against API v1.3.4 on 2026-08-03 (Developer Edition, Docker Compose, fresh install from the pinned v1.3.4 release). Re-run live: signing-key generation, the env swap and restart, the rotate endpoint, and the registry confirmation (new key active, compromised key retired with a retiredAt, on both /v1/admin/vault/signing-keys and the public /v1/verification-keys). A full chain scan returned healthy, and a post-rotation offline dump verified every entry across both the active and the retired key. One correction landed in this pass: step 3's interpretation list had no row for an install where anchoring was never enabled, which also returns match: false and was therefore indistinguishable from the tamper row during an incident.

Validated against API v1.4.0 on 2026-08-09 (Developer Edition, Docker Compose). Re-run live: key generation, the env swap, the restart, the rotate endpoint answering already_active, and the registry confirmation on both the admin and public key surfaces. A full chain scan returned healthy: true across both keys, and an offline verification of a post-rotation dump verified every entry against whichever of the two keys signed it. The unsupported_algorithm section was reproduced end to end rather than reasoned about: an Ed25519 chain restored onto a v1.4.0 host running the OpenSSL FIPS provider with an ES256 vault key returned exactly the body quoted above, while a dump of that same database verified clean off-host at 11 of 11 signatures across 3 keys.

Validated against API v1.1.0 on 2026-06-10.

Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 hardens audit-vault chain-scan detection and adds opt-in verdict per-actor signatures; the anchoring and key-rotation controls this runbook depends on are unchanged.