Signing-key compromise runbook

Scenario: the Ed25519 private key in VAULT_SIGNING_KEY is suspected or confirmed exposed — a leaked secret store, an exfiltrated .env, a compromised host.

What the key can and cannot do in an attacker's hands: with the key and database write access, an attacker can rewrite chain history and re-sign it so it verifies cleanly. With the key alone they can forge records elsewhere but cannot alter your database. External anchoring is the control that bounds the rewrite risk — if you have not enabled it, do so as part of this incident (VAULT_ANCHOR_ENABLED and the VAULT_ANCHOR_* settings in your install's .env.example).

The sequence is contain, then scope, then prove: rotation stops future forgery, the scan establishes internal consistency, and anchors are what bound the window in which signatures cannot be trusted individually.

The admin calls below require a platform key with the admin:system scope — an org-admin key gets a 403. Confirm you hold one before the incident, not during it.

1. Contain — rotate the key

Generate a new key pair (on the Server host; nothing leaves the box):

docker compose run --rm agledger-api dist/scripts/generate-signing-key.js

Update the environment: put the new private key in VAULT_SIGNING_KEY and remove the compromised key from the environment entirely — do not park it in VAULT_SIGNING_KEY_PREVIOUS. That variable is an optional bootstrap fallback for routine rotation, and historical records do not need it: verification resolves public keys from the signing-key registry in the database, which already holds the compromised key's public half. The private key has no remaining legitimate use, and keeping it resident on the host class that may have leaked it only extends the exposure.

Restart the Server and worker. On boot, the engine reconciles the registry to the configured key: the compromised key is retired, the new key is activated, and every entry notarized from that moment is signed by the new key.

Confirm the registry state — and register the rotation explicitly if you prefer an API-driven record of it:

curl -X POST -H "Authorization: Bearer $PLATFORM_KEY" \
  "$AGLEDGER_URL/v1/admin/vault/signing-keys/rotate"

The call is idempotent. After the restart has already reconciled the registry it returns "status": "already_active" — that is confirmation, not failure. Verify the final state: GET /v1/admin/vault/signing-keys shows the new key active and the compromised key retired with a retiredAt timestamp. Verifiers pick up the published key set from GET /v1/verification-keys and /.well-known/agledger-vault-keys.json automatically.

Rotate the surrounding credentials in the same incident: wherever the key leaked from (secret store, host, backup), assume neighbors leaked too. API_KEY_SECRET has its own graceful rotation path via API_KEY_SECRET_PREVIOUS, and the federation signing key and webhook secrets each rotate separately (see the authentication guide and day-2 operations).

2. Scope — establish the compromise window

Rotation stops future forgery. It does not tell you whether the attacker used the key while they had it. Establish the window: from the earliest plausible exposure to the moment the rotation landed.

Run a full chain scan. The scan is an asynchronous job — the POST returns 202 with a job id:

curl -X POST -H "Authorization: Bearer $PLATFORM_KEY" "$AGLEDGER_URL/v1/admin/vault/scan"

{"jobId":"<uuid>","state":"created"}

Poll GET /v1/admin/vault/scan/{jobId} until state is completed, then read result.healthy. A healthy scan proves internal consistency — every hash links, every signature verifies. It does not by itself rule out a rewrite-and-re-sign by the key holder; a forger with the signing key could have re-signed checkpoints too. That is what anchors are for.

3. Prove — compare against anchors

Anchors live in Object Lock storage the attacker could not modify, so they are ground truth for everything written before they landed:

curl -X POST -H "Authorization: Bearer $PLATFORM_KEY" -H "Content-Type: application/json" \
  -d '{"recordId": "<uuid>"}' \
  "$AGLEDGER_URL/v1/admin/vault/anchors/verify"

Interpretation:

Anchored before the window, matches — that history is intact. The attacker could not have rewritten it without the mismatch you are looking at right now.
Anchor mismatch — concrete evidence of tampering. Preserve the anchor object and the database state; the anchor is the version to trust.
Written inside the window, no covering anchor — the honest residue. Signatures from the compromised key in this span are individually indistinguishable from legitimate ones. Corroborate from outside the chain: your SIEM stream of system_audit_log, webhook deliveries to downstream systems, and counterparty Servers' slices of federated chains.

Checkpoint metadata at GET /v1/audit-vault/checkpoints (an org-admin key with compliance:read suffices here) includes each checkpoint's signingKeyId — checkpoints signed by the compromised key after your rotation timestamp would themselves be suspect, which is another reason the anchor (not the database row) is the authority.

What the product deliberately does not do

There is no compromised key status; keys are active or retired, and retired keys continue to verify their historical records. This is correct for routine rotation — and it means the product will not flag records signed by a compromised key for you. The trust decision over the compromise window is operational and forensic, bounded by your anchors. If your compliance regime needs a machine-readable compromise marker, raise it with us — it has not yet been needed.

The chain is never rewritten, including after an incident. A forged-then-detected span stays in the chain as evidence; remediation is new records and your incident report, not history edits.

Reduce the next window

Enable external anchoring to a bucket in a separately-administered account — it is the single control that bounds this entire runbook. Checkpoints sweep every six hours.
Keep VAULT_SIGNING_KEY in a real secret manager, not a flat .env on disk.
Alert on agledger_vault_anchor_upload_total{outcome="error"} so the anchor stream never silently stops.

Validated against API v1.0.0 on 2026-06-10.