Signing-key compromise runbook

Scenario: the vault signing key's private half (VAULT_SIGNING_KEY, or the file or SSM parameter that supplies it) is suspected or confirmed exposed: a leaked secret store, an exfiltrated .env, a compromised host.

A key held in AWS KMS (VAULT_SIGNING_KEY_KMS_ARN) cannot leak that way, because its private half never leaves KMS. The equivalent incident there is a principal that should not hold kms:Sign on the key: tighten the key policy first, read CloudTrail for every Sign call in the window, and run the same three steps with a new key ARN (the old one goes in VAULT_SIGNING_KEY_PREVIOUS_KMS_ARN where the steps below use VAULT_SIGNING_KEY_PREVIOUS).

What the key can and cannot do in an attacker's hands: database write access alone cannot plant a key, because a key row that no signed key statement links to a key held outside the database verifies nothing and is not published. With the key and database write access, an attacker can rewrite chain history and re-sign it so it verifies cleanly. With the key alone they can forge records elsewhere but cannot alter your database. External anchoring is the control that bounds the rewrite risk. If you have not enabled it, do so as part of this incident (VAULT_ANCHOR_ENABLED and the VAULT_ANCHOR_* settings in your install's .env.example; see external anchoring).

The sequence is contain, then scope, then prove: rotation and distrust stop future forgery, the scan establishes internal consistency, and anchors bound the window in which signatures cannot be trusted individually.

The admin calls below require a platform key with the admin:system scope; an org-admin key gets a 403. Confirm you hold one before the incident, not during it.

1. Contain: rotate, retire, distrust

Check first whether the attacker already retired the compromised key. If GET /v1/admin/vault/signing-keys shows it retired though you have not retired it, usually beside an active key you did not stage, the steps in this section do not apply. Go to If the attacker already retired the key instead.

Stage a fresh key with the compromised key as its predecessor

Generate a new key on the Server host (nothing leaves the box):

docker compose run --rm agledger-api dist/scripts/generate-signing-key.js

Put the new private key in VAULT_SIGNING_KEY and the compromised key in VAULT_SIGNING_KEY_PREVIOUS, then restart the api and worker processes. On Helm the same settings are secrets.vaultSigningKey, secrets.vaultSigningKeyPrevious (signing.previousKmsKeyArn for a KMS-held key), signing.trustAnchors and signing.distrustedKeys; the variables named below map onto them. A new key is trusted only through a succession statement signed by both it and the key in use, so the compromised key has to sign this one; the forced retirement below is what stops it admitting anything else. A process on a new key with no predecessor to sign for it registers nothing and signs nothing (signingKey.gate: "unanchored" on /health, 503 on /health/ready). If you cannot trust any successor the compromised key signs, use No successor you trust instead.

The restart stages the key and retires nothing. On boot the engine registers the new key and activates it beside every key already active, the compromised one included, and every process you have not restarted yet keeps signing with whatever key it holds.

Before you retire, confirm each api and worker process has rolled onto the new key: GET /health on each process, not through a load balancer, reports signingKey.keyId. GET /v1/admin/vault/signing-keys reports lastSignedAt per key, but a null there does not prove a process has rolled: an idle process, or one whose only use of the key is webhook deliveries, certificates or Receipts, appends nothing while still holding the key.

POST /v1/admin/vault/signing-keys/rotate performs the same staging the restart does and confirms it landed. Against a process already restarted on the new key it answers already_active:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" \
  "$AGLEDGER_API_URL/v1/admin/vault/signing-keys/rotate"

{"newKeyId":"6ded5f16fd0530e1","status":"already_active","activeKeys":[...],"nextSteps":[...]}

Retire the compromised key with force

Retirement is a separate, explicit call, and for a compromise you make it immediately rather than waiting for the fleet to roll. Send it to a process on the new key; a process on the compromised key answers 409.

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" -H "Content-Type: application/json" \
  -d '{"force": true}' \
  "$AGLEDGER_API_URL/v1/admin/vault/signing-keys/<compromised-key-id>/retire"

Without force the endpoint refuses with 422 while the key has appended a chain entry in the last 300 seconds, on the assumption you want every process confirmed off it first. Here you do not wait. Retiring the only active key is always refused, which is why staging the replacement comes first.

What the forced retirement does:

• It closes the key's window for chain appends the instant it commits. The retirement and any append still in flight take the same row lock, so no chain entry is written under the compromised key after the call returns. Nor can one be forged later and dated back: the database stamps each entry's write time itself, so an entry signed with the compromised key after the retirement reads as written after its retiredAt and fails as key_expired (CHAIN_KEY_EXPIRED offline).
• It writes a forced closure statement, signed by the calling process's key. Every edge out of the compromised key is void, whenever it signed it. No key it admitted (such as one an attacker staged with it) is trusted through it, and nor is any key it names as its predecessor, since a leaked key can sign a succession naming a key of the attacker's as its own predecessor. Entries the compromised key signed before the retirement still verify, because the edge from your new key back to it is the new key's own and the forced closure does not void it.
• It lists what it cut off. The engine walks the key registry again with the closure in place, and every key the walk reached before the call and no longer reaches is in unanchoredKeyIds beside the closureDigest. Each such row is retired in the same transaction and entries it signed break as signing_key_unanchored. A key already unanchored before the call is not in the list; the scan's keyRegistry.unanchoredKeyIds shows those.
• It revokes certificates. Every unexpired ephemeral cert the compromised key, or a key in unanchoredKeyIds, minted is revoked in the same transaction (revokedCertCount). Their holders mint replacements with POST /v1/auth/oidc/cert.

A process that has not yet noticed the retirement (api and worker processes notice at once over the LISTEN connection the retirement broadcasts on, and otherwise at their 30-second re-read of the registry) can still sign RFC 9421 webhook deliveries, ephemeral-certificate JWS and Receipts under the compromised key until it does, and a receiver checking those against the published window rejects them. A process that has noticed fails GET /health/ready with signingKey.gate: "retired", which takes it out of the load balancer's rotation, and a worker also stops consuming jobs (jobConsumption: "stopped_signing_key_retired" on /health), so queued work waits for a process on the new key. Restart each one.

A key you retired on schedule and then find leaked takes the same call. On a key already retired, force: true writes a forced closure with the same effect and revokes the certs. The response carries alreadyRetired: true and the original retiredAt, which does not move unless it lies in the future, in which case the closure signs the moment of the call. On a key a forced closure already closes, it writes no second closure.

Distrust the compromised key

The forced retirement voids what the compromised key admitted, but not the closures it signs. A closure only takes trust away, so its worst case is shortening a key's window, and that is severe: a closure it signs later, dated before your current key was activated, ends that key's window at the date it names, so every entry the current key signed reads as key_expired and its certificates stop authenticating. Nothing inside the database undoes that. A key retired on schedule whose private half leaks later can do the same.

VAULT_DISTRUSTED_KEYS stops it. On every process, set it to the compromised key's pin, remove VAULT_SIGNING_KEY_PREVIOUS wherever it names the compromised key, and restart. The retire response carries the pin as retiredSpkiSha256, and its nextSteps spell out the entry:

VAULT_DISTRUSTED_KEYS=sha256:<retiredSpkiSha256>

What the key signs from its retirement on then counts for nothing; what it signed before keeps counting, so its own honest closure of its predecessor still holds. If it leaked before it was retired, give the instant too (sha256:<pin>@2026-09-01T00:00:00Z), so what it signed from the leak on counts for nothing as well. The list is comma-separated, one entry per key.

Removing the predecessor is safe once the process's own key is registered (signingKey.gate: "usable" on GET /health), because the succession it signed is already stored. A process refuses to boot when VAULT_DISTRUSTED_KEYS names its own key or its VAULT_SIGNING_KEY_PREVIOUS, and the error says to drop the predecessor, not the entry. It also refuses a key that is both in VAULT_TRUST_ANCHORS and distrusted with no instant; a pin beside a dated entry for the same key is accepted, and vouches only for what the key signed before the instant.

Give auditors the same entry with the fresh key's pin. @agledger/verify 2.0.0 or later takes them as --distrusted-key and --trust-anchor, on an audit export and on a vault dump alike:

npx @agledger/verify ./dump \
  --trust-anchor sha256:<fresh-key-pin> \
  --distrusted-key sha256:<compromised-key-pin>@<instant>

A dump carries the registry rows as stored and no distrust entry, so an auditor who leaves the entry out of a dump run passes the entries the instant took away. An audit export lists the key cut at the instant, with distrustedFrom saying so, and an auditor without the entry fails it as CHAIN_KEY_WINDOW_DRIFT on that key. That field is the Server's unsigned word, so the auditor confirms the entry with you before passing it.

A closure the compromised key signs names it as the endorser, so the scan's keyRegistry findings and the vault_key_statements rows show where it came from: a closure that still counts though its signer stored it after its own retirement, or that dates a retirement before its subject's activation, is reported as key_closure_invalid with the entry to add in its detail. A genesis or succession the compromised key signs for itself after it was admitted (naming a new predecessor, or redating itself) is reported as key_statement_invalid, travels with the key's statements, and fails every audit export offline; no setting removes it, because it is the record of the leak. Once VAULT_DISTRUSTED_KEYS names the key from an instant no later than that statement was stored, and a key you trust has retired it with force, the scan marks the finding standing: true: still listed, and no longer counted against healthy, the daily integrity check or system health.

Re-anchor what you vouch for, and hand out a new pin

A forced closure takes trust away from everything reached only through the compromised key, honest keys included: the keys it admitted, and the history before it, which was reached back through it.

• For each key in unanchoredKeyIds that you know is honest, add its pin to VAULT_TRUST_ANCHORS on every process and restart. Until you do, the scan reports the statements that admit it as key_statement_invalid. Leave out every key you cannot vouch for; its statements stay reported.
• If the scan then reports key_closure_invalid for a key you re-anchored (one the forced retirement retired), call the retire endpoint for it without force to sign that retirement. A key whose retirement a closure already signs needs only the pin, and the call refuses it.
• A process that signed with such a key restarts on a fresh key, with VAULT_SIGNING_KEY_PREVIOUS set to a key that is still anchored.
• An auditor whose pin reached the current keys only through the compromised key no longer reaches them, and one pinned on a current key no longer reaches the history behind it. Give each auditor the pin of the key that ran the retirement (anchoredFrom on GET /v1/verification-keys, checked against your own copy) plus every pin you added to VAULT_TRUST_ANCHORS.

signing-key-digest.js prints a key's pin. It reads VAULT_SIGNING_KEY from its environment and prints only the pin. Inside a running container that is the key the process signs with. On Compose, from the install directory:

cd compose && docker compose exec -T -e NODE_OPTIONS= agledger-api /nodejs/bin/node dist/scripts/signing-key-digest.js

For a key no container holds yet (a fresh key, or one you are vouching for), hand it in for that one run:

cd compose && docker compose run --rm --no-deps -T -e NODE_OPTIONS= -e VAULT_SIGNING_KEY='<base64 key>' \
  --entrypoint /nodejs/bin/node agledger-api dist/scripts/signing-key-digest.js

On Kubernetes:

kubectl exec -n <namespace> deploy/<release>-agledger-chart-api -- \
  env NODE_OPTIONS= /nodejs/bin/node dist/scripts/signing-key-digest.js

A key held in AWS KMS takes --kms-arn <arn> instead, and --help lists every form.

Confirm containment

GET /v1/admin/vault/signing-keys shows the new key active and the compromised key retired with a retiredAt timestamp; that timestamp is the end of the compromise window you scope in step 2. Both halves are notarized: the staging and the retirement each write a KEY_ROTATED entry on the platform-ops chain and a vault.signing_key_rotated or vault.signing_key_retired row on the SIEM stream, so a SOC watching the stream sees the containment step rather than reading it out of the registry afterwards. Verifiers pick up the published key set from GET /v1/verification-keys and /.well-known/agledger-vault-keys.json automatically.

Replacing the key with one of a different algorithm. Generate with --algorithm es256 and set AGLEDGER_ALLOW_NON_DEFAULT_SIGNING_ALG=true alongside it; FIPS 140 hosts carries that path in full. Read it before doing this on a FIPS host, because it changes what your own scan reports.

Rotate the surrounding credentials in the same incident: wherever the key leaked from (secret store, host, backup), assume neighbors leaked too. API_KEY_SECRET has its own graceful rotation path via API_KEY_SECRET_PREVIOUS, and the federation signing key and webhook secrets each rotate separately (see the authentication guide and day-2 operations).

If the attacker already retired the key

With the leaked half, an attacker can stage a key of their own as its successor and have that key retire the leaked one. The compromised key can then no longer sign a successor in: a closure already counts against it, so a new key staged with it as VAULT_SIGNING_KEY_PREVIOUS stays unanchored, and that stays true after you distrust the attacker's key. /health/ready on that process says so. It names an active key as the predecessor to use only when the trust walk still anchors one, and otherwise names this path. An attacker's key is anchored, and can be named, only when the compromised key admitted it and was then retired without force; if /health/ready names a key you did not stage, do not use it.

No key surface publishes the attacker's key, and every key document carries its closure with the leaked key's statements, so no auditor can verify it. Until you recover, the scan reports it as key_closure_invalid, and every record's audit export answers chainIntegrity: false with chainIntegrityReason: signing_key_material_invalid, whichever key signed the record, as @agledger/verify does with KEY_STATEMENT_INVALID. Recover this way:

1. On every api and worker process set VAULT_SIGNING_KEY to a fresh key, remove VAULT_SIGNING_KEY_PREVIOUS, set VAULT_TRUST_ANCHORS to the compromised key's pin plus the pin of each earlier key whose history you vouch for, and restart. The fresh key registers under a fresh genesis: GET /health reads signingKey.gate: "usable". What the compromised key signed before its retirement keeps verifying, forged or not, which step 3's anchors bound. An earlier key you leave out stays cut off: the forced retirement in the next step voids the compromised key's edge back to it, the scan then reports the compromised key's admission as key_statement_invalid naming that key's pin, and every export stays uncertified until you add it.
2. From a process on the fresh key, retire the compromised key with force. It answers alreadyRetired: true and writes a forced closure signed by a key you hold. Read the retiredAt it returns: it keeps the retirement already recorded, or the moment of this call when that instant lies in the future (an attacker can date a retirement ahead to keep the key valid), and the compromised key's window ends there.
3. Run the scan (step 2 of this runbook). It reports the attacker's closure as key_closure_invalid, and its detail gives the attacker key's VAULT_DISTRUSTED_KEYS entry, sha256:<pin>@<instant>, the instant being when the closure was written. Add that entry on every process and restart. Until you do, every record's audit export answers chainIntegrity: false with chainIntegrityReason: signing_key_material_invalid, and offline verifiers fail each one with KEY_STATEMENT_INVALID.
4. Retire the attacker's key with force from the fresh key too. Nothing it signs verifies either way; the retirement is what bounds what the distrust entry accounts for. Until it lands, the scan reports the attacker's key as key_closure_invalid (no closure a key you trust signs retires it), and the entries it signed break the platform-ops chain as signing_key_unanchored.
5. Give auditors the fresh key's pin and every pin from step 1 as --trust-anchor, and the attacker key's entry as --distrusted-key. The Server certifies an export only as a verifier given the same pins and distrust entries reads it.

Where this path ends. After step 4 a full scan reads healthy: true, with broken, signatureErrors and globalChains.broken at 0 and keyRegistry.findings empty. What the attacker signed is listed, not hidden and not verified: the two KEY_ROTATED entries the attacker's key wrote on the platform-ops chain (staging itself, and retiring the compromised key) appear under distrustedEntries (the signing_key_distrusted class), that chain is counted in globalChains.distrustedSigned, and the attacker's closure of the compromised key appears under keyRegistry.distrustedStatements. A record chain carrying an entry the attacker's key signed is counted the same way, and its audit export answers chainIntegrity: false with chainIntegrityReason: signing_key_distrusted; every other export certifies. A ./scripts/vault-dump.sh dump verified with the same --trust-anchor and --distrusted-key lists the same entries and statement as accounted for and passes. Anything the attacker's key signs after step 4 is not accounted for: the scan reports it and reads healthy: false again.

If the compromised key signs again after this. The pin in step 1 trusts what the compromised key signs inside its window, and its private half is still out. With database write access, whoever holds it can store a closure of your fresh key dated just after the fresh key's activation. The scan reports it as key_closure_invalid, the fresh key's published window ends at the date it names, and every process on the fresh key gates as retired: /health/ready answers 503 and /status stops reading chain writes operational. The Server cannot tell which of two keys that retire each other is the honest one, so the walk counts both until you say. The finding's detail gives the entry that does: VAULT_DISTRUSTED_KEYS with the compromised key's pin and an instant (sha256:<pin>@<instant>). Use the earliest time the key may have leaked; the detail suggests its retirement only for when nothing earlier is known. Add it on every process beside the pin and restart. The pin then vouches for everything the key stored before the instant, which is why the instant must not be later than the leak. The closure counts for nothing, the fresh key's window reopens, and its processes read usable again. The closure stays a scan finding, because nothing you did accounts for it: someone with write access to the database still holds the compromised key, which is the next thing to close.

No successor you trust, or a lost VAULT_SIGNING_KEY_PREVIOUS

Set VAULT_TRUST_ANCHORS to the pins (sha256:<64 hex>, comma-separated) of the keys whose history you vouch for, set the new VAULT_SIGNING_KEY, and restart. A process whose key nothing links to that history then registers under a fresh genesis; give auditors the new key's pin. There is no way around this step: without an authority held outside the database, nothing can tell your successor from one an attacker staged.

A rewind open during the compromise

When the compromised key signed a rewind acknowledgement, the forced retirement (or the VAULT_DISTRUSTED_KEYS entry) makes it cover nothing, and chain writes answer 409 CHAIN_REWIND_DETECTED again. The same happens when the next step of this incident is a restore to a backup taken before the compromise and the anchors show positions past it. The way out never runs on the compromised key, and its order keeps the compromised key's honest history anchored:

1. Stage the fresh key with the compromised key as its predecessor (VAULT_SIGNING_KEY the fresh key, VAULT_SIGNING_KEY_PREVIOUS the compromised one, no VAULT_DISTRUSTED_KEYS entry yet) and restart every process. It registers even while writes are refused: the signing-key registry's own entries, and a retirement's, are what the refusal lets through, because the acknowledgement has to be signed by a registered key. Confirm signingKey.gate: "usable" and the new signingKey.keyId on GET /health of each process.
2. Retire the compromised key with force, from a process on the fresh key. After a restore, do it again even if you retired it before: the retirement is held in the database, so a backup taken before it brings the key back active, and a restore replays revoked credentials but not signing-key retirements.
3. Swap the predecessor for the distrust entry on every process: remove VAULT_SIGNING_KEY_PREVIOUS, set VAULT_DISTRUSTED_KEYS to the compromised key's pin (with @<instant> if it leaked before the retirement), and restart. The fresh key's succession is stored, so it stays anchored, and the edge back from it keeps what the compromised key signed before its cutoff verifying. The forced retirement in step 2 cut the history before the compromised key, so the scan reports what that history needs: a key_statement_invalid (or, for a key it retired without signing it, key_closure_invalid) naming each predecessor's pin. Add the pins of the predecessors you vouch for to VAULT_TRUST_ANCHORS on every process and restart, sign each retirement the scan then asks for with an unforced retire call, and give auditors the same pins with the distrust entry. Until then every record's audit export answers chainIntegrity: false with chainIntegrityReason: signing_key_material_invalid. Entries and statements the compromised key signed from the instant until step 2's retirement stop verifying, honest ones included, and the scan lists them rather than counting them broken: entries under distrustedEntries (the signing_key_distrusted class), statements under keyRegistry.distrustedStatements. If it signed an acknowledgement after the instant, that RESTORE_EPOCH is one of those entries: never verified, covering nothing. AGLedgerVaultKeyWindowViolation fires for each chain carrying such an entry (reason="signing_key_distrusted"). An entry the compromised key signed after step 2 stays broken as key_expired.
4. Read GET /v1/admin/vault/rewind. openFindings lists every finding the compromised key's acknowledgements covered, each with acknowledgement.reason naming the retirement or the distrust entry, beside any finding no acknowledgement has touched. They carry no previousAcknowledgement, because that is shown only for an acknowledgement whose RESTORE_EPOCH still verifies. A reconciliation on a process whose signingKey.gate is not usable cannot judge them and records them open again; the exception is a KMS-held key that is not answering (signer_unreachable), where the reconciliation records and clears nothing and reports itself incomplete.
5. Acknowledge at POST /v1/admin/vault/rewind/acknowledge on a process on the fresh key. The call records those findings again and covers them in a RESTORE_EPOCH signed under the fresh key; coveredFindings says how many. A process whose signingKey.gate is not usable answers 503 SIGNING_KEY_UNUSABLE and changes nothing.

Do not stage the fresh key through VAULT_TRUST_ANCHORS with its own pin for this. That starts a fresh genesis with no link to the compromised key, so everything it signed (entries from before the leak included), the platform-ops chain and every chain carrying one of its entries read as signing_key_unanchored. The checkpoint sweep then withholds checkpoints for those chains while they keep taking writes (agledger_vault_checkpoint_skipped_broken_total, AGLedgerCheckpointSkippedBrokenChain), and while VAULT_DISTRUSTED_KEYS names the compromised key no pin can re-anchor it. That path is for the case where no key in hand links to the history.

2. Scope: establish the compromise window

Rotation stops future forgery. It does not tell you whether the attacker used the key while they had it. Establish the window: from the earliest plausible exposure to the moment the retirement landed.

Run a full chain scan. The scan is an asynchronous job; the POST returns 202 with a job id:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/vault/scan"

Poll GET /v1/admin/vault/scan/{jobId} until state is completed, then read result.healthy. A clean result reads "healthy": true with broken and signatureErrors at 0, and proves internal consistency: every hash links, every signature verifies, and the key statements walk cleanly. Anything signed with the leaked key after its retiredAt falls outside the key's published window and is reported as key_expired.

After a distrust entry, what the distrusted key signed from its instant until a key you trust retired it is listed, not counted broken: chain entries under distrustedEntries (the signing_key_distrusted class) and statements under keyRegistry.distrustedStatements, none of them verified. healthy reads true beside them when nothing else is wrong. What it signed after that retirement, or anything a distrusted key signed while no key you trust has retired it, stays a finding and keeps the scan at healthy: false; the audit runbook has the detail.

A healthy scan does not by itself rule out a rewrite-and-re-sign by the key holder inside the window; a forger with the signing key could have re-signed checkpoints too. That is what anchors are for.

What auditors see while a statement cannot be verified

An offline verifier given --trust-anchor walks the key statements an export or dump carries before any entry, and a finding there fails the whole run at position 0. KEY_STATEMENT_INVALID is a statement it cannot verify, most often one signed by a key no key surface publishes (an attacker's closure, or the admission of a key whose endorser a forced retirement cut off). KEY_CLOSURE_INVALID is a key listed as retired with no closure it can verify, and CHAIN_KEY_WINDOW_DRIFT a listed activatedAt or retiredAt its own walk dates differently; both usually follow from the first. The scan names each statement with its remedy, and the audit runbook lists them. An export or dump taken before the remedy still carries the old statements: take a new one.

If the scan reports unsupported_algorithm, read this before escalating

A host whose crypto provider cannot compute the algorithm a retired key used cannot check the signatures that key made. The scan counts those chains under result.unsupportedAlgorithm and lists them in brokenRecords with reason unsupported_algorithm. They are not in broken and do not change healthy, so a scan can read "healthy": true while leaving them unchecked. That is a host capability gap rather than tamper evidence, and healthy: true here says nothing was found wrong, not that those chains were checked. A chain that also breaks on a hash, link, drift, signature or key window at or after the unsupported entry is counted in broken instead, with firstFinding naming the unsupported entry; FIPS 140 hosts covers the case that produces it. Those chains are provable off-host: dump the vault with ./scripts/vault-dump.sh and verify the same bytes somewhere without the restriction, following the off-box verification in the audit runbook. Do that before treating such a scan as clean or as an incident. A genuine tamper reports signature_invalid, payload_drift, or a broken hash link, not unsupported_algorithm.

3. Prove: compare against anchors

Anchor writes are create-only (If-None-Match): a second write to a position already anchored is refused by the store and reported as a fork rather than landing over the first. Under Object Lock that refusal is backed by a version nothing can delete, so anchors written before the compromise window are ground truth for everything written before they landed. POST /v1/admin/vault/anchors/reconcile names a store that does not honour conditional writes in its posture report; what follows assumes it does.

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" -H "Content-Type: application/json" \
  -d '{"recordId": "<uuid>"}' \
  "$AGLEDGER_API_URL/v1/admin/vault/anchors/verify"

Read outcome, not match. match is true only for verified; every other outcome answers false, and only tamper, fork, hidden and empty_body are findings. The ones that matter in an incident:

• anchoring_disabled (detail: "Vault anchoring is not enabled"): you have no anchors at all, and this is not a tamper signal. Check for it first: with anchoring disabled every record returns this outcome, and reading it as a mismatch is the easiest way to escalate an incident on data that does not support it. Treat it as the no_such_key case below applied to your whole history, and go straight to the corroboration paths there.
• verified (match: true): anchored before the window, and it matches. That history is intact; the attacker could not have rewritten it without producing tamper or fork here.
• tamper: the anchor exists and disagrees with the database. Concrete evidence of tampering: preserve the anchor object and the database state, and treat the anchor as the version to trust.
• fork: the anchor key holds two anchor documents of this position over different chain tips, which is what a rewind followed by a fresh anchor write looks like even when the latest version matches the database. storedVersions carries the count. This is also concrete evidence, and it stops chain writes on this Server with 409 CHAIN_REWIND_DETECTED until an operator acknowledges it at POST /v1/admin/vault/rewind/acknowledge.
• superseded: a tamper or fork at a position inside a rewind already acknowledged on this Server, in an anchor written no later than that acknowledgement. The anchor holds the history the restore replaced; it does not stop writes. If the compromise could have reached the acknowledgement itself, compare its RESTORE_EPOCH entry and the chain.rewind_acknowledged SIEM row against your own change record. An acknowledgement signed by a key you have retired with force, or named in VAULT_DISTRUSTED_KEYS, covers nothing, whenever it was signed; see A rewind open during the compromise.
• hidden: the anchor was deleted without a version id, so a delete marker hides it while the retention-locked version remains. Nothing the Server does deletes an anchor, so someone with write access to the bucket did; detail says whether the hidden version matches the database. Treat the hidden version as the anchor and the deletion as part of the incident.
• empty_body: the anchor object exists and is zero length, which is what overwriting one looks like without also forging a matching envelope. Treat it as tamper.
• no_such_key: no covering anchor for this position, and no hidden version of one. That is the ordinary state of any checkpoint younger than the anchor interval and is not a finding by itself. Where it falls inside the compromise window it is the residue: signatures from the compromised key in that span are individually indistinguishable from legitimate ones. Corroborate from outside the chain: your SIEM stream of system_audit_log, webhook deliveries to downstream systems, and counterparty Servers' slices of federated chains.

missing_or_hidden and s3_error compared nothing: the store could not be read, or a plain read answered NoSuchKey and the key's versions could not be listed, so a missing anchor cannot be told from a hidden one (versionsNote or detail says why). Fix the store access and verify again before drawing a conclusion. no_checkpoint means the record has no checkpoint at that position yet, so there is nothing to compare. The anchoring runbook covers every outcome.

Checkpoint metadata at GET /v1/audit-vault/checkpoints (an org-admin key with compliance:read suffices here) includes each checkpoint's signingKeyId. Checkpoints signed by the compromised key after your retirement timestamp would themselves be suspect, which is another reason the anchor (not the database row) is the authority.

Key status after an incident

Keys are active or retired; there is no compromised status. A retired key continues to verify what it signed before its retirement, with two exceptions, both from step 1. A key named in VAULT_DISTRUSTED_KEYS with an instant stops verifying what it signed from that instant on, because its published window ends there; the scan lists what it signed between that instant and its retirement as signing_key_distrusted rather than counting it broken. A key a forced retirement left in unanchoredKeyIds stops verifying everything it signed until you pin it in VAULT_TRUST_ANCHORS.

Verifying up to the retirement is the behaviour routine rotation needs, and it means records signed by a compromised key inside its window stay verifiable, so you can scope the window instead of losing it. The trust decision over that window is yours, operational and forensic, bounded by your anchors. If your compliance regime needs a machine-readable compromise marker, raise it with us.

The chain stays as written, including after an incident. A forged-then-detected span stays in the chain as evidence; remediation is new records and your incident report, not history edits.

Reduce the next window

• Enable external anchoring to a bucket in a separately-administered account; it is the single control that bounds this entire runbook. The sweep interval (VAULT_ANCHOR_INTERVAL_MINUTES, default six hours) is your tamper-exposure window.
• Keep VAULT_SIGNING_KEY in a real secret manager, not a flat .env on disk, or hold the key in AWS KMS (VAULT_SIGNING_KEY_KMS_ARN; signing.kmsKeyArn on Helm), where the private half never leaves KMS and CloudTrail records every signature. KMS signs ES256 only, so it takes AGLEDGER_ALLOW_NON_DEFAULT_SIGNING_ALG=true and verifiers upgraded first.
• Keep the pins of your keys outside the database before you need them. An auditor's pin, and your own VAULT_TRUST_ANCHORS recovery, both depend on a copy no database writer can change.
• Alert on agledger_vault_anchor_upload_total{outcome="error"} so the anchor stream never silently stops.