Audit & Verification

A Server exposes two separate audit surfaces. They answer different questions, and an auditor treats them differently. Read this page as two runbooks under one cover.

| Surface | What it holds | Who reads it | Question it answers | |---|---|---|---| | system_audit_log | Operational events - orgs created, records written, keys rotated, admin reads | Your SIEM / SOC, continuously | "What is happening on this Server?" | | audit_vault | The signed, hash-chained records themselves | An auditor, offline | "Is this record authentic and unaltered?" |

The first surface is operational telemetry: it tells you the Server is behaving. The second is the proof: it stands on its own cryptography, so an auditor can verify the chain without trusting - or even reaching - the Server that produced it. Stream the first into your SIEM. Hand the second to an auditor. Do not substitute one for the other.

Surface 1 - Stream operational events to your SIEM

Poll GET /v1/siem/stream on an interval and forward the result to your SIEM. The endpoint merges the event stream and the operational system_audit_log into one time-ordered feed. It requires a key with the audit:read scope (see the API reference for scope details).

Parameters: since (ISO-8601, required), limit (default 100), format (ocsf default, or raw).

curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" \
  "$AGLEDGER_API_URL/v1/siem/stream?since=2026-05-01T00:00:00Z&limit=5&format=raw"

{"type":"admin.org_bootstrapped","payload":{"name":"Default","reason":"single-org-install","actorId":"00000000-0000-0000-0000-000000000000","actorRole":"platform","targetType":"org","targetId":"019ead17-fbd4-7381-ac3f-5ee1474830f1"},"timestamp":"2026-06-09T15:54:50.707Z","id":"019ead17-fbd4-7d84-8941-ff140a540fc7"}
{"type":"schema.registered","payload":{"type":"notarize-generic-v1","orgId":"019ead17-fbd4-7381-ac3f-5ee1474830f1","version":1,"category":"general","publisher":"local","compatibilityMode":"backward","fieldMappingCount":0,"actorId":"00000000-0000-0000-0000-000000000000","actorRole":"platform","targetType":"schema_subject","targetId":"019ead17-fbf0-72b2-8817-44467a2ede3d"},"timestamp":"2026-06-09T15:54:50.734Z","id":"019ead17-fc09-7dcd-b152-b238559c6853"}

Set format=ocsf to emit OCSF 1.4.0 events your SIEM maps natively (Splunk, Elastic, Sentinel):

curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" \
  "$AGLEDGER_API_URL/v1/siem/stream?since=2026-05-01T00:00:00Z&limit=1&format=ocsf"

{"metadata":{"product":{"name":"AGLedger","vendor_name":"AGLedger","version":"1.3.3"},"version":"1.4.0","log_name":"audit","uid":"da5289ae-0c0f-48c8-a686-fcad6d701758"},"time":"2026-06-09T15:54:50.707Z","severity_id":1,"class_uid":3004,"category_uid":3,"type_uid":300403,"activity_id":3,"status_id":1,"message":"admin.org_bootstrapped","actor":{"user":{"uid":"00000000-0000-0000-0000-000000000000","type":"platform"}},"entity":{"uid":"019ead17-fbd4-7381-ac3f-5ee1474830f1","type":"org","data":{"name":"Default","reason":"single-org-install"}}}

To run a continuous poller, advance since to the time of the last event you forwarded on each pass and keep limit modest. The feed is operational telemetry - it is not the tamper-evident proof. A record that appears here has not been independently verified by appearing here; that is the job of Surface 2.

Surface 2 - Verify the chain offline (the real audit)

The audit of record is performed off the Server, against only the published public keys. The verifier has no database, no network, and no AGLedger engine in its dependency tree - so it remains trustworthy even if the Server that produced the chain is later compromised. This is the default posture for a serious audit, and it is fully air-gapped.

Three steps: publish the keys, produce a dump, verify it.

Step 1 - Publish the verification keys

The public signing keys are served unauthenticated and always on. An auditor needs only these.

curl -s "$AGLEDGER_API_URL/v1/verification-keys"

{"data":[{"keyId":"c4dd3e20388b594d","algorithm":"Ed25519","publicKey":"MCowBQYDK2VwAyEAo95XH8DQ6ZYqhC761LqlCq0b9wxYgHPyHs67OkQ9Frw=","publicKeyRaw":"o95XH8DQ6ZYqhC761LqlCq0b9wxYgHPyHs67OkQ9Frw=","status":"active","activatedAt":"2026-06-09T17:22:41.108Z","retiredAt":null}],"envelope":"COSE_Sign1","payloadFormat":"application/vnd.in-toto+cbor","canonicalization":"RFC8949-CDE","coseAlgorithm":-8,"signatureAlgorithm":"Ed25519"}

The same key set is also published at GET /.well-known/agledger-vault-keys.json. Retired keys stay in the set with the exact instants they were active (activatedAt / retiredAt), so records signed before a rotation still verify against the key that actually signed them - the key registry is part of the dump in Step 2, so the auditor never has to ask which key signed what. Each key also carries its algorithm, which is what lets a chain whose history spans a rotation between algorithms verify entry by entry.

Step 2 - Produce a dump

scripts/vault-dump.sh - from the agledger-ai/install repository - is the one component that touches Postgres. It runs the dump tool that already ships inside the Server image (no source checkout, Node.js, or pnpm on the host), writing a self-contained set of NDJSON files the verifier consumes. Run it against a live install, then hand the directory to the auditor.

./scripts/vault-dump.sh ./dump

{
  "outDir": "/dump",
  "orgId": null,
  "counts": {
    "audit_vault": 9,
    "vault_checkpoints": 0,
    "vault_signing_keys": 1,
    "org_admin_reads": 0,
    "org_admin_reads_checkpoints": 0
  }
}

(For a Helm / non-Compose install, run the same shipped tool directly and copy the directory out: kubectl exec deploy/agledger -- /nodejs/bin/node dist/scripts/dump-vault.js /tmp/dump, then kubectl cp <pod>:/tmp/dump ./dump.)

Pass --org <id> to scope the dump to a single org. The output directory holds five files:

audit_vault.ndjson                 the per-record hash chains
vault_checkpoints.ndjson           periodic signed checkpoints over the chains
vault_signing_keys.ndjson          the public-key registry, with rotation history
org_admin_reads.ndjson             the cross-party admin-read log
org_admin_reads_checkpoints.ndjson signed tree heads over the read log

The dump is database-independent. Keep a copy alongside your database backup - it is the artifact an auditor verifies, and it does not depend on a live Server to be meaningful. (See the backup runbook for where this fits in a backup schedule.)

As of v1.3.4, vault_checkpoints is append-only at the database, matching the guard audit_vault already carried: a DELETE or UPDATE against a checkpoint row is refused outright by a trigger. Checkpoints are the out-of-band high-water mark that makes terminal truncation detectable at all - a chain truncated from the end still hash-links cleanly - so the witness now carries the same immutability as the ledger it protects. If you have automation that prunes or rewrites checkpoint rows, it will start failing against the guard; that is the intended effect. Each checkpoint also names which chain it anchors (record, schema, or admin). That keying is committed inside the signed checkpoint payload itself, so it cannot be changed without breaking the checkpoint's signature.

Step 3 - Verify with stock libraries

The verification needs no AGLedger software. Each row of audit_vault.ndjson carries its canonical cose_sign1 envelope (RFC 9052 COSE_Sign1 over an in-toto Statement, signed Ed25519); vault_signing_keys.ndjson carries the public-key registry. Decode each envelope with any stock COSE library - go-cose, coset (Rust), or pycose - and verify its Ed25519 signature against the key resolved by signing_key_id. The Sig_structure is constructed per RFC 9052 §4.4; the signatureInputTemplate field at /v1/verification-keys documents it exactly.

This example uses Python with cbor2 and cryptography - neither of them ours - to walk the whole dump. Save it as verify-dump.py:

import json, base64, sys, cbor2
from cryptography.hazmat.primitives.serialization import load_der_public_key
from cryptography.exceptions import InvalidSignature

keys = {k["key_id"]: k["public_key"]
        for k in map(json.loads, open(sys.argv[1] + "/vault_signing_keys.ndjson"))}

ok = fail = 0
for row in map(json.loads, open(sys.argv[1] + "/audit_vault.ndjson")):
    cose = base64.b64decode(row["cose_sign1"])
    protected, _unprotected, payload, signature = cbor2.loads(cose).value
    pub = load_der_public_key(base64.b64decode(keys[row["signing_key_id"]]))
    sig_structure = cbor2.dumps(["Signature1", protected, b"", payload])  # RFC 9052 §4.4
    try:
        pub.verify(signature, sig_structure); ok += 1
    except InvalidSignature:
        fail += 1; print("FAIL pos", row["chain_position"], row["record_id"])

print(f"[{'PASS' if not fail else 'FAIL'}] stock-library offline verification")
print(f"  audit_vault entries : {ok + fail}")
print(f"  signatures verified : {ok}")
print(f"  failures            : {fail}")
print(f"  signing keys        : {len(keys)}")
sys.exit(1 if fail else 0)

python3 verify-dump.py ./dump

[PASS] stock-library offline verification
  audit_vault entries : 9
  signatures verified : 9
  failures            : 0
  signing keys        : 1

The audit_vault row count includes the schema-registration chain alongside your record chains. The script exits non-zero on any signature failure, so it drops straight into a CI gate. A few stricter COSE libraries refuse AGLedger's vendor-private header labels by default - for pycose, decode with Sign1Message.from_cose_obj(..., allow_unknown_attributes=True); go-cose and coset accept them as-is. The full library-quirk notes live under "Offline cryptographic verification" in GET /llms-full.txt.

A packaged zero-dependency verifier, @agledger/verify, reproduces this end to end (chains plus SCITT checkpoints) as a single command, and the @agledger/cli verify subcommand consumes the same dump format. Both are published on npm. The stock-library path above remains the dependency-free way to verify a chain when you do not want to install our tooling.

When verification fails

A failure is the verifier doing its job. Tamper with one byte of a signed envelope and the stock-library script above rejects it at that position:

FAIL pos 1 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7
[FAIL] stock-library offline verification
  audit_vault entries : 9
  signatures verified : 8
  failures            : 1

A signature mismatch like that is one of a small set of integrity classes a full verifier reports. The stock-library check above proves the signatures (catching CHAIN_SIGNATURE_INVALID and CHAIN_SIGNATURE_MISSING_KEY); the in-database scripts/vault-verify.sh and the packaged @agledger/verify add the hash, link, and position classes. The complete set on the per-record chain:

| Code | Means | |---|---| | CHAIN_GENESIS_INVALID | The first entry of a chain does not link to genesis | | CHAIN_POSITION_GAP | A chain position is missing - an entry was removed | | CHAIN_LINK_BROKEN | An entry's previous_hash does not match the prior entry | | CHAIN_HASH_MISMATCH | The stored hash does not match sha256(cose_sign1) | | CHAIN_SIGNATURE_INVALID | The signature does not verify against its resolved key | | CHAIN_SIGNATURE_MISSING_KEY | The signing key is not in the published registry | | CHAIN_COSE_DECODE_FAILED | The signed envelope is not decodable | | CHAIN_COSE_HEADER_MISMATCH | Chain mechanics in the protected header disagree with the row | | CHAIN_PAYLOAD_BINDING_MISMATCH | The denormalized row diverges from the signed payload | | CHAIN_OIDC_ACTOR_MISMATCH | The recorded actor identity disagrees with the signed claim | | CHAIN_SIGNING_KEY_DRIFT | The row's key column names a different key than the signature-covered kid | | CHAIN_ALG_MISMATCH | The signed header's algorithm is not one the trusted key can produce | | CHAIN_UNSUPPORTED_ALGORITHM | The key's algorithm is beyond this verifier build; upgrade, never a pass | | CHAIN_KEY_EXPIRED | The entry was signed outside the key's activation..retirement window | | CHAIN_KEY_POLICY_VIOLATION | The entry violates a caller-set key policy (required key id, out-of-band keys) |

CHAIN_UNSUPPORTED_ALGORITHM is a capability gap, not a tamper signal. It means the verifier cannot compute the algorithm the key names, so it fails closed rather than passing something it did not check. The remedy is a newer verifier, or verifying on a host whose crypto provider carries the algorithm - a FIPS-mode host reports it for every Ed25519 entry (see FIPS 140 hosts).

Checkpoint and admin-read chains report their own classes (CHECKPOINT_*, TENANT_READ_*, TENANT_CHECKPOINT_*) on the same model.

Note what does not fail verification: editing the convenience JSON in audit_vault.ndjson without touching the signed envelope. The verifier trusts the signed cose_sign1 artifact as the source of truth, not the denormalized columns - so a privileged-database edit of the readable payload is caught as CHAIN_PAYLOAD_BINDING_MISMATCH, not silently accepted.

On-box reads versus the off-box handoff

An operator can read the chain on the Server through the admin vault endpoints (see the API reference for /v1/admin/vault/*). Every cross-party admin read is itself notarized into the org_admin_reads chain - reading the chain is an accountable act, and it appears in the dump above.

An auditor does the opposite: they take the dump off the Server and verify it on their own machine with only the public keys. On-box reads are for operations. The off-box handoff is the audit. Keep the two roles distinct.

Optional - the SCITT transparency checkpoint

If you operate the SCITT transparency surface (POST /v1/scitt/entries), GET /v1/scitt/checkpoint returns the current signed tree head for an org's transparency log - a stable anchor an external monitor can pin and re-check for consistency over time. The checkpoint is org-scoped: use an org-scoped key (a platform-scope key is refused with 403 ORG_REQUIRED).

curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" "$AGLEDGER_API_URL/v1/scitt/checkpoint"

{"treeSize":0,"rootHex":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","logId":"019ead17-fbd4-7381-ac3f-5ee1474830f1","iat":1781020588,"kid":"c4dd3e20388b594d","signature":"a130648224123432149bf933086a9a25cab63ee7f8a8e6789e52056a5ae7f4f5f1b1447dc3d3ae8e81bdb0d2ddd31f90a93a3d35c83a851d51011ea82502c90f"}

A treeSize of 0 with the empty-tree root above is a fresh log with no SCITT entries registered yet. This surface is independent of the audit_vault chain in Surface 2: the offline verifier is the proof of the record chain; the SCITT checkpoint is the anchor for the separate SCRAPI transparency log.

Updated for API v1.4.0 on 2026-08-09: the per-record failure table gains the five key-and-algorithm classes the verifier now reports (CHAIN_SIGNING_KEY_DRIFT, CHAIN_ALG_MISMATCH, CHAIN_UNSUPPORTED_ALGORITHM, CHAIN_KEY_EXPIRED, CHAIN_KEY_POLICY_VIOLATION), and the key registry now publishes an algorithm per key alongside full-precision activation windows, which is what lets a chain spanning a rotation between algorithms verify entry by entry. The dump and offline-verification flow is unchanged. Not re-run end to end against a live v1.4.0 install.

Validated against API v1.3.4 on 2026-08-03 (Developer Edition, Docker Compose, fresh install from the pinned v1.3.4 release). Re-run live: /v1/verification-keys including signatureInputTemplate, scripts/vault-dump.sh, and the verify-dump.py listing above verbatim, verifying every entry. The append-only claim was exercised directly against the database: a DELETE and an UPDATE against a checkpoint row were both refused by the trigger, TRUNCATE was refused by its own guard, and the documented agledger.allow_audit_drop escape hatch permitted the delete.

*Validated against API v1.1.0 on 2026-06-10 (Developer Edition, Docker Compose: scripts/vault-dump.sh

Reviewed for API v1.3.4 on 2026-08-01: migration 003 makes vault_checkpoints append-only at the database - a DELETE or UPDATE against a checkpoint row is refused by a trigger, matching the guard audit_vault already carried - and each checkpoint names the chain it anchors (record, schema, or admin), a keying committed inside the signed checkpoint payload. Both are covered under Step 2 above. The offline vault-dump and verification flow is unchanged.

Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 hardens audit-vault chain-scan detection (it now flags a schema-registration or platform-ops chain truncated to empty even with no surviving checkpoint) and adds opt-in verdict per-actor signatures. Both are backward-compatible and the offline vault-dump plus verification flow above is unchanged.

Reviewed for API v1.3.2 on 2026-07-13: the offline vault-dump and verification flow above is unchanged. v1.3.2 adds per-record GET /v1/records/{id}/audit-export?evidence=true, which inlines each completion's evidence body at its COMPLETION_SUBMITTED entry and documents the evidenceHash binding (SHA-256 over the RFC 8785 (JCS) canonicalization of the evidence JSON) in the export's verificationGuide.evidenceBinding, so an offline auditor can re-bind a separately-held evidence body to the signed chain without a live fetch. The response shape and re-binding recipe are in the API reference; testbed to capture a transcript for a worked example here.