> Markdown version of https://agledger.ai/docs/operations/audit/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # Audit & Verification A Server exposes two separate audit surfaces. They answer different questions, and an auditor treats them differently. Read this page as two runbooks under one cover. | Surface | What it holds | Who reads it | Question it answers | |---|---|---|---| | `system_audit_log` | Operational events - orgs created, records written, keys rotated, admin reads | Your SIEM / SOC, continuously | "What is happening on this Server?" | | `audit_vault` | The signed, hash-chained records themselves | An auditor, offline | "Is this record authentic and unaltered?" | The first surface is operational telemetry: it tells you the Server is behaving. The second is the proof: it stands on its own cryptography, so an auditor can verify the chain without trusting - or even reaching - the Server that produced it. Stream the first into your SIEM. Hand the second to an auditor. Do not substitute one for the other. ## Surface 1 - Stream operational events to your SIEM Poll `GET /v1/siem/stream` on an interval and forward the result to your SIEM. The endpoint merges the event stream and the operational `system_audit_log` into one time-ordered feed. It requires a key with the `audit:read` scope (see the [API reference](/api/) for scope details). Parameters: `since` (ISO-8601, required), `limit` (default 100), `format` (`ocsf` default, or `raw`). ```bash curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" \ "$AGLEDGER_API_URL/v1/siem/stream?since=2026-05-01T00:00:00Z&limit=5&format=raw" ``` ```json {"type":"admin.org_bootstrapped","payload":{"name":"Default","reason":"single-org-install","actorId":"00000000-0000-0000-0000-000000000000","actorRole":"platform","targetType":"org","targetId":"019ead17-fbd4-7381-ac3f-5ee1474830f1"},"timestamp":"2026-06-09T15:54:50.707Z","id":"019ead17-fbd4-7d84-8941-ff140a540fc7"} {"type":"schema.registered","payload":{"type":"notarize-generic-v1","orgId":"019ead17-fbd4-7381-ac3f-5ee1474830f1","version":1,"category":"general","publisher":"local","compatibilityMode":"backward","fieldMappingCount":0,"actorId":"00000000-0000-0000-0000-000000000000","actorRole":"platform","targetType":"schema_subject","targetId":"019ead17-fbf0-72b2-8817-44467a2ede3d"},"timestamp":"2026-06-09T15:54:50.734Z","id":"019ead17-fc09-7dcd-b152-b238559c6853"} ``` Set `format=ocsf` to emit OCSF 1.4.0 events your SIEM maps natively (Splunk, Elastic, Sentinel): ```bash curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" \ "$AGLEDGER_API_URL/v1/siem/stream?since=2026-05-01T00:00:00Z&limit=1&format=ocsf" ``` ```json {"metadata":{"product":{"name":"AGLedger","vendor_name":"AGLedger","version":"1.3.3"},"version":"1.4.0","log_name":"audit","uid":"da5289ae-0c0f-48c8-a686-fcad6d701758"},"time":"2026-06-09T15:54:50.707Z","severity_id":1,"class_uid":3004,"category_uid":3,"type_uid":300403,"activity_id":3,"status_id":1,"message":"admin.org_bootstrapped","actor":{"user":{"uid":"00000000-0000-0000-0000-000000000000","type":"platform"}},"entity":{"uid":"019ead17-fbd4-7381-ac3f-5ee1474830f1","type":"org","data":{"name":"Default","reason":"single-org-install"}}} ``` To run a continuous poller, advance `since` to the `time` of the last event you forwarded on each pass and keep `limit` modest. The feed is operational telemetry - it is not the tamper-evident proof. A record that appears here has not been independently verified by appearing here; that is the job of Surface 2. ## Surface 2 - Verify the chain offline (the real audit) The audit of record is performed off the Server, against only the published public keys. The verifier has no database, no network, and no AGLedger engine in its dependency tree - so it remains trustworthy even if the Server that produced the chain is later compromised. This is the default posture for a serious audit, and it is fully air-gapped. Three steps: publish the keys, produce a dump, verify it. ### Step 1 - Publish the verification keys The public signing keys are served unauthenticated and always on. An auditor needs only these. ```bash curl -s "$AGLEDGER_API_URL/v1/verification-keys" ``` ```json {"data":[{"keyId":"c4dd3e20388b594d","algorithm":"Ed25519","publicKey":"MCowBQYDK2VwAyEAo95XH8DQ6ZYqhC761LqlCq0b9wxYgHPyHs67OkQ9Frw=","publicKeyRaw":"o95XH8DQ6ZYqhC761LqlCq0b9wxYgHPyHs67OkQ9Frw=","status":"active","activatedAt":"2026-06-09T17:22:41.108Z","retiredAt":null}],"envelope":"COSE_Sign1","payloadFormat":"application/vnd.in-toto+cbor","canonicalization":"RFC8949-CDE","coseAlgorithm":-8,"signatureAlgorithm":"Ed25519"} ``` The same key set is also published at `GET /.well-known/agledger-vault-keys.json`. Retired keys stay in the set with the exact instants they were active (`activatedAt` / `retiredAt`), so records signed before a rotation still verify against the key that actually signed them - the key registry is part of the dump in Step 2, so the auditor never has to ask which key signed what. Each key also carries its `algorithm`, which is what lets a chain whose history spans a rotation between algorithms verify entry by entry. ### Step 2 - Produce a dump `scripts/vault-dump.sh` - from the [`agledger-ai/install`](https://github.com/agledger-ai/install) repository - is the one component that touches Postgres. It runs the dump tool that already ships inside the Server image (no source checkout, Node.js, or pnpm on the host), writing a self-contained set of NDJSON files the verifier consumes. Run it against a live install, then hand the directory to the auditor. ```bash ./scripts/vault-dump.sh ./dump ``` ```json { "outDir": "/dump", "orgId": null, "counts": { "audit_vault": 9, "vault_checkpoints": 0, "vault_signing_keys": 1, "org_admin_reads": 0, "org_admin_reads_checkpoints": 0 } } ``` (For a Helm / non-Compose install, run the same shipped tool directly and copy the directory out: `kubectl exec deploy/agledger -- /nodejs/bin/node dist/scripts/dump-vault.js /tmp/dump`, then `kubectl cp :/tmp/dump ./dump`.) Pass `--org ` to scope the dump to a single org. The output directory holds five files: ``` audit_vault.ndjson the per-record hash chains vault_checkpoints.ndjson periodic signed checkpoints over the chains vault_signing_keys.ndjson the public-key registry, with rotation history org_admin_reads.ndjson the cross-party admin-read log org_admin_reads_checkpoints.ndjson signed tree heads over the read log ``` The dump is database-independent. Keep a copy alongside your database backup - it is the artifact an auditor verifies, and it does not depend on a live Server to be meaningful. (See the [backup runbook](/docs/operations/backup/) for where this fits in a backup schedule.) As of v1.3.4, `vault_checkpoints` is append-only at the database, matching the guard `audit_vault` already carried: a DELETE or UPDATE against a checkpoint row is refused outright by a trigger. Checkpoints are the out-of-band high-water mark that makes terminal truncation detectable at all - a chain truncated from the end still hash-links cleanly - so the witness now carries the same immutability as the ledger it protects. If you have automation that prunes or rewrites checkpoint rows, it will start failing against the guard; that is the intended effect. A deliberate archival delete is still possible by setting `agledger.allow_audit_drop = on` for the session; without that flag the trigger refuses the statement outright. Each checkpoint also names which chain it anchors (`record`, `schema`, or `admin`). That keying is committed inside the signed checkpoint payload itself, so it cannot be changed without breaking the checkpoint's signature. ### Step 3 - Verify with stock libraries The verification needs no AGLedger software. Each row of `audit_vault.ndjson` carries its canonical `cose_sign1` envelope (RFC 9052 COSE_Sign1 over an in-toto Statement, signed Ed25519); `vault_signing_keys.ndjson` carries the public-key registry. Decode each envelope with any stock COSE library - `go-cose`, `coset` (Rust), or `pycose` - and verify its Ed25519 signature against the key resolved by `signing_key_id`. The `Sig_structure` is constructed per RFC 9052 §4.4; the `signatureInputTemplate` field at `/v1/verification-keys` documents it exactly. This example uses Python with `cbor2` and `cryptography` - neither of them ours - to walk the whole dump. Save it as `verify-dump.py`: ```python import json, base64, sys, cbor2 from cryptography.hazmat.primitives.serialization import load_der_public_key from cryptography.exceptions import InvalidSignature keys = {k["key_id"]: k["public_key"] for k in map(json.loads, open(sys.argv[1] + "/vault_signing_keys.ndjson"))} ok = fail = 0 for row in map(json.loads, open(sys.argv[1] + "/audit_vault.ndjson")): cose = base64.b64decode(row["cose_sign1"]) protected, _unprotected, payload, signature = cbor2.loads(cose).value pub = load_der_public_key(base64.b64decode(keys[row["signing_key_id"]])) sig_structure = cbor2.dumps(["Signature1", protected, b"", payload]) # RFC 9052 §4.4 try: pub.verify(signature, sig_structure); ok += 1 except InvalidSignature: fail += 1; print("FAIL pos", row["chain_position"], row["record_id"]) print(f"[{'PASS' if not fail else 'FAIL'}] stock-library offline verification") print(f" audit_vault entries : {ok + fail}") print(f" signatures verified : {ok}") print(f" failures : {fail}") print(f" signing keys : {len(keys)}") sys.exit(1 if fail else 0) ``` ```bash python3 verify-dump.py ./dump ``` ``` [PASS] stock-library offline verification audit_vault entries : 9 signatures verified : 9 failures : 0 signing keys : 1 ``` The `audit_vault` row count includes the schema-registration chain alongside your record chains. The script exits non-zero on any signature failure, so it drops straight into a CI gate. A few stricter COSE libraries refuse AGLedger's vendor-private header labels by default - for `pycose`, decode with `Sign1Message.from_cose_obj(..., allow_unknown_attributes=True)`; `go-cose` and `coset` accept them as-is. The full library-quirk notes live under "Offline cryptographic verification" in `GET /llms-full.txt`. > A packaged zero-dependency verifier, `@agledger/verify`, reproduces this end to end (chains plus > SCITT checkpoints) as a single command, and the `@agledger/cli` `verify` subcommand consumes the > same dump format. Both are published on npm. The stock-library path above remains the > dependency-free way to verify a chain when you do not want to install our tooling. ### When verification fails A failure is the verifier doing its job. Tamper with one byte of a signed envelope and the stock-library script above rejects it at that position: ``` FAIL pos 1 019ead18-c3f9-7b4d-8edf-2dcd8b99fbc7 [FAIL] stock-library offline verification audit_vault entries : 9 signatures verified : 8 failures : 1 ``` A signature mismatch like that is one of a small set of integrity classes a full verifier reports. The stock-library check above proves the signatures (catching `CHAIN_SIGNATURE_INVALID` and `CHAIN_SIGNATURE_MISSING_KEY`); the in-database `scripts/vault-verify.sh` and the packaged `@agledger/verify` add the hash, link, and position classes. The complete set on the per-record chain: | Code | Means | |---|---| | `CHAIN_GENESIS_INVALID` | The first entry of a chain does not link to genesis | | `CHAIN_POSITION_GAP` | A chain position is missing - an entry was removed | | `CHAIN_LINK_BROKEN` | An entry's `previous_hash` does not match the prior entry | | `CHAIN_HASH_MISMATCH` | The stored hash does not match `sha256(cose_sign1)` | | `CHAIN_SIGNATURE_INVALID` | The signature does not verify against its resolved key | | `CHAIN_SIGNATURE_MISSING_KEY` | The signing key is not in the published registry | | `CHAIN_COSE_DECODE_FAILED` | The signed envelope is not decodable | | `CHAIN_COSE_HEADER_MISMATCH` | Chain mechanics in the protected header disagree with the row | | `CHAIN_PAYLOAD_BINDING_MISMATCH` | The denormalized row diverges from the signed payload | | `CHAIN_OIDC_ACTOR_MISMATCH` | The recorded actor identity disagrees with the signed claim | | `CHAIN_SIGNING_KEY_DRIFT` | The row's key column names a different key than the signature-covered kid | | `CHAIN_ALG_MISMATCH` | The signed header's algorithm is not one the trusted key can produce | | `CHAIN_UNSUPPORTED_ALGORITHM` | The key's algorithm is beyond this verifier build; upgrade, never a pass | | `CHAIN_KEY_EXPIRED` | The entry was signed outside the key's activation..retirement window | | `CHAIN_KEY_POLICY_VIOLATION` | The entry violates a caller-set key policy (required key id, out-of-band keys) | `CHAIN_UNSUPPORTED_ALGORITHM` is a capability gap, not a tamper signal. It means the verifier cannot compute the algorithm the key names, so it fails closed rather than passing something it did not check. The remedy is a newer verifier, or verifying on a host whose crypto provider carries the algorithm - a FIPS-mode host reports it for every Ed25519 entry (see [FIPS 140 hosts](/docs/install/fips/)). Checkpoint and admin-read chains report their own classes (`CHECKPOINT_*`, `TENANT_READ_*`, `TENANT_CHECKPOINT_*`) on the same model. Note what does not fail verification: editing the convenience JSON in `audit_vault.ndjson` without touching the signed envelope. The verifier trusts the signed `cose_sign1` artifact as the source of truth, not the denormalized columns - so a privileged-database edit of the readable payload is caught as `CHAIN_PAYLOAD_BINDING_MISMATCH`, not silently accepted. ## On-box reads versus the off-box handoff An operator can read the chain on the Server through the admin vault endpoints (see the [API reference](/api/) for `/v1/admin/vault/*`). Every cross-party admin read is itself notarized into the `org_admin_reads` chain - reading the chain is an accountable act, and it appears in the dump above. An auditor does the opposite: they take the dump off the Server and verify it on their own machine with only the public keys. On-box reads are for operations. The off-box handoff is the audit. Keep the two roles distinct. For a single record, `GET /v1/records/{id}/audit-export?evidence=true` inlines each completion's evidence body at its `COMPLETION_SUBMITTED` entry and documents the `evidenceHash` binding (SHA-256 over the RFC 8785 (JCS) canonicalization of the evidence JSON) in the export's `verificationGuide.evidenceBinding`, so an offline auditor can re-bind a separately-held evidence body to the signed chain without a live fetch. The response shape and re-binding recipe are in the [API reference](/api/). ## The org-reads transparency log Every cross-party read by an org-admin key is notarized into the `org_admin_reads` chain, and that chain gets its own periodic signed tree head. This is the symmetric half of the accountability story: the record chain says what the agents did, and this says who went looking at it. Four endpoints, all org-scoped and readable by an admin or an agent holding `audit:read`: | Endpoint | What it gives you | |---|---| | `GET /v1/audit/org-reads/checkpoints` | The signed tree heads, newest first | | `GET /v1/audit/org-reads/checkpoints/{id}` | One checkpoint envelope by id | | `GET /v1/audit/org-reads/checkpoints/{id}/proof?leaf=N` | The inclusion path for one leaf | | `POST /v1/audit/org-reads/checkpoints/{id}/cosign` | Record an external witness signature | ### An empty list on a fresh install is expected Checkpoints are built by a sweep on the Worker every six hours (`0 */6 * * *`, fixed, not configurable). A fresh install returns ```json {"data":[],"hasMore":false,"nextCursor":null,"total":0} ``` until the first sweep runs, no matter how many reads the org has already taken. The same applies to the `signedCheckpointRef` in a read response: it is null immediately after the read, and the next sweep populates it. Do not read either as a missing or broken feature. As of v1.5.0 there is no endpoint that lists the leaves themselves, so the checkpoints and the per-read `recordRead` block are how you observe this log. ### Which tree this is, and which it is not This log uses AGLedger's own hex-string Merkle tree. **It is not the RFC 9162 tree that SCITT Receipts carry** (`/v1/scitt/*`, VDS `RFC9162_SHA256`). A proof from one does not verify under the other, and the three properties that usually trip an implementer are all the opposite way round here: - Nodes combine over the **hex ASCII text**, not the raw bytes those hex strings denote. - An odd level pairs its **last node with itself**, so the path length is the level count for every leaf rather than RFC 9162's variable-length audit path. - There is **no `0x00`/`0x01` domain separation**. Every node is a lowercase hex SHA-256 string. A leaf is `sha256(cose_sign1_bytes)` of its `org_admin_reads` row. Two nodes combine as: ``` H(L, R) = hex(sha256(utf8(L + R))) ``` Levels repeat until one node remains, and that is `rootHash`. An empty tree (`treeSize` 0) has root `sha256("")`, which is `e3b0c442...b855`. ### Verifying an inclusion proof `path` carries one sibling per level, leaf level first, so its length is `ceil(log2(treeSize))` for every leaf, the first and the last included. The leaf index chooses the side at each step. A self-paired node needs no special case, because its path entry is its own hash: ```python cur, idx, size, i = leafHash, leafIndex, treeSize, 0 while size > 1: cur = H(cur, path[i]) if idx % 2 == 0 else H(path[i], cur) idx, size, i = idx // 2, -(-size // 2), i + 1 assert cur == rootHash and i == len(path) ``` The checkpoint envelope itself is a COSE_Sign1 (RFC 9052) over the tree head, signed by the same vault key registry the record chain uses, so `coseSign1Base64` verifies with the public keys from `GET /v1/verification-keys` exactly as Surface 2 above does. An external monitor that wants to pin a head can post its own signature back with the cosign endpoint; that witness signature is single-use per checkpoint. ## Optional - the SCITT transparency checkpoint If you operate the SCITT transparency surface (`POST /v1/scitt/entries`), `GET /v1/scitt/checkpoint` returns the current signed tree head for an org's transparency log - a stable anchor an external monitor can pin and re-check for consistency over time. The checkpoint is org-scoped: use an org-scoped key (a platform-scope key is refused with `403 ORG_REQUIRED`). ```bash curl -s -H "Authorization: Bearer $AGLEDGER_API_KEY" "$AGLEDGER_API_URL/v1/scitt/checkpoint" ``` ```json {"treeSize":0,"rootHex":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","logId":"019ead17-fbd4-7381-ac3f-5ee1474830f1","iat":1781020588,"kid":"c4dd3e20388b594d","signature":"a130648224123432149bf933086a9a25cab63ee7f8a8e6789e52056a5ae7f4f5f1b1447dc3d3ae8e81bdb0d2ddd31f90a93a3d35c83a851d51011ea82502c90f"} ``` A `treeSize` of `0` with the empty-tree root above is a fresh log with no SCITT entries registered yet. This surface is independent of the `audit_vault` chain in Surface 2: the offline verifier is the proof of the record chain; the SCITT checkpoint is the anchor for the separate SCRAPI transparency log.