> Markdown version of https://agledger.ai/docs/operations/anchoring/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # External anchoring External anchoring periodically writes each record chain's signed checkpoint to S3-compatible object storage, by default under **Object Lock in COMPLIANCE mode**. Once written, an anchor is immutable for its retention period - neither an attacker nor an administrator of the AGLedger Server can alter or delete it. This is the layer that makes tamper-evidence hold against the operator of the system itself, not only against an outside attacker. ## Why anchoring matters Every chain entry is Ed25519-signed and hash-linked, so any modification to stored history is detectable on verification - provided the verifier trusts the signing key. An administrator who holds both the database and the active signing key can, in principle, rewrite a chain and re-sign it; the rewritten chain would verify cleanly. Anchoring closes this gap: - **Entries covered by an anchor** cannot be rewritten without exposure: the rewritten chain no longer matches the immutable checkpoint in the bucket. - **Entries written since the last anchor sweep** rely on the signature and hash chain alone. The span between sweeps is your **tamper-exposure window** - that is what `VAULT_ANCHOR_INTERVAL_MINUTES` sets. Anchoring detects tampering after the fact; it does not prevent it in real time. Without it, all tamper-evidence layers live in the same PostgreSQL database a privileged attacker would rewrite - which is why a production Server logs a boot warning while anchoring is disabled. ## What gets anchored Checkpoints are **per record chain**, swept incrementally on the anchor interval: each sweep checkpoints chains that advanced since their last checkpoint, up to 100 per sweep (the remainder rolls to the next sweep). Each anchor object is a small JSON document (about a kilobyte) carrying the checkpoint's canonical COSE_Sign1 envelope (base64), the record ID, chain position, payload hash, signing key ID, the instance ID, a document version, and the anchor timestamp. Record payloads never leave your database - the anchor is a cryptographic commitment, not a copy. Objects land at a deterministic key inside the configured bucket: ``` vault-anchors////.json ``` where `` is `AGLEDGER_INSTANCE_ID` and the date is the checkpoint's creation date. The installer generates a UUID for that variable and writes it to your `.env`, so on a scripted install the segment is that UUID rather than a friendly name. Read it back from `.env` (or from `GET /v1/admin/ops-summary`) before going looking for objects by prefix. Because checkpoints are per-chain, the object count scales with how many chains advance per interval, not with a fixed per-tick rate. A low-volume install produces a handful of objects per sweep; a high-volume install produces up to the 100-chain sweep cap per sweep. Checkpoints are created even when anchoring is disabled - they are the in-database layer of the same control. Enabling anchoring adds the external, immutable copy. ## Enable it (Docker Compose) Anchoring is **disabled by default**. In your `.env`: ```bash VAULT_ANCHOR_ENABLED=true VAULT_ANCHOR_S3_BUCKET=my-org-agledger-anchors VAULT_ANCHOR_S3_REGION=us-east-1 # VAULT_ANCHOR_S3_ENDPOINT=http://minio:9000 # non-AWS S3-compatible stores (MinIO, etc.) VAULT_ANCHOR_INTERVAL_MINUTES=360 # tamper-exposure window; see below VAULT_ANCHOR_OBJECT_LOCK=true # COMPLIANCE-mode immutable retention (default) VAULT_ANCHOR_RETENTION_DAYS=2555 # ~7 years (default) ``` Credentials come from the standard AWS credential chain (instance role, `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, or a profile). Two processes touch the bucket: - The **worker** runs the anchor sweep and the daily sample-verify job: it needs `s3:PutObject` (plus `s3:PutObjectRetention` when Object Lock is on) and `s3:GetObject`. - The **API** serves the admin anchor endpoints: it needs `s3:GetObject` and `s3:ListBucket`. Grant only `s3:PutObject` to the worker and every verify surface in this runbook returns `s3_error`. Defaults if unset: bucket `vault-anchors`, region from `AWS_REGION` (falling back to `us-east-1`), Object Lock on, retention 2555 days, interval 360 minutes. ## Enable it (Kubernetes / Helm) The chart passes these through `extraEnv` (there is no dedicated `vault.anchor` values block): ```yaml extraEnv: - name: VAULT_ANCHOR_ENABLED value: "true" - name: VAULT_ANCHOR_S3_BUCKET value: my-org-agledger-anchors - name: VAULT_ANCHOR_S3_REGION value: us-east-1 ``` On EKS, prefer an IRSA role on the worker's service account over static credentials. ## Prepare the bucket Object Lock must be enabled **at bucket creation** - it cannot be added to an existing bucket: ```bash aws s3api create-bucket \ --bucket my-org-agledger-anchors \ --region us-east-1 \ --object-lock-enabled-for-bucket ``` AGLedger applies per-object COMPLIANCE-mode retention of `VAULT_ANCHOR_RETENTION_DAYS` as each anchor is written. In COMPLIANCE mode no one - including the account root - can delete the object or shorten its retention until it expires. That permanence is the point; choose the retention deliberately. For the strongest posture, put the bucket in a **separately-administered account** (or a different provider), so no single set of credentials controls both the database and the anchors. A bucket in the same account as the Server is a single trust domain. Air-gapped installs: point `VAULT_ANCHOR_S3_ENDPOINT` at MinIO (path-style addressing is handled automatically) and enable MinIO's object locking on the bucket. ## Choose the interval `VAULT_ANCHOR_INTERVAL_MINUTES` (default **360**, clamped to **5–2880**, snapped to cron granularity) is a risk-tolerance dial, not a performance knob. It bounds how much recent history has no external evidence yet: - **Tighten** (15–60 min) for high-value, irreversible workloads where hours of unpinned records is too much exposure. - **Default 360 (6 h)** is a reasonable posture for most deployments. - **Loosen** (up to 2880 = 48 h) where object economy matters more than a tight window. The sweep is incremental, so short intervals stay cheap on the database; the cost of tightening is the permanent Object Lock object count. ## Verify and monitor Posture at a glance, under `vault.anchoring` in `GET /v1/admin/ops-summary`: ```json {"enabled": true, "intervalMinutes": 5, "bucket": "my-org-agledger-anchors", "workerEnabled": true, "reconciled": true} ``` List a record's anchors (`recordId` is required): ```bash curl -H "Authorization: Bearer $ADMIN_KEY" \ "$AGLEDGER_API_URL/v1/admin/vault/anchors?recordId=" ``` ```json {"data":[{"key":"vault-anchors/b736ccb2-.../2026-08-10/019fe911-.../1.json", "lastModified":"2026-08-10T00:30:18.234Z","size":1043}], "total":1} ``` Verify anchors against the database (exposes any divergence between the live chain and the pinned state): ```bash curl -X POST -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \ -d '{"recordId": ""}' \ "$AGLEDGER_API_URL/v1/admin/vault/anchors/verify" ``` ```json {"data":[{"recordId":"019fe911-...","chainPosition":1,"match":true, "detail":"Anchor matches database checkpoint"}]} ``` A daily background job also verifies a random sample of 20 checkpoints against their anchors (checkpoints not yet anchored are skipped, so a run may verify fewer than 20). Two Prometheus counters back alerting, and they are exposed by **different processes**, so scrape both: - `agledger_vault_anchor_upload_total{outcome}`, on the **worker**, which runs the sweep. Alert on `outcome="error"`: a misconfigured bucket or IAM regression silently widens the tamper-exposure window. A healthy sweep increments `outcome="success"`. - `agledger_vault_anchor_verify_total{outcome}`, on the **API**, which serves the verify endpoint. Alert on `outcome="tamper"` (and investigate `no_such_key`). A matching verify increments `outcome="verified"`. Scraping only the API leaves the upload counter unread, and that is the one that tells you anchors stopped landing. Uploads are deliberately non-blocking: an S3 outage never stops notarization. The counter is the signal that anchors are not landing. With anchoring disabled, the verify endpoint still answers rather than erroring: it returns `match: false` with `detail: "Vault anchoring is not enabled"`. Read that as configuration, not as a tamper finding. In-database checkpoints (anchored or not) are readable at `GET /v1/audit-vault/checkpoints`. Full request/response shapes are in the [API reference](/api/). ## Honest limitations - **Detect, not prevent.** An operator with the signing key can still rewrite within the current window; the next sweep exposes everything already pinned. - **One bucket = one trust domain.** Separate the account administering the anchors from the one running the Server. - **COMPLIANCE retention is irreversible.** You cannot shorten it after the fact. ## Related - [Key-compromise runbook](/docs/operations/key-compromise/) - anchors are what distinguish pre-compromise history from a forged rewrite. - [Audit runbook](/docs/operations/audit/) - offline chain verification with published keys. - [Day-2 operations](/docs/operations/day-2/) - ops-summary and alerting.