External anchoring

External anchoring periodically writes each record chain's signed checkpoint to S3-compatible object storage, under Object Lock in COMPLIANCE mode whenever Object Lock is on (VAULT_ANCHOR_OBJECT_LOCK, on by default). The mode itself is not a setting: every anchor this Server writes with Object Lock enabled is COMPLIANCE, never GOVERNANCE.

Every write is also create-only (If-None-Match): a second write to a position already anchored is refused by the store rather than layered over the first, and is reported as a fork. Taken together, a locked version cannot be altered or deleted, and a store that honours the precondition refuses the second write outright, for the retention period, whether the write comes from an attacker or from an administrator of the AGLedger Server. POST /v1/admin/vault/anchors/reconcile names a store that does not honour conditional writes in its posture report, because on that store the guarantee is not in force.

This is the layer that makes tamper-evidence hold against an operator who holds the database and the signing key but not the bucket, not only against an outside attacker.

Why anchoring matters

Every chain entry is signed and hash-linked, so any modification to stored history is detectable on verification, provided the verifier trusts the signing key. An administrator who holds both the database and the active signing key (or, under KMS custody, the right to call Sign with it) can, in principle, rewrite a chain and re-sign it; the rewritten chain would verify cleanly. Anchoring closes this gap:

• Entries covered by an anchor cannot be rewritten without exposure: the rewritten chain no longer matches the immutable checkpoint in the bucket.
• Entries written since the last anchor sweep rely on the signature and hash chain alone. The span between sweeps is your tamper-exposure window, and that is what VAULT_ANCHOR_INTERVAL_MINUTES sets.

Anchoring detects tampering after the fact; it does not prevent it in real time. Without it, all tamper-evidence layers live in the same PostgreSQL database a privileged attacker would rewrite, which is why a production Server logs a boot warning while anchoring is disabled.

What gets anchored

Checkpoints are per record chain, swept incrementally on the anchor interval: each sweep checkpoints the record chains that advanced since their last checkpoint, in record order, for as long as its time budget allows, and a sweep that runs out of time resumes where it stopped on the next one. The platform-ops chain and each org's schema chain are checkpointed and anchored on every sweep as well. Each anchor object is a small JSON document (about a kilobyte) carrying the checkpoint's canonical COSE_Sign1 envelope (base64), the record ID, chain position, payload hash, signing key ID, the instance ID, a document version, and the anchor timestamp. Record payloads never leave your database; the anchor is a cryptographic commitment, not a copy.

Objects land at a deterministic key inside the configured bucket:

vault-anchors/<instance>/<recordId>/<chainPosition>.json

where <instance> is the instance id this database recorded on its first boot, from AGLEDGER_INSTANCE_ID. Every install gets its own: the Compose installer generates a UUID and writes it to your .env, and the Helm chart generates one into the release's ConfigMap on first install and keeps it across upgrades (the instanceId value pins it instead). So the segment is that UUID rather than a friendly name. A Server where nothing set an id anchors under default, and a Helm release whose ConfigMap carries no id (one installed by a chart that set none) keeps default, the prefix it has always anchored under.

Once an id is set it lives in the database, so a restore onto a rebuilt host keeps writing under the same prefix, and a boot whose AGLEDGER_INSTANCE_ID disagrees with it is refused, naming both values. Read the id from vault.anchoring.instanceId in GET /v1/admin/ops-summary, which reports the id the database carries, or from .env or the chart's ConfigMap (the values.yaml comment on instanceId has the command), before going looking for objects by prefix. A Server reads only its own prefix.

One chain position is one key for the life of the install, which is what lets the create-only precondition catch a re-anchored position. Verification, reconciliation and the anchor listing read only this layout. Releases before 1.8.0 wrote under a dated key, vault-anchors/<instance>/<YYYY-MM-DD>/<recordId>/<chainPosition>.json, which none of them read. A 2.0 Server starts from a new database, so its own checkpoints never have a dated anchor; a dated object in the bucket belongs to a 1.x install that shared the instance id.

Because checkpoints are per-chain, the object count scales with how many chains advance per interval, not with a fixed per-tick rate. A low-volume install produces a handful of objects per sweep; a high-volume install produces one per chain that advanced.

Checkpoints are created even when anchoring is disabled. They are the in-database layer of the same control. Enabling anchoring adds the external, immutable copy.

Enable it (Docker Compose)

Anchoring is disabled by default. In your .env:

VAULT_ANCHOR_ENABLED=true
VAULT_ANCHOR_S3_BUCKET=my-org-agledger-anchors
VAULT_ANCHOR_S3_REGION=us-east-1
# VAULT_ANCHOR_S3_ENDPOINT=http://minio:9000   # non-AWS S3-compatible stores (MinIO, etc.)
VAULT_ANCHOR_INTERVAL_MINUTES=360              # tamper-exposure window; see below
VAULT_ANCHOR_OBJECT_LOCK=true                  # COMPLIANCE-mode immutable retention (default)
VAULT_ANCHOR_RETENTION_DAYS=2555               # ~7 years (default)

Credentials come from the standard AWS credential chain (instance role, AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, or a profile). Two processes touch the bucket:

• The worker runs the anchor sweep and the daily sample-verify job: it needs s3:PutObject (plus s3:PutObjectRetention when Object Lock is on) and s3:GetObject.
• The API serves the admin anchor endpoints: it needs s3:GetObject and s3:ListBucket. It never writes to the bucket. Whether the store honours create-only writes is what the worker saw, recorded in the database, and the API reports that.
• Either process should also hold s3:GetBucketVersioning, s3:ListBucketVersions and s3:GetObjectVersion, so a verify or reconcile call from it can count a key's versions, report a fork, and find an anchor hidden behind a delete marker. Without them, storedVersions comes back null with a versionsNote saying a key written more than once cannot be detected from that process. That is a silent narrowing of coverage, not a refusal.

Grant only s3:PutObject to the worker and the daily sample-verify job it runs returns s3_error for every checkpoint it samples. The API's own verify endpoint is unaffected, because it authenticates to the bucket with its own s3:GetObject and s3:ListBucket grant rather than the worker's.

Defaults if unset: bucket vault-anchors, region from AWS_REGION (falling back to us-east-1), Object Lock on, retention 2555 days, interval 360 minutes.

http://minio:9000 in the endpoint line is a container named minio on the stack's Compose network, which the installer does not create. Run the store on that network so the api and worker can resolve it (docker network ls shows the network; it is compose_default under the default project name), or point the endpoint at an address they can reach. .env is read when a container is created, so after editing it recreate the stack from the compose/ directory:

cd compose && docker compose up -d

Enable it (Kubernetes / Helm)

The chart passes these through extraEnv (there is no dedicated vault.anchor values block):

extraEnv:
  - name: VAULT_ANCHOR_ENABLED
    value: "true"
  - name: VAULT_ANCHOR_S3_BUCKET
    value: my-org-agledger-anchors
  - name: VAULT_ANCHOR_S3_REGION
    value: us-east-1

The chart generates AGLEDGER_INSTANCE_ID itself, so leave it out of extraEnv; the render refuses an entry of that name and points at instanceId. Pin instanceId when the chart cannot read its own ConfigMap back:

• a renderer such as Argo CD (secrets.gitops refuses the render until it is set);
• secrets.existingSecret with no ConfigMap to read, a first install included (the render refuses until it is set);
• a release restored from another release's backup, which takes the id the post-restore Job prints.

Two releases on one id and one bucket would anchor under one prefix.

With the chart's NetworkPolicy on (the default), the api and worker may reach only DNS, the database, the OTLP collector and port 443 on public addresses. An anchor store anywhere else needs an egress rule in networkPolicy.extraEgress: an in-cluster MinIO on 9000, or a store on a private address such as an S3 interface endpoint.

networkPolicy:
  extraEgress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: minio
          podSelector:
            matchLabels:
              app.kubernetes.io/name: minio
      ports:
        - port: 9000
          protocol: TCP

On EKS, grant the bucket credentials through IRSA: put the role ARN in serviceAccount.annotations (eks.amazonaws.com/role-arn: <role arn>). The chart has no per-workload service account, though: api, worker, migrate and the preflight Job share one ServiceAccount, so the API pod ends up holding the same bucket-write permission the worker needs. There is no chart-native way to give the worker anchor-write access without also giving it to the API.

Prepare the bucket

Create the bucket with Object Lock enabled (an existing bucket takes it only once versioning is on, through aws s3api put-object-lock-configuration):

aws s3api create-bucket \
  --bucket my-org-agledger-anchors \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

AGLedger applies per-object COMPLIANCE-mode retention of VAULT_ANCHOR_RETENTION_DAYS as each anchor is written. In COMPLIANCE mode no one, including the account root, can delete the object or shorten its retention until it expires. That permanence is the point; choose the retention deliberately.

For the strongest posture, put the bucket in a separately-administered account (or a different provider), so no single set of credentials controls both the database and the anchors. A bucket in the same account as the Server is a single trust domain.

Air-gapped installs: point VAULT_ANCHOR_S3_ENDPOINT at MinIO (path-style addressing is handled automatically) and enable MinIO's object locking on the bucket. On Kubernetes, open egress to it with networkPolicy.extraEgress (above).

Choose the interval

VAULT_ANCHOR_INTERVAL_MINUTES (default 360, from 5 to 2880, snapped to cron granularity) is a risk-tolerance dial, not a performance knob. A value outside that range refuses boot, naming the variable. It bounds how much recent history has no external evidence yet:

• Tighten (15–60 min) for high-value, irreversible workloads where hours of unpinned records is too much exposure.
• Default 360 (6 h) is a reasonable posture for most deployments.
• Loosen (up to 2880 = 48 h) where object economy matters more than a tight window.

The sweep is incremental, so short intervals stay cheap on the database; the cost of tightening is the permanent Object Lock object count.

Verify and monitor

Posture at a glance is the vault.anchoring block of GET /v1/admin/ops-summary: enabled, intervalMinutes, bucket, instanceId, and workerEnabled with reconciled, which say whether the worker that runs the sweep agrees with the API. reconciled: false means the two processes disagree on their environment; the worker is the one that decides whether anchoring happens.

The admin anchor calls below take a platform key with the admin:system scope. List a record's anchors (recordId is required; with anchoring off the list is empty):

curl -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" \
  "$AGLEDGER_API_URL/v1/admin/vault/anchors?recordId=<uuid>"

Each entry gives the object key, its chainPosition, lastModified, size, the number of stored versions, and hidden: true when a delete marker hides the key.

Verify anchors against the database (exposes any divergence between the live chain and the pinned state). Without chainPosition in the body it checks the record's latest 10 checkpoints:

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" -H "Content-Type: application/json" \
  -d '{"recordId": "<uuid>"}' \
  "$AGLEDGER_API_URL/v1/admin/vault/anchors/verify"

{"data":[{"recordId":"019fe911-...","chainPosition":1,"match":true,"outcome":"verified",
  "detail":"Anchor matches database checkpoint","storedVersions":1,"versionsNote":null}],
 "nextSteps":[...]}

outcome is one of verified, tamper, fork, superseded, hidden, no_such_key, missing_or_hidden, empty_body, s3_error, no_checkpoint or anchoring_disabled. Only verified answers match: true, and only tamper, fork, hidden and empty_body are findings. Branch on outcome, not on match alone.

• hidden is an anchor deleted without a version id. On a versioned bucket COMPLIANCE retention allows that, leaves the locked version in place, and hides it behind a delete marker, so a plain read answers NoSuchKey. Nothing the Server does deletes an anchor, so treat it as tampering with the bucket. detail says whether the hidden version matches the database, and the anchor listing shows the key with hidden: true.
• no_such_key means no version of the key is stored at all, which is the ordinary state of a checkpoint younger than the anchor interval.
• missing_or_hidden is a plain read answering NoSuchKey when the key's versions could not be listed (the store did not answer, or the process lacks s3:ListBucketVersions), so it cannot tell a missing anchor from a hidden one. versionsNote says why; verify again once the listing works.
• s3_error means the store could not be read and nothing was compared. anchoring_disabled and no_checkpoint mean there is nothing to compare yet.
• superseded is a tamper or fork at a position inside a rewind you have already acknowledged: the anchor holds the history the restore replaced. It does not refuse writes.

An anchor that is a different signed checkpoint over the same chain tip (one re-created for its position after a restore, signed at a different time) is verified, because the chain both commit to is the same. A tamper or fork whose anchor was signed for an issuer other than this Server's current AGLEDGER_EXTERNAL_URL carries it as signedForIssuer, and detail names both issuers.

A fork is two anchor documents of the key's own position (this Server's instance, the record and position the key names) that commit to different chain tips. Anything else a key holds, such as a version that is not JSON, another position's or another instance's document, or the same document twice, is not a fork and refuses nothing; detail names those versions. Verify reads up to 25 versions of a key, newest first. A key holding more than one version that it could not read in full (more versions than that, or versions the store refused because the process lacks s3:GetObjectVersion) is reported as a fork, because a fork among the unread versions cannot be ruled out; reconcile reads further. Anchors are not checked for a valid signature before they count as evidence, so a well-formed document with a different chain tip, written by anything that can write to the bucket, is reported and refuses writes like any other.

A daily background job, on the worker, verifies a random sample of 20 checkpoints against their anchors (checkpoints not yet anchored are skipped, so a run may verify fewer than 20) and compares the bucket against the database the same way POST /v1/admin/vault/anchors/reconcile does. That daily job is the unattended tamper detector. The worker also runs the same bucket comparison once at its own boot, so a rewind between backup and restore surfaces at startup rather than waiting for the clock. Three Prometheus counters back alerting:

• agledger_vault_anchor_upload_total{outcome}, on the worker, which runs the sweep. outcome is created, replayed, fork, superseded, unconditional or error; there is no success label. Alert on outcome="error": a misconfigured bucket or IAM regression silently widens the tamper-exposure window. A healthy sweep increments outcome="created", or "replayed" for a re-anchor of the same checkpoint or of another checkpoint signed over the same chain tip. superseded counts checkpoints at positions inside an acknowledged rewind, which are not anchored because the old history holds those keys.
• agledger_vault_anchor_verify_total{outcome}, incremented by both the API's verify endpoint and the worker's daily sample-verify job, so scrape it from both processes. Alert on outcome="tamper", outcome="fork" or outcome="hidden", and investigate outcome="empty_body", which is what overwriting an anchor without also forging a matching envelope looks like.
• agledger_vault_anchor_reconcile_findings_total{finding} counts rewound, missing_locally, fork and hidden keys found by the bucket comparison. A hidden key counts toward its record's highest anchored position like any other, and a key holding two anchors over different chain tips is a fork whether or not a delete marker still hides one, so hiding an anchor and re-anchoring its position is found too. A version that is not an anchor document of its key counts under unverified in the reconcile report and refuses nothing, and one planted before the Server's first anchor at that key hides neither a later fork nor a rewind.

When the store cannot be read, the anchor listing and the reconciliation answer 503 with type: /problems/anchor-store-unavailable, naming the bucket, the endpoint and what the store answered; retryable is false for a refusal (credentials, permissions, a missing bucket). Chain writes are unaffected: the worker keeps checkpoints queued until the store takes them.

The first two counters can go silent rather than record error. A store the S3 client cannot reach at all, or a failure before the anchor key is even resolved, throws before any outcome is counted. An absent series where the sweep or the daily job should have run is as much a signal as outcome="error". The checkpoints whose anchor has not landed stay queued and are retried every sweep, and two gauges count them whatever the counters saw: agledger_vault_anchor_pending and agledger_vault_anchor_pending_oldest_age_seconds (both 0 while anchoring is off). The shipped AGLedgerVaultAnchorsNotLanding alert fires when that age passes twice the anchor interval plus an hour. Nothing shipped covers a daily sample-verify job that stops running; watch for the verify counter not updating on its cadence.

Uploads are deliberately non-blocking: an S3 outage never stops notarization. The counter is the signal that anchors are not landing.

With anchoring disabled, the verify endpoint still answers rather than erroring: it returns outcome: "anchoring_disabled" and match: false, with detail: "Vault anchoring is not enabled". Read that as configuration, not as a tamper finding.

In-database checkpoints (anchored or not) are readable at GET /v1/audit-vault/checkpoints. Full request/response shapes are in the API reference.

Reconcile the bucket against the database

anchors/verify starts from the checkpoints the database holds, so it cannot see a position that is in the bucket and no longer in the database. That is exactly the shape a restore from an older backup leaves behind, and POST /v1/admin/vault/anchors/reconcile is the call that catches it: it walks this Server's anchor prefix in the bucket and compares the highest anchored position per record against the current chain head.

curl -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" \
  "$AGLEDGER_API_URL/v1/admin/vault/anchors/reconcile"

It reports rewound for an anchored position past this database's chain head, missing_locally for a record the bucket holds anchors for and this database has no entries and no checkpoints for at all, and a posture note on whether the store honours create-only writes and reports Object Lock enabled. The walk stops at a key cap and a time budget (maxKeys and deadlineMs in the body set them); a walk that stops early says truncated: true and why, so read that before reading the report as clean. Either finding refuses every chain write with 409 CHAIN_REWIND_DETECTED until an operator acknowledges it at POST /v1/admin/vault/rewind/acknowledge; GET /v1/admin/vault/rewind carries the evidence in the meantime. The Server runs the same comparison at worker boot and daily, and restore.sh runs it before it starts a restored Server, so an anchored install checks itself rather than waiting to be asked. The recovery runbook covers the acknowledge step.

Every anchor under this Server's prefix whose document parses and agrees with the key it was found at is evidence, whatever issuer its checkpoint names and whichever key signed it. One signed for an issuer other than this Server's current AGLEDGER_EXTERNAL_URL (compared as URLs, so a trailing slash is not a difference) is a rewound, missing_locally or fork finding like any other and refuses writes. The finding carries signedForIssuer, and the log names both issuers: another Server may share AGLEDGER_INSTANCE_ID, or the external URL changed. The way out is the acknowledgement, or fixing the configuration. The per-install instance id is what keeps two Servers sharing a bucket apart.

Limitations

• Detect, not prevent. An operator with the signing key can still rewrite within the current window; the next sweep exposes everything already pinned.

• One bucket = one trust domain. Separate the account administering the anchors from the one running the Server.

• COMPLIANCE retention is irreversible. You cannot shorten it after the fact.

• An acknowledged rewind leaves a gap in anchor coverage. The positions the restore lost stay anchored with the lost history, so the checkpoints the restored chain writes at those positions are superseded rather than anchored, and a later change to them is not visible to the anchors. Anchor coverage resumes at the first position past the highest one the acknowledgement covered.

• An acknowledgement covers what it signed, and the store's clock decides "before". The RESTORE_EPOCH entry lists every range it covers under coveredFindings, and only a range listed in an entry that verifies under this Server's key counts. A row written into the database by anything else covers nothing, and a rewind state marked acknowledged without such an entry keeps refusing writes until you acknowledge through the route. GET /v1/admin/vault/rewind shows the ranges under openFindings before you acknowledge; read them, because a range no detection accounts for was written to the database by something else, and acknowledging it signs it. One loss is one range: a reconciliation that finds the chain head has moved up inside an open range since it was recorded (the restore's own RESTORE_EPOCH moves the platform-ops head this way) records it as that range, not as a second one, while a range that reaches past an open one, with an anchor above it or a database further behind, is new evidence and is recorded as its own. An anchor counts as seen only when the store's Last-Modified is at least a second before the time the acknowledgement signs, so keep the store's clock and the Server's in step. Each reconciliation validates a bounded number of findings, so a restore that touched many records can take more than one acknowledgement before a reconciliation reports only acknowledged. On an install that has never signed, an unsigned acknowledgement is accepted, because nothing stronger exists there.

• An acknowledgement counts only on this chain, and only under a key you still trust. Its RESTORE_EPOCH has to sit on an intact link of the platform-ops chain, and it covers only findings the database recorded at or before the detection it signs (both times are the database's own). A genuine acknowledgement copied back into the database after a later restore therefore covers nothing the restored Server has found since, however the restore was made and whether or not its marker was written. One the restored backup already held is not a copy: it keeps covering the findings it signed, judged by a process that trusts the key that signed it.

  An acknowledgement signed by a key you later retire with force, or name in VAULT_DISTRUSTED_KEYS, covers nothing whenever it was signed: a leaked key could have signed it at any time, and it is a standing permission rather than a record of the past. Chain writes are refused again at once on the process that ran the retirement, and on the others on its broadcast or their next re-read of the rewind state. GET /v1/admin/vault/rewind lists each finding such an acknowledgement covered under openFindings, with acknowledgement.reason saying why it no longer counts, and the next acknowledgement, made on a process on the current key, records them again and covers them. Entries the retired key signed before its retirement still verify as chain entries when the current key was staged with it as VAULT_SIGNING_KEY_PREVIOUS (the key-compromise runbook's order); only its acknowledgements stop counting.

  What counts for a finding is the acknowledgement on its latest row, and reconciliation, the write guard and GET /v1/admin/vault/rewind all read that row. Once its acknowledgement stops counting, writes are refused even if an older acknowledgement of the same range still verifies. A reconciliation on a process that can judge it (below) that finds the same evidence again, or part of it because the chain head has moved up inside the range since, records nothing new over it, so acknowledgement.reason stays readable. Once a newer acknowledgement covers the finding, the older one is history and is neither reported nor judged again.

• A process that cannot judge an acknowledgement waits or fails closed, by its gate. Whether one counts is decided by the keys the judging process trusts, which start from its own key, so a process whose signingKey.gate is not usable cannot say one does. When the gate is signer_unreachable (the KMS holding its key is not answering right now), it could not tell: the evidence is neither recorded nor cleared, an anchor the checkpoint sweep was writing stays queued, a reconciliation reports status: incomplete with the cause in truncatedReason, and the next run asks again, so a KMS outage at a worker's boot does not re-open an acknowledged rewind.

  The other gates (unanchored, unregistered, retired) hold until someone changes the process's configuration, so there it fails closed. Its reconciliation and its checkpoint sweep take the evidence as not covered: a rewind acknowledged earlier, or a fork at a position it covered, is recorded as an open finding and chain writes are refused on every process until you fix that process's key and acknowledge again. Nothing is dropped and nothing is assumed covered. In GET /v1/admin/vault/rewind, each such finding carries previousAcknowledgement: the earlier acknowledgement's RESTORE_EPOCH, time, range and the note it signed, with reopenedBecause inside it naming the gate that caused it. The Server renders reopenedBecause from a fixed cause code, and shows previousAcknowledgement only when the answering process verifies a RESTORE_EPOCH this Server signed listing that range, so a row written straight into the database cannot put words or a borrowed note there.

  POST /v1/admin/vault/rewind/acknowledge on such a process answers 503 (SIGNING_KEY_UNUSABLE) and changes nothing, and a chain write sent to it answers the same 503 rather than 409 CHAIN_REWIND_DETECTED. restore.sh's own reconciliation does not run on such a process: it prints the gate and its fix instead. The worker it starts reconciles at boot, and under a key that is still not usable that records the rewind open again, which is why the recovery runbook has you set the predecessor before running restore.sh.

• A rewind does not stop a key from registering. The refusal holds back every chain write except the entry that marks a restore and the signing-key registry's own entries, because the acknowledgement that lifts it is signed by a registered key. A key the restored registry does not hold, staged with VAULT_SIGNING_KEY_PREVIOUS or VAULT_TRUST_ANCHORS, registers at boot while writes are refused, POST /v1/admin/vault/signing-keys/{keyId}/retire works too, and the acknowledgement is then signed under the staged key. Registering acknowledges nothing: records and every other write keep answering 409 until you acknowledge.

• The comparison trusts the database's chain head. Reconciliation compares anchored positions with the highest position the database holds. A writer who can insert rows into audit_vault or vault_checkpoints directly can raise that head and so hide a rewind from the comparison. The vault scan (POST /v1/admin/vault/scan) reports such a row as a broken chain, so run it after any restore.

Related

• Key-compromise runbook: anchors are what distinguish pre-compromise history from a forged rewrite.
• Audit runbook: offline chain verification with published keys.
• Day-2 operations: ops-summary and alerting.