Day-2 Operations

This page covers the recurring work after install: watching health, scraping metrics, rotating the signing key, keeping partitions ahead of growth, reloading config, and upgrading. Two facts shape everything below.

"detail": "Action 'ROTATE_VAULT_SIGNING_KEY' requires platform role; caller resolved as 'admin'.",
"recoveryHint": "This action is platform-scoped ... mint a platform-role key via POST /v1/admin/api-keys (platform-only)."

Health and readiness probes

Three unauthenticated endpoints, shaped for orchestration probes:

curl -s "$AGLEDGER_API_URL/livez"          # liveness: is the process up?
curl -s "$AGLEDGER_API_URL/readyz"         # readiness: can it serve (DB reachable)?
curl -s "$AGLEDGER_API_URL/health"         # detailed status + version

{"status":"alive","timestamp":"2026-06-09T15:55:26.247Z"}
{"status":"ready","version":"1.3.3","timestamp":"2026-06-09T15:55:26.251Z"}
{"status":"ok","version":"1.3.3","timestamp":"2026-06-09T15:55:26.237Z"}

Wire livez to your liveness probe and readyz to your readiness probe. /health/ready is an alias of readyz. None require auth, so probes need no credentials.

The aggregate health view

GET /v1/admin/system-health (platform key) is the one-call operator summary: database latency and pool, every pg-boss queue, and process memory.

curl -s -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/system-health"

{
  "status": "healthy",
  "degradedReasons": [],
  "uptime": 68.95,
  "database": { "status": "healthy", "latencyMs": 0.23, "pool": { "total": 4, "idle": 4, "waiting": 0 } },
  "queues": {
    "phase2-gate":          { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "webhook-delivery":     { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "maintenance":          { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "federation-outbound":  { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "federation-outbound-dlq": { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 }
  },
  "process": { "rssMb": 143.53, "heapUsedMb": 59.17, "heapTotalMb": 64.8 }
}

A growing queues.*.failed count or a climbing pool.waiting is your earliest signal of trouble.

status is the field to alert on, and degradedReasons is why it moved. It reads degraded when the database cannot serve, and when any dead-letter queue holds work: a dead-lettered job has exhausted its retries and stays there until you recover it, so it is a condition that needs a person. A failed count on a live queue is not one of those conditions and deliberately does not move status: the job is being retried and clears itself, and a field that flickers stops being read. That is why the failed column above is worth watching by eye even while status is healthy.

{
  "status": "degraded",
  "degradedReasons": [
    "2716 dead-lettered job(s) in federation-outbound-dlq; they will not be retried until an operator recovers them"
  ]
}

Both dead-letter surfaces count, and they are different shapes. Queue dead letters (federation, cascading gates, and the rest) live in pg-boss and are recovered from their own admin route, e.g. GET /federation/v1/admin/dlq, which lists each entry with the peer and record it belongs to. Webhook dead letters live in a table, not a queue: a permanent failure (an SSRF refusal, a 410, a 4xx, an undecryptable secret) never reaches the queue at all, so watching queue names alone would read healthy through a total webhook outage. Those are at GET /v1/admin/webhook-dlq, and degradedReasons names that route when they are what moved status.

When you recover a queue DLQ with POST /federation/v1/admin/dlq/recover, the recovered count is the number of rows the call actually removed. Before 1.5.0 it counted the rows it had reached rather than the rows it changed, so a run could report recovered: 500 against a queue that had not moved by a single row. Treat the number as an instruction to re-check the depth, not as a receipt: a row another worker took in the meantime is now reported as what it is instead of padding the total.

Metrics

GET /metrics exposes Prometheus metrics (unauthenticated; restrict at your ingress). All series are prefixed agledger_. The ones worth alerting on:

| Metric | Watch for | |---|---| | agledger_vault_integrity_check_results_total{result="broken"} | Any increase - a chain failed periodic verification | | agledger_db_pool_waiting_connections | Sustained nonzero - pool saturation | | agledger_pgboss_queue_size{queue=~".*-dlq",state="total"} | Growth - jobs dead-lettering into a DLQ | | agledger_pgboss_queue_size{state="queued"} | Sustained growth on any queue: a worker is down or cannot keep up. Every queue the Server runs reports here, federation included | | agledger_pg_listener_reconnect_failures_total | Increase - cross-replica cache coherence degraded | | agledger_vault_checkpoint_skipped_broken_total | Increase - a record went un-anchored | | agledger_outbound_ssrf_blocked_total | Increase - outbound calls hitting the SSRF guard | | agledger_federation_zero_row_oldest_candidate_age_seconds | Climbing toward the recovery horizon - crash-orphaned records are aging out unrecovered. Federated deployments only; see Federation delivery and recovery |

curl -s "$AGLEDGER_API_URL/metrics" | grep agledger_vault_integrity_check_results_total

Note: agledger_partition_runway_days (next section) is exposed by the worker process, not the API process. Scrape both processes, not just the API. The federation series are worker-only too.

You do not have to write those alerts yourself. monitoring/alerts/agledger.rules.yml in the install repository ships 14 rules covering silent drops, chain integrity, federation delivery, and availability, and the bundled Prometheus loads them already. No Alertmanager is bundled and no routing is configured, because receivers and escalation policy are yours to decide; until you point them somewhere the rules evaluate on Prometheus' own /alerts page, and each carries a severity label of critical or warning as the routing hook.

Three Grafana dashboards auto-provision with install.sh --with-monitoring: an overview (traffic, chain throughput, saturation), data-integrity surveillance, and a silent-drop board with a panel per fire-and-forget path that swallows its error. monitoring/README.md describes all three, including how to import them into your own Grafana instead of the bundled one.

Signing-key rotation

Rotation is the load-bearing day-2 task. The guarantee that makes it safe:

Rotating the signing key never breaks verification of already-signed records. Retired keys stay in the published registry, so a record signed under an old key still verifies after any number of rotations. No re-signing, no downtime.

How rotation works

The engine signs with the key in VAULT_SIGNING_KEY. To rotate: generate a new key, set it as VAULT_SIGNING_KEY, move the prior key to VAULT_SIGNING_KEY_PREVIOUS, and restart the process. On boot the engine retires the old active key in the registry and promotes the new one:

INFO: Retired previous active signing key during bootstrap
INFO: Bootstrapped active signing key into registry

POST /v1/admin/vault/signing-keys/rotate (platform key) reconciles the registry to the env-configured key. In a normally-booted process the boot step has already promoted it, so the endpoint reports already_active - use it to confirm, not to mint:

{ "previousKeyId": null, "newKeyId": "c4dd3e20388b594d", "status": "already_active" }

After rotation, both keys appear at GET /v1/verification-keys - the new one active, the prior one retired but still resolvable:

curl -s "$AGLEDGER_API_URL/v1/verification-keys"

6a639248683aab56 | active  | activated 2026-05-26 | retired null
affc2b9bfb22144e | retired | activated 2026-05-26 | retired 2026-05-26

Proving the guarantee

Records signed before the rotation must still verify. Dump the vault and verify it offline (see the audit runbook for the full handoff) - the dump now carries two signing keys and entries signed by both:

./scripts/vault-dump.sh ./dump      # "vault_signing_keys": 2
python3 verify-dump.py ./dump       # stock RFC 9052 / Ed25519 check — see the audit runbook

[PASS] stock-library offline verification
  audit_vault entries : 5
  signatures verified : 5
  failures            : 0
  signing keys        : 2

Zero failures across records signed by the retired key and the active key - retired keys travel in the dump, so the verifier resolves whichever key signed each entry. That is the guarantee, demonstrated end to end.

Partition maintenance

Several high-volume tables are range-partitioned by month, each with a DEFAULT catch-all partition so a write never fails for lack of a partition. The worker pre-creates upcoming partitions and exposes runway as a gauge (agledger_partition_runway_days). You can also query the source function directly:

psql "$DATABASE_URL" -c "SELECT table_name, runway_days, default_rows FROM partition_runway();"

     table_name     | runway_days | default_rows
--------------------+-------------+--------------
 audit_vault        |         570 |            0
 events             |         570 |            0
 webhook_deliveries |         570 |            0
 system_audit_log   |          83 |            0

runway_days is days until the latest pre-created partition is reached; default_rows should stay 0 - a nonzero value means writes are landing in the DEFAULT partition and the worker is falling behind. Alert on low runway_days and on default_rows > 0.

Config-as-code hot reload

If you run with PROVISIONING_CONFIG_PATH set, orgs, agents, webhooks, and contract schemas are declared in YAML and reconciled on every boot - see the provisioning runbook for the directory layout. Reload changes without a restart via SIGHUP or POST /v1/admin/provisioning/reload (platform key). Check current state first:

curl -s -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/provisioning/status"

{"configured":true,"configPath":"/etc/agledger/provisioning","dryRun":false,"prune":false,"lastReloadAt":"2026-06-09T15:59:53.404Z","managed":{"orgs":1,"agents":2,"webhooks":0,"schemas":2},"loadErrors":["webhooks/acme.yaml: Environment variable ACME_WEBHOOK_SECRET is not set and has no default"]}

curl -s -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/provisioning/reload"

{
  "orgs":    { "created": 0, "updated": 1, "pruned": 0 },
  "agents":  { "created": 0, "updated": 2, "pruned": 0 },
  "schemas": { "created": 0, "updated": 2, "pruned": 0 },
  "apiKeys": { "created": 0, "skipped": 3, "generated": [] },
  "errors": [
    { "resource": "config", "error": "webhooks/acme.yaml: Environment variable ACME_WEBHOOK_SECRET is not set and has no default" }
  ]
}

Reload is idempotent: unchanged resources count as updated, existing keys as skipped. It is also fail-open - a single invalid file (here, an unset ${ACME_WEBHOOK_SECRET} substitution in a webhook file of your own) is reported in errors[] while every valid resource still applies. Newly minted keys appear in apiKeys.generated[] with their raw value exactly once, in that response body - capture them then.

Fail-open is the shape to watch. A file that fails to parse is skipped whole: its resources are silently absent, the rest of the reconcile succeeds, and the Server comes up healthy. status re-reads the directory on every call and lists every unloadable file in loadErrors[], so that field, not the boot log, is the durable signal. The agledger_provisioning_errors gauge (labels stage="load" / stage="reconcile") carries the same state for alerting and stays non-zero until a clean reconcile clears it.

A bare ${VAR} with no default is the usual cause: it is a hard parse error when the variable is unset. Use ${VAR:-default} wherever a sensible default exists, and inject real secrets through the pod environment (extraEnv / extraEnvFrom on the chart).

Version upgrades

Upgrade with scripts/upgrade.sh (from the agledger-ai/install repository - see the install runbook for the full procedure and the air-gapped path). Migrations run automatically and are advisory-locked and checksum-verified, so a migration runs once and only once even across replicas, and a tampered or reordered migration is refused rather than applied. For diagnostics to attach to a support request, scripts/support-bundle.sh collects health, config (secrets redacted), and recent operational events.

Restart the process after swapping the image. Already-signed records verify across versions - the chain format is stable and historical keys resolve.

On an external database, the upgrade checks the runtime role twice: before the pre-upgrade backup, and again after migrations. That role can stop being able to serve without the install changing. A credential-rotation policy that drops and recreates it brings it back without its agledger_app membership, because role membership does not survive a DROP ROLE. Left unchecked, the first thing to fail is pg_dump during the backup, which reports the table it was refused and prints its whole LOCK TABLE statement without naming the role.

Each check names the role and the exact GRANT. Stopping at the first one costs nothing: no backup has been taken, no migration has run, and neither the version nor the image digest in compose/.env has moved, so the running install is serving exactly as it was. Stopping at the second one leaves the upgrade part-done, which it says: the migrations are applied, the worker stays stopped until the upgrade finishes, and the previous version is still serving against the new schema. Either way, applying the grant and re-running finishes it, and migrations already applied are skipped.

v1.2.0: migration 002 and the PostgreSQL 18 upgrade order

v1.2.0 ships one schema migration. It adds the record_types column behind type-segregated webhook subscriptions, repairs two indexes, and adopts PostgreSQL 18's native uuidv7() where the PG17 polyfill is still in place. It runs automatically under the same advisory-lock, checksum-verified runner as every migration; on its own it needs no operator action.

If you are also moving from PostgreSQL 17 to 18 (see the in-place upgrade procedure), the order matters:

-- list every column default still bound to the polyfill
SELECT d.adrelid::regclass AS tbl, a.attname AS col
FROM pg_depend pd
JOIN pg_attrdef d ON d.oid = pd.objid AND pd.classid = 'pg_attrdef'::regclass
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE pd.refobjid = 'public.uuidv7()'::regprocedure;

-- for each row returned:
ALTER TABLE <tbl> ALTER COLUMN <col> SET DEFAULT uuidv7();

-- then, once nothing depends on it:
DROP FUNCTION public.uuidv7();

The AWS install guide carries the same remediation as a single idempotent DO block you can run instead of the per-column form.

Running on the polyfill is not a correctness problem - it mints spec-compliant UUIDv7 values - so treat this as tidy-up: native uuidv7() is a modest bulk-write win, and the cleanup can wait for a maintenance window.

v1.3.4: migration 003

v1.3.4 ships the first new migration since v1.2.0, bringing the set to three. It is additive and needs no operator action:

The upgrade is still a rolling image bump, but it now runs a schema step where v1.3.2 and v1.3.3 did not. A fresh install applies all three in one Job and reports Complete 1/1; re-running against a database already at that head applies nothing and reports a count of 0.

If you have automation that deletes or rewrites vault_checkpoints rows, it will start failing against the new guard. That is the intended effect, but it is the one behavior change here that can surface as a broken job rather than a silent no-op.

Request timeouts

CONNECTION_TIMEOUT_MS (default 60000) is the socket inactivity timeout: how long a connection may go without bytes moving before the server destroys it.

The ordering between the three timeout knobs is what matters. Keep it above HANDLER_TIMEOUT_MS (default 30000) and below KEEP_ALIVE_TIMEOUT_MS (default 72000). If the socket timeout fires first, a slow request is severed underneath the handler instead of failing through the route, and the failure mode is unpleasant: on a bulk write the batch commits, the client sees a dead socket with no response, and it has no way to tell a committed batch from a lost one.

Recovering from a severed socket: retry the request with the same Idempotency-Key. A commit that already landed is returned rather than repeated.

Two signals tell you the value is too low for your traffic, and you need both because a severed socket never reaches the normal response path:

Raise CONNECTION_TIMEOUT_MS if you see them on legitimately slow bulk work, keeping it under the keep-alive value.

Federation delivery and recovery

Skip this section if you run a single Server with no peers. Nothing here applies until federation is configured, and the recovery sweeps below short-circuit before they touch the records table when a Server has no active peers.

Is this peer reachable?

GET /federation/v1/admin/peers answers it. Read lastDeliveryAt (the last outbound message that got a 2xx), consecutiveDeliveryFailures (reset to 0 on a success) and lastDeliveryError (cleared on the next success). A frozen lastDeliveryAt with a climbing failure count and a transport error is a peer that has gone away, with retries still in flight until the job dead-letters.

Two fields on the same object do not answer it, and it is worth knowing why:

For a count rather than a list, GET /v1/admin/ops-summary reports federation.peers with the registration counts and, partitioning the active ones, delivering / failing / neverDelivered. That last split is the one a dashboard needs: active alone reads the same for a peer taking every delivery and a peer that has never taken one.

For the backlog behind those deliveries, watch agledger_pgboss_queue_size{queue="federation-outbound"} or read GET /v1/admin/system-health, which reports the same queues as JSON, and whose status goes degraded once those retries give up and dead-letter.

Expect peers to lag, and size for it

POST /v1/records returns as soon as the record is notarized locally; peers receive their signed copy afterwards. Your local chain is complete and verifiable the entire time a peer is behind, so peer lag is a delivery concern, never an integrity one.

The outbound worker drains at a per-pod ceiling of FEDERATION_OUTBOUND_CONCURRENCY * FEDERATION_OUTBOUND_BATCH_SIZE / FEDERATION_OUTBOUND_POLLING_INTERVAL_SECONDS. On the v1.3.4 defaults that is 16 * 4 / 1.0 = 64 legs/sec/pod, up from 16 in v1.3.3. A leg is one record to one peer, so a record shared to two peers costs two legs. Notarizing 4,100 shared records against two peers queues 8,200 legs and puts peers roughly two minutes behind. Ingest is faster than drain by design, so a bulk share always builds a queue.

FEDERATION_OUTBOUND_BATCH_SIZE now defaults to 4 rather than 1. This is the one throughput change in v1.3.4 that takes effect without you setting anything. If you had pinned the value explicitly, your setting still wins and you keep the old rate.

Raise batch size first if you need more throughput. It widens the fetch each worker loop already issues, so it lifts the ceiling without adding database round-trips. Concurrency and polling interval are the expensive pair and interact super-additively on database CPU: each in isolation costs roughly 30 to 40 percent on POST /v1/records p50/p95, but together at (32 / 0.5) they cost about +110 percent p50 against the (16 / 1.0) baseline. Re-measure record-creation latency after touching any of the three, and prefer scaling out worker pods over cranking a single pod far past the defaults.

Same-(peer, record) ordering is preserved regardless of these settings, so raising throughput never reorders state transitions for a given record.

The recovery sweep, and the window that abandons records

Outbound work is enqueued after the record's state change commits. A crash in that gap can leave a committed terminal record with no federation work queued at all. A sweep runs every two minutes on the worker process to find those and re-drive them through the normal publish path: same share gate, same peer set, same idempotency, so recovery can never deliver something the live path would not have.

Two knobs size it, and they are a pair:

| Knob | Default | What it controls | |---|---|---| | AGLEDGER_FEDERATION_ZERO_ROW_HORIZON_MINUTES | 360 (6h) | How far back the sweep looks | | AGLEDGER_FEDERATION_ZERO_ROW_BATCH_SIZE | 50 | Records re-driven per cycle (~25/min) |

The horizon is an abandonment boundary, not just a scan bound. A record whose updated_at falls outside the window is never recovered. Six hours covers a crash gap with wide margin, since the normal case recovers within a sweep interval or two. Widen it only if this Server can be down longer than that.

Each cycle takes the oldest candidates first, which is the only order that cannot starve the record closest to aging out.

The one number to alert on is agledger_federation_zero_row_oldest_candidate_age_seconds, the age of the head of that queue. Flat and low is healthy. Climbing toward the horizon means the candidate pool is refilling faster than one batch drains, which is the only remaining way a genuinely orphaned record ages out unrecovered. Page well before the horizon.

Which knob to turn when that age climbs: raise AGLEDGER_FEDERATION_ZERO_ROW_BATCH_SIZE. Widening the horizon in that state makes it worse, because it adds candidates to a pool that is already draining too slowly. The horizon is the knob for a different problem, a Server that is legitimately offline longer than six hours.

Records received from peers

A record this Server received from a peer is a read-only view of a row the originating Server owns, and only that Server fans it out. As of v1.3.4 such records are never recovery candidates. Before that they were, on the default AGLEDGER_DEFAULT_SHARE=true, which produced redundant deliveries back to the origin and permanently-rejected deliveries to third peers, growing by one for every projection a Server had ever received.

This matters operationally because a busy receiving Server accumulates those records without bound. agledger_federation_zero_row_projections_excluded reports how many were held out of the candidate pool on the last cycle, capped at the batch size. Non-zero is the healthy reading on any Server receiving federation traffic; it is the exclusion doing its job, not a problem.

Its companion agledger_federation_outbound_projection_skipped_total is a last-resort guard further down the publish path and is expected to stay at 0 permanently. Treat any increase there as a regression to report, not as confirmation that the exclusion is working.

Updated for API v1.5.0 on 2026-08-21 for the federation DLQ recovered count, read from the shipped fix rather than re-measured here. The rest of this document carries the earlier stamps below.

Updated for API v1.4.0 on 2026-08-09: the monitoring section now points at the 14 alert rules and three Grafana dashboards that ship with the install rather than leaving operators to write their own, and the provisioning section documents loadErrors[] on GET /v1/admin/provisioning/status plus the agledger_provisioning_errors gauge, which are the durable signals for a fail-open config load. Not re-run end to end against a live v1.4.0 install.

Validated against API v1.2.0 on 2026-07-06 (Developer Edition, Docker Compose: health/readiness, system-health, /metrics, partition_runway(), the signing-keys/rotate endpoint (already_active), and provisioning status + reload - all re-run live). The post-rotation two-key /v1/verification-keys listing and the two-key offline-verify proof are carried over from the v0.25.4 rotation run (a full key swap was not re-performed this release); rotation behavior is version-stable and the no-break-on-verify guarantee holds across versions. The migration-002 and PostgreSQL-18 upgrade-ordering notes reflect the v1.2.0 migration as shipped.

Reviewed for API v1.3.4 on 2026-08-03: unchanged. Spot-checked live on a v1.3.4 install: GET /v1/admin/system-health returns the documented summary (status, uptime, database latency and pool, per-queue depths), GET /metrics exposes the agledger_* series including agledger_vault_integrity_check_results_total, and a fresh install reports 3/3 migrations applied. This page's description of signing-keys/rotate as reconciling the registry is the accurate one; the key-compromise runbook matches it.

Reviewed for API v1.3.4 on 2026-07-27: 1.3.4 ships migration 003, the first new migration since v1.2.0, documented under "v1.3.4: migration 003". It is additive and needs no operator action, but the upgrade now runs a schema step. Adds CONNECTION_TIMEOUT_MS under "Request timeouts", and a "Federation delivery and recovery" section for the outbound-throughput default change (per-pod drain 16 to 64 legs/sec, the one change that takes effect without operator action), the two new recovery-sizing knobs, and the series to alert on. Federation content applies only to multi-Server deployments. Rotation, provisioning, and the health surfaces are unchanged.

Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 ships no new migration (the set stays at 001_consolidated.sql + 002_webhook_record_types.sql), so the upgrade remains a rolling image bump with no schema step. Day-2 operational surfaces are unchanged.

Reviewed for API v1.3.2 on 2026-07-13: v1.3.2 ships no new migration (the migration set stays at 001_consolidated.sql + 002_webhook_record_types.sql), so the upgrade is a rolling image bump with no schema step. Day-2 operational surfaces are unchanged.