> Markdown version of https://agledger.ai/docs/operations/day-2/
> Full index of this site for AI assistants: https://agledger.ai/llms.txt

# Day-2 Operations

This page covers the recurring work after install: watching health, scraping metrics, rotating the
signing key, keeping partitions ahead of growth, reloading config, and upgrading. Two facts shape
everything below.

- **Two processes.** A Server runs an API process and a worker process. The API serves requests;
  the worker runs scheduled maintenance (partition upkeep, recovery sweeps) and exposes its own
  metrics. Some gauges live only on the worker - noted where it matters. Both tiers are stateless
  and scale independently; see the [high-availability runbook](/docs/operations/high-availability/)
  for the supported multi-replica shape.
- **Two key roles.** Org-scoped `admin` keys govern one org. Cross-org operator surfaces
  (`system-health`, signing-key rotation, provisioning) require a `platform` key. An admin key
  calling these gets a `403` that says so:

```
"detail": "Action 'ROTATE_VAULT_SIGNING_KEY' requires platform role; caller resolved as 'admin'.",
"recoveryHint": "This action is platform-scoped ... mint a platform-role key via POST /v1/admin/api-keys (platform-only)."
```

## Health and readiness probes

Three unauthenticated endpoints, shaped for orchestration probes:

```bash
curl -s "$AGLEDGER_API_URL/livez"          # liveness: is the process up?
curl -s "$AGLEDGER_API_URL/readyz"         # readiness: can it serve (DB reachable)?
curl -s "$AGLEDGER_API_URL/health"         # detailed status + version
```

```json
{"status":"alive","timestamp":"2026-06-09T15:55:26.247Z"}
{"status":"ready","version":"1.3.3","timestamp":"2026-06-09T15:55:26.251Z"}
{"status":"ok","version":"1.3.3","timestamp":"2026-06-09T15:55:26.237Z"}
```

Wire `livez` to your liveness probe and `readyz` to your readiness probe. `/health/ready` is an
alias of `readyz`. None require auth, so probes need no credentials.

## The aggregate health view

`GET /v1/admin/system-health` (platform key) is the one-call operator summary: database latency and
pool, every pg-boss queue, and process memory.

```bash
curl -s -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/system-health"
```

```json
{
  "status": "healthy",
  "degradedReasons": [],
  "uptime": 68.95,
  "database": { "status": "healthy", "latencyMs": 0.23, "pool": { "total": 4, "idle": 4, "waiting": 0 } },
  "queues": {
    "phase2-gate":          { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "webhook-delivery":     { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "maintenance":          { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "federation-outbound":  { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 },
    "federation-outbound-dlq": { "waiting": 0, "active": 0, "delayed": 0, "failed": 0 }
  },
  "process": { "rssMb": 143.53, "heapUsedMb": 59.17, "heapTotalMb": 64.8 }
}
```

A growing `queues.*.failed` count or a climbing `pool.waiting` is your earliest signal of trouble.

`status` is the field to alert on, and `degradedReasons` is why it moved. It reads `degraded` when
the database cannot serve, and when any dead-letter queue holds work: a dead-lettered job has
exhausted its retries and stays there until you recover it, so it is a condition that needs a
person. A `failed` count on a live queue is not one of those conditions and deliberately does not
move `status`: the job is being retried and clears itself, and a field that flickers stops being
read. That is why the `failed` column above is worth watching by eye even while `status` is
`healthy`.

```json
{
  "status": "degraded",
  "degradedReasons": [
    "2716 dead-lettered job(s) in federation-outbound-dlq; they will not be retried until an operator recovers them"
  ]
}
```

Both dead-letter surfaces count, and they are different shapes. Queue dead letters (federation,
cascading gates, and the rest) live in pg-boss and are recovered from their own admin route, e.g.
`GET /federation/v1/admin/dlq`, which lists each entry with the peer and record it belongs to.
Webhook dead letters live in a **table**, not a queue: a permanent failure (an SSRF refusal, a 410,
a 4xx, an undecryptable secret) never reaches the queue at all, so watching queue names alone would
read healthy through a total webhook outage. Those are at `GET /v1/admin/webhook-dlq`, and
`degradedReasons` names that route when they are what moved `status`.

When you recover a queue DLQ with `POST /federation/v1/admin/dlq/recover`, the `recovered` count is
the number of rows the call actually removed. Before 1.5.0 it counted the rows it had reached rather
than the rows it changed, so a run could report `recovered: 500` against a queue that had not moved
by a single row. Treat the number as an instruction to re-check the depth, not as a receipt: a row
another worker took in the meantime is now reported as what it is instead of padding the total.

## Metrics

`GET /metrics` exposes Prometheus metrics (unauthenticated; restrict at your ingress). All series
are prefixed `agledger_`. The ones worth alerting on:

| Metric | Watch for |
|---|---|
| `agledger_vault_integrity_check_results_total{result="broken"}` | Any increase - a chain failed periodic verification |
| `agledger_db_pool_waiting_connections` | Sustained nonzero - pool saturation |
| `agledger_pgboss_queue_size{queue=~".*-dlq",state="total"}` | Growth - jobs dead-lettering into a DLQ |
| `agledger_pgboss_queue_size{state="queued"}` | Sustained growth on any queue: a worker is down or cannot keep up. Every queue the Server runs reports here, federation included |
| `agledger_pg_listener_reconnect_failures_total` | Increase - cross-replica cache coherence degraded |
| `agledger_vault_checkpoint_skipped_broken_total` | Increase - a record went un-anchored |
| `agledger_outbound_ssrf_blocked_total` | Increase - outbound calls hitting the SSRF guard |
| `agledger_federation_zero_row_oldest_candidate_age_seconds` | Climbing toward the recovery horizon - crash-orphaned records are aging out unrecovered. Federated deployments only; see [Federation delivery and recovery](#federation-delivery-and-recovery) |

```bash
curl -s "$AGLEDGER_API_URL/metrics" | grep agledger_vault_integrity_check_results_total
```

Note: `agledger_partition_runway_days` (next section) is exposed by the **worker** process, not the
API process. Scrape both processes, not just the API. The federation series are worker-only too.

You do not have to write those alerts yourself. `monitoring/alerts/agledger.rules.yml` in the
[install repository](https://github.com/agledger-ai/install) ships 14 rules covering silent drops,
chain integrity, federation delivery, and availability, and the bundled Prometheus loads them
already. No Alertmanager is bundled and no routing is configured, because receivers and escalation
policy are yours to decide; until you point them somewhere the rules evaluate on Prometheus' own
`/alerts` page, and each carries a `severity` label of `critical` or `warning` as the routing hook.

Three Grafana dashboards auto-provision with `install.sh --with-monitoring`: an overview (traffic,
chain throughput, saturation), data-integrity surveillance, and a silent-drop board with a panel per
fire-and-forget path that swallows its error. `monitoring/README.md` describes all three, including
how to import them into your own Grafana instead of the bundled one.

## Signing-key rotation

Rotation is the load-bearing day-2 task. The guarantee that makes it safe:

> **Rotating the signing key never breaks verification of already-signed records.** Retired keys
> stay in the published registry, so a record signed under an old key still verifies after any
> number of rotations. No re-signing, no downtime.

### How rotation works

The engine signs with the key in `VAULT_SIGNING_KEY`. To rotate: generate a new key, set it as
`VAULT_SIGNING_KEY`, move the prior key to `VAULT_SIGNING_KEY_PREVIOUS`, and restart the process.
On boot the engine retires the old active key in the registry and promotes the new one:

```
INFO: Retired previous active signing key during bootstrap
INFO: Bootstrapped active signing key into registry
```

`POST /v1/admin/vault/signing-keys/rotate` (platform key) reconciles the registry to the
env-configured key. In a normally-booted process the boot step has already promoted it, so the
endpoint reports `already_active` - use it to confirm, not to mint:

```json
{ "previousKeyId": null, "newKeyId": "c4dd3e20388b594d", "status": "already_active" }
```

After rotation, both keys appear at `GET /v1/verification-keys` - the new one active, the prior one
retired but still resolvable:

```bash
curl -s "$AGLEDGER_API_URL/v1/verification-keys"
```

```
6a639248683aab56 | active  | activated 2026-05-26 | retired null
affc2b9bfb22144e | retired | activated 2026-05-26 | retired 2026-05-26
```

### Proving the guarantee

Records signed before the rotation must still verify. Dump the vault and verify it offline (see the
[audit runbook](/docs/operations/audit/) for the full handoff) - the dump now carries two signing keys
and entries signed by both:

```bash
./scripts/vault-dump.sh ./dump      # "vault_signing_keys": 2
python3 verify-dump.py ./dump       # stock RFC 9052 / Ed25519 check — see the audit runbook
```

```
[PASS] stock-library offline verification
  audit_vault entries : 5
  signatures verified : 5
  failures            : 0
  signing keys        : 2
```

Zero failures across records signed by the retired key and the active key - retired keys travel in
the dump, so the verifier resolves whichever key signed each entry. That is the guarantee,
demonstrated end to end.

## Partition maintenance

Several high-volume tables are range-partitioned by month, each with a DEFAULT catch-all partition
so a write never fails for lack of a partition. The worker pre-creates upcoming partitions and
exposes runway as a gauge (`agledger_partition_runway_days`). You can also query the source
function directly:

```bash
psql "$DATABASE_URL" -c "SELECT table_name, runway_days, default_rows FROM partition_runway();"
```

```
     table_name     | runway_days | default_rows
--------------------+-------------+--------------
 audit_vault        |         570 |            0
 events             |         570 |            0
 webhook_deliveries |         570 |            0
 system_audit_log   |          83 |            0
```

`runway_days` is days until the latest pre-created partition is reached; `default_rows` should stay
`0` - a nonzero value means writes are landing in the DEFAULT partition and the worker is falling
behind. Alert on low `runway_days` and on `default_rows > 0`.

## Config-as-code hot reload

If you run with `PROVISIONING_CONFIG_PATH` set, orgs, agents, webhooks, and contract schemas are
declared in YAML and reconciled on every boot - see the
[provisioning runbook](/docs/install/provisioning/) for the directory layout. Reload changes without
a restart via `SIGHUP` or `POST /v1/admin/provisioning/reload` (platform key). Check current state
first:

```bash
curl -s -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/provisioning/status"
```

```json
{"configured":true,"configPath":"/etc/agledger/provisioning","dryRun":false,"prune":false,"lastReloadAt":"2026-06-09T15:59:53.404Z","managed":{"orgs":1,"agents":2,"webhooks":0,"schemas":2},"loadErrors":["webhooks/acme.yaml: Environment variable ACME_WEBHOOK_SECRET is not set and has no default"]}
```

```bash
curl -s -X POST -H "Authorization: Bearer $AGLEDGER_PLATFORM_KEY" "$AGLEDGER_API_URL/v1/admin/provisioning/reload"
```

```json
{
  "orgs":    { "created": 0, "updated": 1, "pruned": 0 },
  "agents":  { "created": 0, "updated": 2, "pruned": 0 },
  "schemas": { "created": 0, "updated": 2, "pruned": 0 },
  "apiKeys": { "created": 0, "skipped": 3, "generated": [] },
  "errors": [
    { "resource": "config", "error": "webhooks/acme.yaml: Environment variable ACME_WEBHOOK_SECRET is not set and has no default" }
  ]
}
```

Reload is idempotent: unchanged resources count as `updated`, existing keys as `skipped`. It is also
fail-open - a single invalid file (here, an unset `${ACME_WEBHOOK_SECRET}` substitution in a webhook
file of your own) is reported in `errors[]` while every valid resource still applies. Newly minted
keys appear in `apiKeys.generated[]` with their raw value exactly once, in that response body -
capture them then.

Fail-open is the shape to watch. A file that fails to parse is skipped whole: its resources are
silently absent, the rest of the reconcile succeeds, and the Server comes up healthy. `status`
re-reads the directory on every call and lists every unloadable file in `loadErrors[]`, so that
field, not the boot log, is the durable signal. The `agledger_provisioning_errors` gauge (labels
`stage="load"` / `stage="reconcile"`) carries the same state for alerting and stays non-zero until a
clean reconcile clears it.

A bare `${VAR}` with no default is the usual cause: it is a hard parse error when the variable is
unset. Use `${VAR:-default}` wherever a sensible default exists, and inject real secrets through the
pod environment (`extraEnv` / `extraEnvFrom` on the chart).

## Version upgrades

Upgrade with `scripts/upgrade.sh` (from the
[`agledger-ai/install`](https://github.com/agledger-ai/install) repository - see the
[install runbook](/docs/install/) for the full procedure and the air-gapped path). Migrations run
automatically and are advisory-locked and checksum-verified, so a migration runs once and only once
even across replicas, and a tampered or reordered migration is refused rather than applied. For
diagnostics to attach to a support request, `scripts/support-bundle.sh` collects health, config
(secrets redacted), and recent operational events.

Restart the process after swapping the image. Already-signed records verify across versions - the
chain format is stable and historical keys resolve.

On an external database, the upgrade checks the runtime role twice: before the pre-upgrade backup,
and again after migrations. That role can stop being able to serve without the install changing. A
credential-rotation policy that drops and recreates it brings it back without its `agledger_app`
membership, because role membership does not survive a `DROP ROLE`. Left unchecked, the first thing
to fail is `pg_dump` during the backup, which reports the table it was refused and prints its whole
`LOCK TABLE` statement without naming the role.

Each check names the role and the exact `GRANT`. Stopping at the first one costs nothing: no backup
has been taken, no migration has run, and neither the version nor the image digest in `compose/.env`
has moved, so the running install is serving exactly as it was. Stopping at the second one leaves
the upgrade part-done, which it says: the migrations are applied, the worker stays stopped until the
upgrade finishes, and the previous version is still serving against the new schema. Either way,
applying the grant and re-running finishes it, and migrations already applied are skipped.

### v1.2.0: migration 002 and the PostgreSQL 18 upgrade order

v1.2.0 ships one schema migration. It adds the `record_types` column behind
[type-segregated webhook subscriptions](/docs/guides/webhooks/), repairs two indexes, and adopts
PostgreSQL 18's native `uuidv7()` where the PG17 polyfill is still in place. It runs automatically
under the same advisory-lock, checksum-verified runner as every migration; on its own it needs no
operator action.

If you are also moving from PostgreSQL 17 to 18 (see the
[in-place upgrade procedure](/docs/install/aws/)), the order matters:

- **Upgrade the database first, then deploy v1.2.0.** The migration detects native `uuidv7()`,
  re-points every column default still bound to the PG17 polyfill (`public.uuidv7`), and drops the
  polyfill. Nothing to do by hand.
- **If you deploy v1.2.0 first and upgrade the database later**, the migration has already run (a
  migration runs exactly once) and the polyfill stays pinned. Re-point the defaults yourself:

```sql
-- list every column default still bound to the polyfill
SELECT d.adrelid::regclass AS tbl, a.attname AS col
FROM pg_depend pd
JOIN pg_attrdef d ON d.oid = pd.objid AND pd.classid = 'pg_attrdef'::regclass
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE pd.refobjid = 'public.uuidv7()'::regprocedure;

-- for each row returned:
ALTER TABLE <tbl> ALTER COLUMN <col> SET DEFAULT uuidv7();

-- then, once nothing depends on it:
DROP FUNCTION public.uuidv7();
```

The [AWS install guide](/docs/install/aws/) carries the same remediation as a single idempotent
`DO` block you can run instead of the per-column form.

Running on the polyfill is not a correctness problem - it mints spec-compliant UUIDv7 values -
so treat this as tidy-up: native `uuidv7()` is a modest bulk-write win, and the cleanup can wait
for a maintenance window.

### v1.3.4: migration 003

v1.3.4 ships the first new migration since v1.2.0, bringing the set to three. It is additive and
needs no operator action:

- A nullable `total_records` column on `compliance_exports`. Compliance exports cap at 10,000
  records; this stores the pre-cap match count so the status re-read and the downloaded packet can
  both say a result was capped, rather than presenting a truncated slice as the whole org. Rows
  written before the column exists read as `NULL`, which readers treat as not-truncated.
- A DELETE and UPDATE guard on `vault_checkpoints`, matching the one `audit_vault` already carried.
  Checkpoints are the out-of-band high-water-mark that makes terminal truncation detectable at all,
  because a chain truncated from the end still hash-links cleanly. Leaving the witness deletable
  while the ledger was immutable inverted that protection.

The upgrade is still a rolling image bump, but it now runs a schema step where v1.3.2 and v1.3.3
did not. A fresh install applies all three in one Job and reports `Complete 1/1`; re-running
against a database already at that head applies nothing and reports a count of `0`.

If you have automation that deletes or rewrites `vault_checkpoints` rows, it will start failing
against the new guard. That is the intended effect, but it is the one behavior change here that
can surface as a broken job rather than a silent no-op.

## Request timeouts

`CONNECTION_TIMEOUT_MS` (default 60000) is the socket inactivity timeout: how long a connection
may go without bytes moving before the server destroys it.

The ordering between the three timeout knobs is what matters. Keep it above `HANDLER_TIMEOUT_MS`
(default 30000) and below `KEEP_ALIVE_TIMEOUT_MS` (default 72000). If the socket timeout fires
first, a slow request is severed underneath the handler instead of failing through the route, and
the failure mode is unpleasant: on a bulk write the batch commits, the client sees a dead socket
with no response, and it has no way to tell a committed batch from a lost one.

**Recovering from a severed socket:** retry the request with the same `Idempotency-Key`. A commit
that already landed is returned rather than repeated.

Two signals tell you the value is too low for your traffic, and you need both because a severed
socket never reaches the normal response path:

- the `agledger_requests_timed_out_total` counter
- a `request severed by socket inactivity timeout` warning in the API log

Raise `CONNECTION_TIMEOUT_MS` if you see them on legitimately slow bulk work, keeping it under the
keep-alive value.

## Federation delivery and recovery

Skip this section if you run a single Server with no peers. Nothing here applies until federation
is configured, and the recovery sweeps below short-circuit before they touch the records table
when a Server has no active peers.

### Is this peer reachable?

`GET /federation/v1/admin/peers` answers it. Read `lastDeliveryAt` (the last outbound message
that got a 2xx), `consecutiveDeliveryFailures` (reset to 0 on a success) and `lastDeliveryError`
(cleared on the next success). A frozen `lastDeliveryAt` with a climbing failure count and a
transport error is a peer that has gone away, with retries still in flight until the job
dead-letters.

Two fields on the same object do **not** answer it, and it is worth knowing why:

- `status` is registration state (`active` / `suspended` / `revoked`), not reachability. A peer
  you have never successfully reached still reads `active`.
- `lastSyncAt` is agent-directory sync. V1 federation is push-only, so it moves when the peer
  pushes its directory and not when you deliver to it. A peer that has taken hundreds of
  deliveries can legitimately show `lastSyncAt: null`.

For a count rather than a list, `GET /v1/admin/ops-summary` reports `federation.peers` with the
registration counts and, partitioning the active ones, `delivering` / `failing` / `neverDelivered`.
That last split is the one a dashboard needs: `active` alone reads the same for a peer taking every
delivery and a peer that has never taken one.

For the backlog behind those deliveries, watch `agledger_pgboss_queue_size{queue="federation-outbound"}`
or read `GET /v1/admin/system-health`, which reports the same queues as JSON, and whose `status`
goes `degraded` once those retries give up and dead-letter.

### Expect peers to lag, and size for it

`POST /v1/records` returns as soon as the record is notarized locally; peers receive their signed
copy afterwards. Your local chain is complete and verifiable the entire time a peer is behind, so
peer lag is a delivery concern, never an integrity one.

The outbound worker drains at a per-pod ceiling of `FEDERATION_OUTBOUND_CONCURRENCY *
FEDERATION_OUTBOUND_BATCH_SIZE / FEDERATION_OUTBOUND_POLLING_INTERVAL_SECONDS`. On the v1.3.4
defaults that is `16 * 4 / 1.0` = **64 legs/sec/pod**, up from 16 in v1.3.3. A *leg* is one record
to one peer, so a record shared to two peers costs two legs. Notarizing 4,100 shared records
against two peers queues 8,200 legs and puts peers roughly two minutes behind. Ingest is faster
than drain by design, so a bulk share always builds a queue.

**`FEDERATION_OUTBOUND_BATCH_SIZE` now defaults to 4 rather than 1.** This is the one throughput
change in v1.3.4 that takes effect without you setting anything. If you had pinned the value
explicitly, your setting still wins and you keep the old rate.

**Raise batch size first** if you need more throughput. It widens the fetch each worker loop
already issues, so it lifts the ceiling without adding database round-trips. Concurrency and
polling interval are the expensive pair and interact super-additively on database CPU: each in
isolation costs roughly 30 to 40 percent on `POST /v1/records` p50/p95, but together at
`(32 / 0.5)` they cost about +110 percent p50 against the `(16 / 1.0)` baseline. Re-measure
record-creation latency after touching any of the three, and prefer scaling out worker pods over
cranking a single pod far past the defaults.

Same-`(peer, record)` ordering is preserved regardless of these settings, so raising throughput
never reorders state transitions for a given record.

### The recovery sweep, and the window that abandons records

Outbound work is enqueued after the record's state change commits. A crash in that gap can leave a
committed terminal record with no federation work queued at all. A sweep runs every two minutes on
the **worker** process to find those and re-drive them through the normal publish path: same share
gate, same peer set, same idempotency, so recovery can never deliver something the live path would
not have.

Two knobs size it, and they are a pair:

| Knob | Default | What it controls |
|---|---|---|
| `AGLEDGER_FEDERATION_ZERO_ROW_HORIZON_MINUTES` | `360` (6h) | How far back the sweep looks |
| `AGLEDGER_FEDERATION_ZERO_ROW_BATCH_SIZE` | `50` | Records re-driven per cycle (~25/min) |

**The horizon is an abandonment boundary, not just a scan bound.** A record whose `updated_at`
falls outside the window is never recovered. Six hours covers a crash gap with wide margin, since
the normal case recovers within a sweep interval or two. Widen it only if this Server can be down
longer than that.

Each cycle takes the oldest candidates first, which is the only order that cannot starve the
record closest to aging out.

**The one number to alert on** is `agledger_federation_zero_row_oldest_candidate_age_seconds`, the
age of the head of that queue. Flat and low is healthy. Climbing toward the horizon means the
candidate pool is refilling faster than one batch drains, which is the only remaining way a
genuinely orphaned record ages out unrecovered. Page well before the horizon.

**Which knob to turn** when that age climbs: raise `AGLEDGER_FEDERATION_ZERO_ROW_BATCH_SIZE`.
Widening the horizon in that state makes it worse, because it adds candidates to a pool that is
already draining too slowly. The horizon is the knob for a different problem, a Server that is
legitimately offline longer than six hours.

### Records received from peers

A record this Server received from a peer is a read-only view of a row the originating Server owns,
and only that Server fans it out. As of v1.3.4 such records are never recovery candidates. Before
that they were, on the default `AGLEDGER_DEFAULT_SHARE=true`, which produced redundant deliveries
back to the origin and permanently-rejected deliveries to third peers, growing by one for every
projection a Server had ever received.

This matters operationally because a busy receiving Server accumulates those records without
bound. `agledger_federation_zero_row_projections_excluded` reports how many were held out of the
candidate pool on the last cycle, capped at the batch size. **Non-zero is the healthy reading** on
any Server receiving federation traffic; it is the exclusion doing its job, not a problem.

Its companion `agledger_federation_outbound_projection_skipped_total` is a last-resort guard
further down the publish path and is expected to stay at **0** permanently. Treat any increase
there as a regression to report, not as confirmation that the exclusion is working.

---

*Updated for API v1.5.0 on 2026-08-21 for the federation DLQ `recovered` count, read from the
shipped fix rather than re-measured here. The rest of this document carries the earlier stamps
below.*

*Updated for API v1.4.0 on 2026-08-09: the monitoring section now points at the 14 alert rules and
three Grafana dashboards that ship with the install rather than leaving operators to write their
own, and the provisioning section documents `loadErrors[]` on `GET /v1/admin/provisioning/status`
plus the `agledger_provisioning_errors` gauge, which are the durable signals for a fail-open config
load. Not re-run end to end against a live v1.4.0 install.*

*Validated against API v1.2.0 on 2026-07-06 (Developer Edition, Docker Compose: health/readiness,
`system-health`, `/metrics`, `partition_runway()`, the `signing-keys/rotate` endpoint
(`already_active`), and provisioning status + reload - all re-run live). The post-rotation
two-key `/v1/verification-keys` listing and the two-key offline-verify proof are carried over from
the v0.25.4 rotation run (a full key swap was not re-performed this release); rotation behavior is
version-stable and the no-break-on-verify guarantee holds across versions. The migration-002 and
PostgreSQL-18 upgrade-ordering notes reflect the v1.2.0 migration as shipped.*

*Reviewed for API v1.3.4 on 2026-08-03: unchanged. Spot-checked live on a v1.3.4 install:
`GET /v1/admin/system-health` returns the documented summary (status, uptime, database latency and
pool, per-queue depths), `GET /metrics` exposes the `agledger_*` series including
`agledger_vault_integrity_check_results_total`, and a fresh install reports 3/3 migrations applied.
This page's description of `signing-keys/rotate` as reconciling the registry is the accurate one;
the [key-compromise runbook](/docs/operations/key-compromise/) matches it.*

*Reviewed for API v1.3.4 on 2026-07-27: 1.3.4 ships migration 003, the first new migration since
v1.2.0, documented under "v1.3.4: migration 003". It is additive and needs no operator action, but
the upgrade now runs a schema step. Adds `CONNECTION_TIMEOUT_MS` under "Request timeouts", and a
"Federation delivery and recovery" section for the outbound-throughput default change (per-pod
drain 16 to 64 legs/sec, the one change that takes effect without operator action), the two new
recovery-sizing knobs, and the series to alert on. Federation content applies only to multi-Server
deployments. Rotation, provisioning, and the health surfaces are unchanged.*

*Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 ships no new migration (the set stays at
`001_consolidated.sql` + `002_webhook_record_types.sql`), so the upgrade remains a rolling image
bump with no schema step. Day-2 operational surfaces are unchanged.*

*Reviewed for API v1.3.2 on 2026-07-13: v1.3.2 ships no new migration (the migration set stays at
`001_consolidated.sql` + `002_webhook_record_types.sql`), so the upgrade is a rolling image bump
with no schema step. Day-2 operational surfaces are unchanged.*