> Markdown version of https://agledger.ai/docs/operations/high-availability/
> Full index of this site for AI assistants: https://agledger.ai/llms.txt

# High availability

An AGLedger deployment has three tiers with three different availability stories:

| Tier | State | HA approach |
|---|---|---|
| API | Stateless | Scale replicas; one env footgun (`RATE_LIMIT_STORE`) |
| Worker | Stateless (queue in Postgres) | Scale replicas; no special handling |
| PostgreSQL | All of it | Managed failover (Aurora, RDS Multi-AZ, Patroni) |

The chain lives entirely in PostgreSQL. Application pods hold no durable state - every
tamper-evidence guarantee survives any pod being killed at any moment, because chain appends are
single transactions serialized per record by a database advisory lock. There is no HA mode to turn
on in the application; there is a deployment shape to configure.

## API tier

API pods are stateless and scale horizontally. Three changes from the single-node default, all in
the chart's values:

```yaml
api:
  replicaCount: 2          # or enable the HPA
  strategy:
    type: RollingUpdate    # default is Recreate, sized for single-node clusters

extraEnv:                  # required when running more than one API replica
  - name: RATE_LIMIT_STORE
    value: postgresql
```

The default rate-limit store is in-process memory, which is correct for one replica and silently
wrong for N - each replica enforces its own counters, multiplying every effective limit by N. The
PostgreSQL store gives cluster-wide limits. (If the store cannot reach a database pool it falls
back to in-memory rather than failing requests, so set it and confirm rather than assume.)

Everything else is already multi-replica-safe: signing-key and discovery caches invalidate across
replicas via Postgres LISTEN/NOTIFY, and migrations cannot race. In the chart, migrations run as a
pre-install/pre-upgrade Job - one runner per release, not one per pod. The optional startup runner
(`RUN_MIGRATIONS`, off by default) takes a global advisory lock and verifies migration checksums;
a pod that loses the lock exits and restarts rather than running migrations twice.

The chart ships an HPA, a PodDisruptionBudget, and a commented pod-anti-affinity example (spread
replicas across nodes) for the API deployment - see `values.yaml`.

### Rolling restarts behind a load balancer

Endpoint removal and load-balancer target deregistration are asynchronous. Without a delay, an old
pod begins shutting down while the balancer is still routing to it, which shows up as a burst of
timeouts and 502/503s that `kubectl rollout status` reports as a clean rollout. Two chart values
added in v1.3.4 close the two halves of that race:

```yaml
api:
  preStopSleepSeconds: 15            # keep serving while deregistration propagates
  terminationGracePeriodSeconds: 45  # must exceed preStop + in-flight drain (~25s) + headroom
```

The preStop sleep keeps the container fully in service until the balancer stops sending it traffic;
SIGTERM and the in-flight drain follow. It renders only under `strategy.type: RollingUpdate`, since
`Recreate` has no traffic handoff to drain. Set `preStopSleepSeconds: 0` to disable. If the grace
period does not cover the sleep plus the drain, the kubelet SIGKILLs the pod mid-drain and you lose
the in-flight requests you were trying to protect.

That covers shutdown. The startup half is load-balancer specific: a new pod can be counted Ready as
soon as its readinessProbe passes, even though the balancer is still registering it, so the old pod
is torn down before the new one receives traffic. On AWS with `target-type: ip`, label the release
namespace so the Load Balancer Controller injects its pod readiness gate:

```bash
kubectl label namespace <ns> elbv2.k8s.aws/pod-readiness-gate-inject=enabled
```

Pair it with a shorter deregistration delay so old targets drain promptly instead of holding the
300s default:

```yaml
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
```

### Scheduling the migration pod

On the external-database path the migration Job is a pre-install/pre-upgrade hook, so a migration
pod that cannot schedule fails the whole install or upgrade. If your API pods are pinned to a node
class (architecture, taints), set `migrate.nodeSelector` / `migrate.tolerations` /
`migrate.affinity` to match. Each is empty by default and falls back to the matching `api.*`
value, so an install that pins only the API is already consistent.

## Worker tier

Worker replicas scale the same way (`worker.replicaCount`), with no special configuration:

- **Queue jobs** (gate evaluation, webhook delivery, federation outbound) are claimed from
  Postgres with `SKIP LOCKED` semantics - each job is processed by exactly one replica.
- **Scheduled maintenance** (expiry sweeps, vault checkpoints - which include anchor uploads - 
  and recovery sweeps) is cluster-singleton: pg-boss claims each cron tick atomically in the
  database, so one replica fires it regardless of replica count. We tested this directly - three
  concurrent workers, one schedule, exactly one job per tick.

A single worker replica is fully functional and is the right default; add replicas for queue
throughput or for zero-gap coverage across pod restarts. The chart ships a worker HPA and PDB.

## Database tier

PostgreSQL is the single stateful component, so the availability of your deployment **is** the
availability of your database. Use a managed failover-capable Postgres: Aurora, RDS Multi-AZ, or
an equivalent (Patroni/CloudNativePG for self-managed clusters; any failover-capable PostgreSQL
17+ works - nothing here is AWS-specific).

During a failover, in-flight writes fail and clients receive errors until the new primary is up;
the application's connection pool recovers on its own as connections are checked out. Chain
integrity is unaffected by design - an append either committed as one transaction or it did not.
There is no partial chain state to repair, and verification after a failover requires nothing
beyond the normal [audit runbook](/docs/operations/audit/).

The **bundled Postgres** (the Compose overlay `docker-compose.postgres.yml`, or the chart's
`postgres.bundled` deployment) is a single-node convenience, explicitly not an HA database.
Running on an external database is the Enterprise license boundary: the Developer Edition license
covers production use on the bundled single-node Postgres, and pointing the Server at Aurora or
any other external Postgres is Enterprise scope, which is why high availability is an
Enterprise-tier line item.

That boundary is real, and it is contractual rather than technical. What the license buys is the
right to run that way and the written terms behind it, not a capability, because every feature is
already present in every build. So the software does not gate on it: a Server on an external
database keeps working and logs a persistent licensing notice rather than degrading or refusing
to start. Read that as how we choose to treat operators, not as the boundary being optional.

External anchoring is unaffected by all of this: anchors are written to object storage outside
the database and remain valid ground truth across any failover.

## The supported HA shape, summarized

- API: `replicaCount: 2+`, `RollingUpdate`, `RATE_LIMIT_STORE=postgresql`, anti-affinity across
  nodes.
- Worker: `replicaCount: 1` default, `2+` for throughput - safe either way.
- PostgreSQL: managed failover (Enterprise license), TLS required in production.
- Anchoring enabled, to a separately-administered bucket.

What is **not** supported today: multi-region active-active against one logical chain, and any
deployment shape where two Servers write the same database schema independently. One Server, one
database; scale the stateless tiers; let the database layer own failover. For multi-site
topologies, run sovereign Servers per site and connect them with [federation](/federation/).