High availability

An AGLedger deployment has three tiers with three different availability stories:

| Tier | State | HA approach | |---|---|---| | API | Stateless | Scale replicas; one env footgun (RATE_LIMIT_STORE) | | Worker | Stateless (queue in Postgres) | Scale replicas; no special handling | | PostgreSQL | All of it | Managed failover (Aurora, RDS Multi-AZ, Patroni) |

The chain lives entirely in PostgreSQL. Application pods hold no durable state - every tamper-evidence guarantee survives any pod being killed at any moment, because chain appends are single transactions serialized per record by a database advisory lock. There is no HA mode to turn on in the application; there is a deployment shape to configure.

API tier

API pods are stateless and scale horizontally. Three changes from the single-node default, all in the chart's values:

api:
  replicaCount: 2          # or enable the HPA
  strategy:
    type: RollingUpdate    # default is Recreate, sized for single-node clusters

extraEnv:                  # required when running more than one API replica
  - name: RATE_LIMIT_STORE
    value: postgresql

The default rate-limit store is in-process memory, which is correct for one replica and silently wrong for N - each replica enforces its own counters, multiplying every effective limit by N. The PostgreSQL store gives cluster-wide limits. (If the store cannot reach a database pool it falls back to in-memory rather than failing requests, so set it and confirm rather than assume.)

Everything else is already multi-replica-safe: signing-key and discovery caches invalidate across replicas via Postgres LISTEN/NOTIFY, and migrations cannot race. In the chart, migrations run as a pre-install/pre-upgrade Job - one runner per release, not one per pod. The optional startup runner (RUN_MIGRATIONS, off by default) takes a global advisory lock and verifies migration checksums; a pod that loses the lock exits and restarts rather than running migrations twice.

The chart ships an HPA, a PodDisruptionBudget, and a commented pod-anti-affinity example (spread replicas across nodes) for the API deployment - see values.yaml.

Rolling restarts behind a load balancer

Endpoint removal and load-balancer target deregistration are asynchronous. Without a delay, an old pod begins shutting down while the balancer is still routing to it, which shows up as a burst of timeouts and 502/503s that kubectl rollout status reports as a clean rollout. Two chart values added in v1.3.4 close the two halves of that race:

api:
  preStopSleepSeconds: 15            # keep serving while deregistration propagates
  terminationGracePeriodSeconds: 45  # must exceed preStop + in-flight drain (~25s) + headroom

The preStop sleep keeps the container fully in service until the balancer stops sending it traffic; SIGTERM and the in-flight drain follow. It renders only under strategy.type: RollingUpdate, since Recreate has no traffic handoff to drain. Set preStopSleepSeconds: 0 to disable. If the grace period does not cover the sleep plus the drain, the kubelet SIGKILLs the pod mid-drain and you lose the in-flight requests you were trying to protect.

That covers shutdown. The startup half is load-balancer specific: a new pod can be counted Ready as soon as its readinessProbe passes, even though the balancer is still registering it, so the old pod is torn down before the new one receives traffic. On AWS with target-type: ip, label the release namespace so the Load Balancer Controller injects its pod readiness gate:

kubectl label namespace <ns> elbv2.k8s.aws/pod-readiness-gate-inject=enabled

Pair it with a shorter deregistration delay so old targets drain promptly instead of holding the 300s default:

alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30

Scheduling the migration pod

On the external-database path the migration Job is a pre-install/pre-upgrade hook, so a migration pod that cannot schedule fails the whole install or upgrade. If your API pods are pinned to a node class (architecture, taints), set migration.nodeSelector / migration.tolerations / migration.affinity to match. Each is empty by default and falls back to the matching api.* value, so an install that pins only the API is already consistent.

Worker tier

Worker replicas scale the same way (worker.replicaCount), with no special configuration:

• Queue jobs (gate evaluation, webhook delivery, federation outbound) are claimed from Postgres with SKIP LOCKED semantics - each job is processed by exactly one replica.
• Scheduled maintenance (expiry sweeps, vault checkpoints - which include anchor uploads - and recovery sweeps) is cluster-singleton: pg-boss claims each cron tick atomically in the database, so one replica fires it regardless of replica count. We tested this directly - three concurrent workers, one schedule, exactly one job per tick.

A single worker replica is fully functional and is the right default; add replicas for queue throughput or for zero-gap coverage across pod restarts. The chart ships a worker HPA and PDB.

Database tier

PostgreSQL is the single stateful component, so the availability of your deployment is the availability of your database. Use a managed failover-capable Postgres: Aurora, RDS Multi-AZ, or an equivalent (Patroni/CloudNativePG for self-managed clusters; any failover-capable PostgreSQL 17+ works - nothing here is AWS-specific).

During a failover, in-flight writes fail and clients receive errors until the new primary is up; the application's connection pool recovers on its own as connections are checked out. Chain integrity is unaffected by design - an append either committed as one transaction or it did not. There is no partial chain state to repair, and verification after a failover requires nothing beyond the normal audit runbook.

The bundled Postgres (the Compose overlay docker-compose.postgres.yml, or the chart's postgres.bundled deployment) is a single-node convenience, explicitly not an HA database. Running on an external database is the Enterprise license boundary: the Developer Edition license covers production use on the bundled single-node Postgres, and pointing the Server at Aurora or any other external Postgres is Enterprise scope, which is why high availability is an Enterprise-tier line item.

That boundary is real, and it is contractual rather than technical. What the license buys is the right to run that way and the written terms behind it, not a capability, because every feature is already present in every build. So the software does not gate on it: a Server on an external database keeps working and logs a persistent licensing notice rather than degrading or refusing to start. Read that as how we choose to treat operators, not as the boundary being optional.

External anchoring is unaffected by all of this: anchors are written to object storage outside the database and remain valid ground truth across any failover.

The supported HA shape, summarized

• API: replicaCount: 2+, RollingUpdate, RATE_LIMIT_STORE=postgresql, anti-affinity across nodes.
• Worker: replicaCount: 1 default, 2+ for throughput - safe either way.
• PostgreSQL: managed failover (Enterprise license), TLS required in production.
• Anchoring enabled, to a separately-administered bucket.

What is not supported today: multi-region active-active against one logical chain, and any deployment shape where two Servers write the same database schema independently. One Server, one database; scale the stateless tiers; let the database layer own failover. For multi-site topologies, run sovereign Servers per site and connect them with federation.

Validated against API v1.3.4 on 2026-08-03 (Developer Edition, Docker Compose, fresh install from the pinned v1.3.4 release). The worker-scaling claim was re-run rather than carried forward: three concurrent worker containers against one database, sampled over five consecutive per-minute cron ticks on two independent schedules, produced exactly one job per tick per schedule (ten ticks, all count 1).

Validated against API v1.1.0 on 2026-06-10. Worker-scaling claim run-tested: three concurrent worker instances against one database produced exactly one scheduled job per cron tick.

Reviewed for API v1.3.4 on 2026-07-27: the topology is unchanged. 1.3.4 adds chart values for the two halves of the rolling-restart race behind a load balancer (api.preStopSleepSeconds, api.terminationGracePeriodSeconds), documented under "Rolling restarts behind a load balancer", and scheduling constraints for the migration pod (migration.nodeSelector / .tolerations / .affinity) that default to the matching api.* values.

Reviewed for API v1.3.3 on 2026-07-20: 1.3.3 (audit-vault chain-scan detection plus opt-in verdict per-actor signatures) changes no deployment topology; the one-Server-one-database model and worker-scaling behavior are unchanged.

Validated against API v1.4.0 on 2026-08-09 (Developer Edition, Docker Compose). The worker-scaling claim was re-run rather than carried forward, on a wider sample than previous passes: three concurrent worker containers against one database, sampled across two independent per-minute schedules over the full run. Every one of the 122 schedule ticks observed produced exactly one job, maximum count 1, which is the cluster-singleton behaviour this page rests on. The topology is otherwise unchanged in 1.4.0. One addition: OpenShift is now reachable from the documented install path via openshift.enabled, which drops the pod-level uid requests that collide with the assigned-uid model while keeping every container-level control in place. It changes no tier in the table above; the Kubernetes install guide carries the detail.