High availability

An AGLedger deployment has three tiers with three different availability stories:

The chain lives entirely in PostgreSQL. Application pods hold no durable state. No committed chain entry is lost or left half-written when a pod is killed at any moment, because chain appends are single transactions serialized per record by a database advisory lock. There is no HA mode to turn on in the application; there is a deployment shape to configure.

API tier

API pods are stateless and scale horizontally. Two changes from the single-node default, both in the chart's values:

api:
  replicaCount: 2          # or enable the HPA
  strategy:
    type: RollingUpdate    # default is Recreate, sized for single-node clusters

The rate-limit store follows from that. In-process memory is correct for one replica and silently wrong for N, because each replica enforces its own counters and every effective limit multiplies by N. The chart sets RATE_LIMIT_STORE=postgresql for you as soon as api.replicaCount is above 1 or the HPA is enabled, which gives cluster-wide limits. Override with config.rateLimitStore if you want per-replica counting on purpose. (If the store cannot reach a database pool it falls back to in-memory rather than failing requests, so confirm the rendered ConfigMap rather than assume.)

Everything else is already multi-replica-safe: signing-key and discovery caches invalidate across replicas via Postgres LISTEN/NOTIFY, and migrations cannot race. In the chart, migrations run as one Job per release, not one per pod: a pre-install/pre-upgrade hook on an external database, and an ordinary Job gated on the database being ready when the chart runs the bundled Postgres. Each api and worker process refuses to start against a database that has not applied every migration its image ships, naming the reason in its log. The optional startup runner (RUN_MIGRATIONS, off by default) takes a global advisory lock and verifies migration checksums; a pod that loses the lock exits and restarts rather than running migrations twice.

The chart ships an HPA, a PodDisruptionBudget, and a commented pod-anti-affinity example (spread replicas across nodes) for the API deployment; values.yaml carries all three.

RollingUpdate also means pods of two releases answer the same Service for the length of an upgrade. The schema is safe in that window; a security control the new release introduces binds only once the old pods are gone. Version upgrades covers what that binds and how to read it from connectedVersions.

Rolling restarts behind a load balancer

Endpoint removal and load-balancer target deregistration are asynchronous. Without a delay, an old pod begins shutting down while the balancer is still routing to it, which shows up as a burst of timeouts and 502/503s that kubectl rollout status reports as a clean rollout. Two chart values close the two halves of that race:

api:
  preStopSleepSeconds: 15            # keep serving while deregistration propagates
  terminationGracePeriodSeconds: 90  # chart default; floor is preStop + drain (~25s) + 5

45 is a floor the chart enforces, not a value to copy: a render below preStopSleepSeconds plus the 25-second in-flight drain plus 5 seconds of headroom is refused. The default is 90 because a pod that needs its whole grace period is a pod under load, and the only cost of unused margin is how long a rolling update waits on a replica that has already exited.

The preStop sleep keeps the container fully in service until the balancer stops sending it traffic; SIGTERM and the in-flight drain follow. It renders only under strategy.type: RollingUpdate, since Recreate has no in-service traffic to drain. Set preStopSleepSeconds: 0 to disable. If the grace period does not cover the sleep plus the drain, the kubelet SIGKILLs the pod mid-drain and you lose the in-flight requests you were trying to protect.

That covers shutdown. The startup half is load-balancer specific: a new pod can be counted Ready as soon as its readinessProbe passes, even though the balancer is still registering it, so the old pod is torn down before the new one receives traffic. On AWS with target-type: ip, label the release namespace so the Load Balancer Controller injects its pod readiness gate:

kubectl label namespace <ns> elbv2.k8s.aws/pod-readiness-gate-inject=enabled

Pair it with a shorter deregistration delay so old targets drain promptly instead of holding the 300s default:

alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30

Scheduling the migration pod

On the external-database path the migration Job is a pre-install/pre-upgrade hook, so a migration pod that cannot schedule fails the whole install or upgrade. If your API pods are pinned to a node class (architecture, taints), set migrate.nodeSelector / migrate.tolerations / migrate.affinity to match. Each is empty by default and falls back to the matching api.* value, so an install that pins only the API is already consistent.

Worker tier

Worker replicas scale the same way (worker.replicaCount):

• Queue jobs (gate evaluation, webhook delivery, federation outbound) are claimed from Postgres with SKIP LOCKED semantics, and delivery is at-least-once rather than exactly-once. A lease overrun re-queues a job while the first copy is still running, and pg-boss's own retry budget can redeliver one that failed partway, so a receiver can see the same job twice. The webhook delivery worker ships an idempotencyKey on every delivery for exactly this reason; a consumer that assumes exactly-once double-applies whatever the job carried.
• Scheduled maintenance (expiry sweeps, vault checkpoints including anchor uploads, and recovery sweeps) is cluster-singleton for the tick only: pg-boss claims each cron tick atomically in the database, so one replica fires it regardless of replica count (three concurrent workers, one schedule, exactly one job per tick). The job that tick enqueues is an ordinary job on a standard queue, where singletonKey does not deduplicate, so a lease overrun can re-queue it and two copies of the same sweep can run concurrently on a multi-replica worker. Every shipped sweep defends itself against that, claiming each row it touches under its own lock or predicate. A new periodic task has to do the same rather than assume the singleton tick makes it the only copy running.

Worker replicas beyond one add queue throughput; they do not add restart coverage. The worker Deployment defaults to strategy.type: Recreate, the same single-node default the API tier overrides above, so every worker pod terminates before any replacement starts and replicaCount: 2+ still has zero workers running for the length of a rollout. Set worker.strategy.type: RollingUpdate if pod-restart coverage is what you are scaling for. The worker HPA and PDB (worker.autoscaling.enabled, worker.pdb.enabled) are both off by default. A single replica is fully functional and is the right default.

Database tier

PostgreSQL is the single stateful component, so the availability of your deployment is the availability of your database. Use a managed failover-capable Postgres: Aurora, RDS Multi-AZ, or an equivalent (Patroni/CloudNativePG for self-managed clusters; any failover-capable PostgreSQL 17+ works; nothing here is AWS-specific).

During a failover, in-flight writes fail and clients receive 503 with the problem type /problems/database-unavailable, naming the dependency and carrying a recoveryHint, until the new primary is up. /health/ready (and its /readyz alias) answers 503 with a reason beginning database unavailable: for the same window. The application's connection pool recovers on its own as connections are checked out, and readiness and the data path return to 200 without a restart. Chain integrity is unaffected by design: an append either committed as one transaction or it did not. There is no partial chain state to repair, and verification after a failover requires nothing beyond the normal audit runbook.

A transaction-mode connection pooler (PgBouncer with pool_mode = transaction, RDS Proxy) in front of the database works for ordinary queries, but three things need a session that belongs to one caller: the LISTEN connection behind cross-replica cache invalidation, the session-level advisory locks, and migrations. Point DATABASE_URL at the pooler and DATABASE_URL_DIRECT (secrets.databaseUrlDirect on the chart) at PostgreSQL itself, and those three use the direct connection while everything else is multiplexed. A Server whose DATABASE_URL is a transaction-mode pooler refuses to start without it.

The bundled Postgres (the Compose overlay docker-compose.postgres.yml, or the chart's postgres.bundled deployment) is a single-node convenience, explicitly not an HA database. Running on an external database is the Enterprise license boundary: the Developer Edition license covers production use on the bundled single-node Postgres, and pointing the Server at Aurora or any other external Postgres is Enterprise scope, which is why high availability is an Enterprise-tier line item.

That boundary is real, and it is contractual rather than technical. What the license buys is the right to run that way and the written terms behind it, not a capability, because every feature is already present in every build. So the software does not gate on it: a Server on an external database keeps working rather than degrading or refusing to start. It logs a licensing warning every 24 hours, and GET /v1/admin/license reports the state in its notice object (kind unlicensed with no key, dev-external for a Developer Edition key); once the install is 45 days old, every successful /v1/admin/* response also carries a Warning header. Read that as how we choose to treat operators, not as the boundary being optional.

External anchoring is unaffected by all of this: anchors are written to object storage outside the database and remain valid ground truth across any failover.

The supported HA shape, summarized

• API: replicaCount: 2+, RollingUpdate, anti-affinity across nodes. The chart derives RATE_LIMIT_STORE=postgresql from the replica count.
• Worker: replicaCount: 1 default, 2+ for throughput; safe either way.
• PostgreSQL: managed failover (Enterprise license), TLS required in production.
• Anchoring enabled, to a separately-administered bucket.

What is not supported today: multi-region active-active against one logical chain, and any deployment shape where two Servers write the same database schema independently. One Server, one database; scale the stateless tiers; let the database layer own failover. For multi-site topologies, run sovereign Servers per site and connect them with federation.