Install AGLedger on Amazon EKS

This guide brings up a single AGLedger Server on Amazon EKS, reachable on a real hostname over TLS, using AWS-native services: an AWS Load Balancer Controller Application Load Balancer (ALB), an AWS Certificate Manager (ACM) certificate, and Amazon Aurora PostgreSQL as the database. It is the AWS companion to Install AGLedger on Kubernetes; read that first for the base model (signing key, readiness gate). Every command below is a literal transcript captured against a live Server reachable at https://agledger-aws.agledger.ai.

Live-validated against v1.4.0 on 2026-08-19 on a real Amazon EKS 1.36 + Amazon Aurora PostgreSQL 18 cluster (us-west-2), following this runbook literally end to end at the pinned sha256:75c9590f… index digest and chart 1.4.0. Everything below reproduced: the Aurora role and rds_superuser grant (the event trigger created without permission denied), helm install reporting STATUS: deployed / REVISION: 1, the migrate Job at Complete 1/1 in 7s applying four migrations on Aurora (previously only confirmed on bundled PostgreSQL), the ALB target reaching healthy in about two minutes with networkPolicy.enabled: false, and every cell of the readiness gate including the ACM subject/issuer strings, the HTTP→HTTPS 301, and /v1/verification-keys returning the keygen fingerprint. Beyond the gate, a record notarized against Aurora over sslmode=verify-full was exported and verified offline with the published @agledger/verify 1.5.1, pinned to that key id with --require-out-of-band-keys.

Two things in this guide were not exercised on that run: the Upgrading the Aurora engine section (the cluster was already on 18, and a major version cannot be downgraded in place to re-test it), and AWS licensing (no Marketplace entitlement on the validation account). Both remain as previously recorded. DNS was pointed at the ALB by host pinning rather than a Route 53 alias, so the ALB hostname read-back is validated and the record creation itself is not.

The runbook was live-validated against v1.3.4 on 2026-07-27 on a real Amazon EKS 1.36 + Amazon Aurora PostgreSQL 18.3 cluster (us-west-2), following this runbook literally end to end. Every command and every quoted output below reproduced: Aurora prep (role + rds_superuser), helm install at the pinned digest, the migrate Job reporting Complete 1/1 with three migrations applied, ALB wiring (networkPolicy.enabled: false, target-type: ip targets reaching healthy), and the full readiness gate (ACM cert subject/issuer, HTTP→HTTPS 301, verification-keys keyId == keygen fingerprint). Previously live-validated against v1.2.0 on 2026-07-09. For an in-place PostgreSQL 17→18 upgrade, see Upgrading the Aurora engine below.

Verified on Graviton (2026-07-27). The digest pinned below is the multi-arch index digest, and it was confirmed to resolve per-node on an arm64 (Graviton) node: pods report the index digest as their imageID while the runtime is genuinely arm64 (process.arch = arm64, aarch64 kernel, no emulation), migrations apply natively, and Ed25519 signatures produced on arm64 verify against the published offline verifier running on x86. If you pin a per-architecture child digest instead, you pin one architecture; pin the index.

Reviewed for API v1.4.0 on 2026-08-09: a fresh install applies four migrations (001_consolidated.sql, 002_webhook_record_types.sql, 003_export_cap_and_checkpoint_immutability.sql, 004_signing_agility.sql); the migrate Job still reports Complete 1/1, since that counts the Job, not the migrations. Confirmed on a live v1.4.0 install, on bundled PostgreSQL rather than Aurora.

Reviewed for API v1.3.4 on 2026-07-27: the runbook itself is unchanged, but two things in it move. A fresh install then applied three migrations; the migrate Job still reports Complete 1/1, since that counts the Job, not the migrations. And on the external-database path the migrate Job is a pre-install/pre-upgrade hook, so a migration pod that cannot schedule fails the whole install: if your API pods are pinned to a node class, set migrate.nodeSelector / .tolerations / .affinity, each of which defaults to the matching api.* value. Release images are also multi-arch (linux/amd64 and linux/arm64) as of v1.3.4; the digest pinned below is the index digest, which resolves per-node.

AWS prerequisites

An Amazon EKS cluster (1.27+) with kubectl and helm configured against it
The AWS Load Balancer Controller installed in the cluster - it provides the alb IngressClass and provisions the ALB from the Ingress resource. (Install via its Helm chart with an IAM Role for Service Accounts / IRSA, per the AWS docs.)
An ACM certificate in the same region as the ALB, covering your hostname (here a wildcard *.agledger.ai), status ISSUED
An Amazon Aurora PostgreSQL cluster (or Amazon RDS for PostgreSQL) reachable from the EKS node/pod security group on 5432
cosign 3.0+ to verify the release (see Install step 1 - same commands)

1. Prepare the Amazon Aurora database

Create a dedicated database and an application role. Run this from inside the VPC (Aurora is not publicly reachable) - for example a one-off psql pod connecting as the Aurora master user.

CREATE ROLE agledger_aws_app LOGIN PASSWORD '<app-password>';
GRANT agledger_aws_app TO agledger;          -- master must be a member to set ownership
CREATE DATABASE agledger_aws OWNER agledger_aws_app;

Migration privilege (important). The schema migration creates a PostgreSQL event trigger (agledger_block_audit_drop, which protects the audit chain). Event triggers require superuser; on Aurora / RDS that is the rds_superuser role. A plain database-owner role is not enough - the migration fails with permission denied to create event trigger. Grant the migration role rds_superuser:

GRANT rds_superuser TO agledger_aws_app;

For least-privilege role separation (a privileged role for migrations, a restricted role for the running Server), use the chart's secrets.databaseUrlMigrate for the migration DDL and a restricted database.externalUrl for the API and worker.

The connection string uses sslmode=verify-full. Under PostgreSQL semantics, sslmode=require encrypts but performs no certificate validation - only verify-ca/verify-full check the server certificate against a CA bundle (and verify-full also checks the hostname). The agledger image bundles the AWS RDS / Aurora root CA at /etc/ssl/certs/rds-global-bundle.pem; set config.nodeExtraCaCerts to that path and use verify-full so the bundle is actually consulted and the Server validates the Aurora server certificate.

2. Values for the AWS path

aws-values.yaml:

image:
  digest: "sha256:75c9590f353c063735728229f07f941fd063b1ab242caee900406ddc1eb5d20c"  # 1.4.0
database:
  poolMax: 20
config:
  externalUrl: "https://agledger-aws.agledger.ai"          # the Server's signed issuer identity
  nodeExtraCaCerts: "/etc/ssl/certs/rds-global-bundle.pem"  # bundled AWS RDS / Aurora CA
ingress:
  enabled: true
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-2:<acct>:certificate/<id>"
  hosts:
    - host: agledger-aws.agledger.ai
      paths: [{ path: /, pathType: Prefix }]
  tls: []   # TLS terminates at the ALB via the ACM cert above — no in-cluster TLS secret
networkPolicy:
  enabled: false   # see note below

NetworkPolicy on the ALB path. The chart's default NetworkPolicy admits ingress only from ingress-controller pods (ingress-nginx / traefik). With target-type: ip, the ALB sends traffic and health checks from VPC elastic network interfaces - not from a pod - so the default policy blocks the ALB and targets never go healthy. Set networkPolicy.enabled: false on the ALB path (or apply your own policy admitting the VPC CIDR on port 3000).

3. Install

Generate the vault signing key and create the platform key exactly as in Install (steps 2 and 4). Keep the Aurora URL and the signing key out of the values file, and out of the command line too: --set puts a value in helm's argv, where ps shows it to every other user on the machine for the length of the install. Both of these carry a secret, the database password and the private key every record is signed with, so pass them as files.

$ kubectl create namespace agledger-aws
$ umask 077
$ printf %s 'postgresql://agledger_aws_app:<pw>@<aurora-endpoint>:5432/agledger_aws?sslmode=verify-full' > db-url
$ printf %s '<vault-key>' > vault-key
$ helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \
    --version 1.4.0 --namespace agledger-aws \
    --values aws-values.yaml \
    --set-file database.externalUrl=db-url \
    --set-file secrets.vaultSigningKey=vault-key
NAME: agledger
STATUS: deployed
REVISION: 1
  API URL:  https://agledger-aws.agledger.ai

$ rm db-url vault-key

$ kubectl rollout status deploy/agledger-agledger-chart-api -n agledger-aws --timeout=180s
deployment "agledger-agledger-chart-api" successfully rolled out
$ kubectl get pods,jobs -n agledger-aws
pod/agledger-agledger-chart-api-...      1/1   Running
pod/agledger-agledger-chart-migrate-...  0/1   Completed
pod/agledger-agledger-chart-worker-...   1/1   Running
job.batch/agledger-agledger-chart-migrate   Complete   1/1   7s

4. Point DNS at the ALB

The Ingress provisions an ALB; read its hostname and create a DNS record for your host (CNAME, or a Route 53 alias) pointing at it.

$ kubectl get ingress -n agledger-aws
NAME                      CLASS   HOSTS                      ADDRESS
agledger-agledger-chart   alb     agledger-aws.agledger.ai   k8s-agledger-agledger-....us-west-2.elb.amazonaws.com

Wait for the ALB target to register as healthy:

$ aws elbv2 describe-target-health --target-group-arn <tg-arn> \
    --query 'TargetHealthDescriptions[].TargetHealth.State'
[ "healthy" ]

5. The readiness gate: named, TLS-terminated, signing

Reach the Server on its real hostname over HTTPS. The ALB serves the ACM certificate.

$ curl -s https://agledger-aws.agledger.ai/health
{"status":"ok","version":"1.4.0","timestamp":"..."}

$ echo | openssl s_client -connect agledger-aws.agledger.ai:443 \
    -servername agledger-aws.agledger.ai 2>/dev/null | openssl x509 -noout -subject -issuer
subject=CN = *.agledger.ai
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M04

$ curl -s -o /dev/null -w "HTTP %{http_code} -> %{redirect_url}\n" http://agledger-aws.agledger.ai/health
HTTP 301 -> https://agledger-aws.agledger.ai:443/health

The served certificate is issued by Amazon (ACM), and plain HTTP is redirected to HTTPS by the ssl-redirect annotation. Confirm the Server is signing with the key you generated - the keyId matches your vault-key fingerprint:

$ curl -s https://agledger-aws.agledger.ai/v1/verification-keys
{"data":[{"keyId":"c4ddafd6bf06f1ef","algorithm":"Ed25519","status":"active",…}],…}

(Response elided - the live response also carries per-key publicKey/publicKeyRaw/timestamps and top-level signature-format fields. The validation point is the keyId matching your fingerprint.)

The Server is healthy, reachable on its name over an ACM-issued certificate, backed by Aurora, and signing with your key.

Upgrading the Aurora engine (PostgreSQL 17 → 18)

Aurora supports an in-place major-version upgrade (set the cluster's engine version to 18.x with allow_major_version_upgrade). The Server's data, audit chain, and signatures carry through it unchanged - after a 17→18 in-place upgrade we re-verified the full vault (every record's hash-chain and Ed25519 / COSE_Sign1 envelope) with zero breakage, and the API self-recovered once the database returned (no pod restart) after the brief upgrade window. Plan for a short write outage during the upgrade and run it between workloads, not during one.

Adopting native uuidv7 after the upgrade. On PostgreSQL 17 the schema migration installs a small uuidv7() polyfill in the public schema (PostgreSQL gained a native uuidv7() in 18). A fresh 18 install never creates it. But an in-place upgrade does not adopt the native function automatically: the polyfill persists, and every table's id column default stays bound to it, so inserts keep using the polyfill.

From v1.2.0 this adoption is automatic - upgrade the database first, then deploy the release, and its migration re-points the polyfill-pinned column defaults to native uuidv7() and drops the polyfill on its own (see day-2 operations for the ordering rules). Run the block below only if you are on v1.1.0 or earlier, or you upgraded the database after v1.2.0's migration had already run. It is idempotent and self-discovering (it only acts when native uuidv7 exists and the polyfill is present, and finds the columns itself - there is no fixed list to maintain); run it once, as a superuser, against your application database after the upgrade:

-- Re-point every uuidv7() column default so it re-resolves to the native pg_catalog.uuidv7,
-- then remove the now-unused polyfill. No-op on a fresh 18 install or an already-remediated DB.
DO $$
DECLARE r record;
BEGIN
  -- only act if BOTH native and the public polyfill exist
  IF EXISTS (SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
             WHERE p.proname = 'uuidv7' AND n.nspname = 'pg_catalog')
     AND EXISTS (SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
                 WHERE p.proname = 'uuidv7' AND n.nspname = 'public') THEN
    FOR r IN
      SELECT n.nspname AS sch, c.relname AS tbl, a.attname AS col
      FROM pg_attrdef ad
      JOIN pg_class c       ON c.oid = ad.adrelid
      JOIN pg_namespace n   ON n.oid = c.relnamespace
      JOIN pg_attribute a   ON a.attrelid = ad.adrelid AND a.attnum = ad.adnum
      WHERE pg_get_expr(ad.adbin, ad.adrelid) ILIKE '%uuidv7%'
    LOOP
      EXECUTE format('ALTER TABLE %I.%I ALTER COLUMN %I SET DEFAULT uuidv7()', r.sch, r.tbl, r.col);
    END LOOP;
    DROP FUNCTION public.uuidv7();
  END IF;
END $$;

After it runs, new inserts use native uuidv7() and the public.uuidv7 function is gone. (Do not DROP FUNCTION public.uuidv7() on its own - the column defaults depend on it, so a bare drop fails and CASCADE would strip the defaults and break inserts. The re-point above is what makes the drop safe.)

AWS licensing

With no license the Server runs as Developer Edition. Subscribe to Enterprise on the AWS Marketplace listing, then set the Marketplace product ID (prod-…) and grant the Server's service account access to AWS License Manager with IRSA - marketplace.productId and marketplace.serviceAccountAnnotations in the chart values.

Air-gapped / private registries

To run from Amazon ECR instead of Docker Hub, mirror agledger/agledger:1.4.0 into ECR, set image.repository to the ECR repository and image.pullSecrets (or use the node role / IRSA for ECR pull), and pin image.digest.

Mirror the index, not one architecture. A plain docker pull + docker push copies only the image for the machine doing the mirroring, so the digest your registry then reports is a per-architecture child, not the multi-arch index digest pinned above. Pin that child and you have pinned one architecture, which will not schedule on nodes of the other. Copy the whole index (docker buildx imagetools create --tag <your-registry>/agledger:1.4.0 agledger/agledger:1.4.0, or crane copy / skopeo copy --all), then read the index digest back from your registry and pin that. Verified on this path: an ECR mirror holding only the amd64 child runs correctly on amd64 nodes when you pin the child digest, and the index digest simply does not exist in that repository. The cosign verification bundle ships with each GitHub release; see the install repository's air-gap guide for the offline cosign verify --new-bundle-format flow to verify the mirrored image first.