> Markdown version of https://agledger.ai/docs/install/aws/
> Full index of this site for AI assistants: https://agledger.ai/llms.txt

# Install AGLedger on Amazon EKS

This guide brings up a single AGLedger Server on Amazon EKS, reachable on a real hostname over
TLS, using AWS-native services: an AWS Load Balancer Controller Application Load Balancer (ALB),
an AWS Certificate Manager (ACM) certificate, and Amazon Aurora PostgreSQL as the database. It is
the AWS companion to [Install AGLedger on Kubernetes](/docs/install/); read that first for the base
model (signing key, readiness gate). Every command below is a literal transcript captured against
a live Server reachable at `https://agledger-aws.agledger.ai`.

> **Live-validated against v1.4.0 on 2026-08-19** on a real Amazon EKS 1.36 + Amazon Aurora
> PostgreSQL 18 cluster (us-west-2), following this runbook literally end to end at the pinned
> `sha256:75c9590f…` index digest and chart 1.4.0. Everything below reproduced: the Aurora role and
> `rds_superuser` grant (the event trigger created without `permission denied`), `helm install`
> reporting `STATUS: deployed` / `REVISION: 1`, the migrate Job at `Complete 1/1` in 7s applying
> **four** migrations on Aurora (previously only confirmed on bundled PostgreSQL), the ALB target
> reaching `healthy` in about two minutes with `networkPolicy.enabled: false`, and every cell of the
> readiness gate including the ACM subject/issuer strings, the HTTP→HTTPS 301, and
> `/v1/verification-keys` returning the keygen fingerprint. Beyond the gate, a record notarized
> against Aurora over `sslmode=verify-full` was exported and **verified offline** with the published
> `@agledger/verify` 1.5.1, pinned to that key id with `--require-out-of-band-keys`.
>
> Two things in this guide were **not** exercised on that run: the *Upgrading the Aurora engine*
> section (the cluster was already on 18, and a major version cannot be downgraded in place to
> re-test it), and *AWS licensing* (no Marketplace entitlement on the validation account). Both
> remain as previously recorded. DNS was pointed at the ALB by host pinning rather than a Route 53
> alias, so the ALB hostname read-back is validated and the record creation itself is not.
>
> The runbook was **live-validated against v1.3.4 on 2026-07-27** on a real Amazon EKS 1.36 + Amazon Aurora
> PostgreSQL 18.3 cluster (us-west-2), following this runbook literally end to end. Every command and
> every quoted output below reproduced: Aurora prep (role + `rds_superuser`), `helm install` at the
> pinned digest, the migrate Job reporting `Complete 1/1` with **three** migrations applied, ALB
> wiring (`networkPolicy.enabled: false`, `target-type: ip` targets reaching `healthy`), and the full
> readiness gate (ACM cert subject/issuer, HTTP→HTTPS 301, `verification-keys` keyId == keygen
> fingerprint). Previously live-validated against v1.2.0 on 2026-07-09. For an in-place
> PostgreSQL 17→18 upgrade, see *Upgrading the Aurora engine* below.
>
> **Verified on Graviton (2026-07-27).** The digest pinned below is the multi-arch **index** digest,
> and it was confirmed to resolve per-node on an arm64 (Graviton) node: pods report the index digest
> as their `imageID` while the runtime is genuinely arm64 (`process.arch` = `arm64`, `aarch64`
> kernel, no emulation), migrations apply natively, and Ed25519 signatures produced on arm64 verify
> against the published offline verifier running on x86. If you pin a per-architecture child digest
> instead, you pin one architecture; pin the index.
>
> **Reviewed for API v1.4.0 on 2026-08-09:** a fresh install applies **four** migrations
> (`001_consolidated.sql`, `002_webhook_record_types.sql`,
> `003_export_cap_and_checkpoint_immutability.sql`, `004_signing_agility.sql`); the migrate Job
> still reports `Complete 1/1`, since that counts the Job, not the migrations. Confirmed on a live
> v1.4.0 install, on bundled PostgreSQL rather than Aurora.
>
> **Reviewed for API v1.3.4 on 2026-07-27:** the runbook itself is unchanged, but two things in it
> move. A fresh install then applied **three** migrations; the migrate Job still reports
> `Complete 1/1`, since that counts the Job, not the migrations.
> And on the external-database path the migrate Job is a pre-install/pre-upgrade hook, so a
> migration pod that cannot schedule fails the whole install: if your API pods are pinned to a node
> class, set `migrate.nodeSelector` / `.tolerations` / `.affinity`, each of which defaults to the
> matching `api.*` value. Release images are also multi-arch (`linux/amd64` and `linux/arm64`) as of
> v1.3.4; the digest pinned below is the index digest, which resolves per-node.

## AWS prerequisites

- An **Amazon EKS** cluster (1.27+) with `kubectl` and `helm` configured against it
- The **AWS Load Balancer Controller** installed in the cluster - it provides the `alb`
  IngressClass and provisions the ALB from the Ingress resource. (Install via its Helm chart with
  an IAM Role for Service Accounts / IRSA, per the AWS docs.)
- An **ACM certificate** in the same region as the ALB, covering your hostname (here a wildcard
  `*.agledger.ai`), status `ISSUED`
- An **Amazon Aurora PostgreSQL** cluster (or Amazon RDS for PostgreSQL) reachable from the EKS
  node/pod security group on 5432
- `cosign` 3.0+ to verify the release (see [Install](/docs/install/) step 1 - same commands)

## 1. Prepare the Amazon Aurora database

Create a dedicated database and an application role. Run this from inside the VPC (Aurora is not
publicly reachable) - for example a one-off psql pod connecting as the Aurora master user.

```
CREATE ROLE agledger_aws_app LOGIN PASSWORD '<app-password>';
GRANT agledger_aws_app TO agledger;          -- master must be a member to set ownership
CREATE DATABASE agledger_aws OWNER agledger_aws_app;
```

**Migration privilege (important).** The schema migration creates a PostgreSQL event trigger
(`agledger_block_audit_drop`, which protects the audit chain). Event triggers require superuser;
on Aurora / RDS that is the `rds_superuser` role. A plain database-owner role is not enough - the
migration fails with `permission denied to create event trigger`. Grant the migration role
`rds_superuser`:

```
GRANT rds_superuser TO agledger_aws_app;
```

For least-privilege role separation (a privileged role for migrations, a restricted role for the
running Server), use the chart's `secrets.databaseUrlMigrate` for the migration DDL and a
restricted `database.externalUrl` for the API and worker.

The connection string uses `sslmode=verify-full`. Under PostgreSQL semantics, `sslmode=require`
encrypts but performs **no certificate validation** - only `verify-ca`/`verify-full` check the
server certificate against a CA bundle (and `verify-full` also checks the hostname). The agledger
image bundles the AWS RDS / Aurora root CA at `/etc/ssl/certs/rds-global-bundle.pem`; set
`config.nodeExtraCaCerts` to that path and use `verify-full` so the bundle is actually consulted
and the Server validates the Aurora server certificate.

## 2. Values for the AWS path

`aws-values.yaml`:

```yaml
image:
  digest: "sha256:75c9590f353c063735728229f07f941fd063b1ab242caee900406ddc1eb5d20c"  # 1.4.0
database:
  poolMax: 20
config:
  externalUrl: "https://agledger-aws.agledger.ai"          # the Server's signed issuer identity
  nodeExtraCaCerts: "/etc/ssl/certs/rds-global-bundle.pem"  # bundled AWS RDS / Aurora CA
ingress:
  enabled: true
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-2:<acct>:certificate/<id>"
  hosts:
    - host: agledger-aws.agledger.ai
      paths: [{ path: /, pathType: Prefix }]
  tls: []   # TLS terminates at the ALB via the ACM cert above — no in-cluster TLS secret
networkPolicy:
  enabled: false   # see note below
```

**NetworkPolicy on the ALB path.** The chart's default NetworkPolicy admits ingress only from
ingress-controller pods (ingress-nginx / traefik). With `target-type: ip`, the ALB sends traffic
and health checks from VPC elastic network interfaces - not from a pod - so the default policy
blocks the ALB and targets never go healthy. Set `networkPolicy.enabled: false` on the ALB path
(or apply your own policy admitting the VPC CIDR on port 3000).

## 3. Install

Generate the vault signing key and create the platform key exactly as in [Install](/docs/install/)
(steps 2 and 4). Keep the Aurora URL and the signing key out of the values file, and out of the
command line too: `--set` puts a value in helm's argv, where `ps` shows it to every other user on
the machine for the length of the install. Both of these carry a secret, the database password and
the private key every record is signed with, so pass them as files.

```
$ kubectl create namespace agledger-aws
$ umask 077
$ printf %s 'postgresql://agledger_aws_app:<pw>@<aurora-endpoint>:5432/agledger_aws?sslmode=verify-full' > db-url
$ printf %s '<vault-key>' > vault-key
$ helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \
    --version 1.4.0 --namespace agledger-aws \
    --values aws-values.yaml \
    --set-file database.externalUrl=db-url \
    --set-file secrets.vaultSigningKey=vault-key
NAME: agledger
STATUS: deployed
REVISION: 1
  API URL:  https://agledger-aws.agledger.ai

$ rm db-url vault-key

$ kubectl rollout status deploy/agledger-agledger-chart-api -n agledger-aws --timeout=180s
deployment "agledger-agledger-chart-api" successfully rolled out
$ kubectl get pods,jobs -n agledger-aws
pod/agledger-agledger-chart-api-...      1/1   Running
pod/agledger-agledger-chart-migrate-...  0/1   Completed
pod/agledger-agledger-chart-worker-...   1/1   Running
job.batch/agledger-agledger-chart-migrate   Complete   1/1   7s
```

## 4. Point DNS at the ALB

The Ingress provisions an ALB; read its hostname and create a DNS record for your host (CNAME, or
a Route 53 alias) pointing at it.

```
$ kubectl get ingress -n agledger-aws
NAME                      CLASS   HOSTS                      ADDRESS
agledger-agledger-chart   alb     agledger-aws.agledger.ai   k8s-agledger-agledger-....us-west-2.elb.amazonaws.com
```

Wait for the ALB target to register as `healthy`:

```
$ aws elbv2 describe-target-health --target-group-arn <tg-arn> \
    --query 'TargetHealthDescriptions[].TargetHealth.State'
[ "healthy" ]
```

## 5. The readiness gate: named, TLS-terminated, signing

Reach the Server on its real hostname over HTTPS. The ALB serves the ACM certificate.

```
$ curl -s https://agledger-aws.agledger.ai/health
{"status":"ok","version":"1.4.0","timestamp":"..."}

$ echo | openssl s_client -connect agledger-aws.agledger.ai:443 \
    -servername agledger-aws.agledger.ai 2>/dev/null | openssl x509 -noout -subject -issuer
subject=CN = *.agledger.ai
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M04

$ curl -s -o /dev/null -w "HTTP %{http_code} -> %{redirect_url}\n" http://agledger-aws.agledger.ai/health
HTTP 301 -> https://agledger-aws.agledger.ai:443/health
```

The served certificate is issued by Amazon (ACM), and plain HTTP is redirected to HTTPS by the
`ssl-redirect` annotation. Confirm the Server is signing with the key you generated - the `keyId`
matches your vault-key fingerprint:

```
$ curl -s https://agledger-aws.agledger.ai/v1/verification-keys
{"data":[{"keyId":"c4ddafd6bf06f1ef","algorithm":"Ed25519","status":"active",…}],…}
```

(Response elided - the live response also carries per-key `publicKey`/`publicKeyRaw`/timestamps
and top-level signature-format fields. The validation point is the `keyId` matching your
fingerprint.)

The Server is healthy, reachable on its name over an ACM-issued certificate, backed by Aurora, and
signing with your key.

## Upgrading the Aurora engine (PostgreSQL 17 → 18)

Aurora supports an in-place major-version upgrade (set the cluster's engine version to `18.x` with
`allow_major_version_upgrade`). The Server's data, audit chain, and signatures carry through it
unchanged - after a 17→18 in-place upgrade we re-verified the full vault (every record's hash-chain
and Ed25519 / COSE_Sign1 envelope) with zero breakage, and the API self-recovered once the database
returned (no pod restart) after the brief upgrade window. **Plan for a short write outage** during
the upgrade and run it between workloads, not during one.

**Adopting native `uuidv7` after the upgrade.** On PostgreSQL 17 the schema migration installs a
small `uuidv7()` polyfill in the `public` schema (PostgreSQL gained a native `uuidv7()` in 18). A
*fresh* 18 install never creates it. But an *in-place* upgrade does not adopt the native function
automatically: the polyfill persists, and every table's `id` column default stays bound to it, so
inserts keep using the polyfill.

**From v1.2.0 this adoption is automatic** - upgrade the database first, then deploy the release,
and its migration re-points the polyfill-pinned column defaults to native `uuidv7()` and drops the
polyfill on its own (see [day-2 operations](/docs/operations/day-2/) for the ordering rules). Run
the block below only if you are on v1.1.0 or earlier, or you upgraded the database *after*
v1.2.0's migration had already run. It is **idempotent and self-discovering** (it only acts when
native `uuidv7` exists and the polyfill is present, and finds the columns itself - there is no fixed
list to maintain); run it once, as a superuser, against your application database after the upgrade:

```sql
-- Re-point every uuidv7() column default so it re-resolves to the native pg_catalog.uuidv7,
-- then remove the now-unused polyfill. No-op on a fresh 18 install or an already-remediated DB.
DO $$
DECLARE r record;
BEGIN
  -- only act if BOTH native and the public polyfill exist
  IF EXISTS (SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
             WHERE p.proname = 'uuidv7' AND n.nspname = 'pg_catalog')
     AND EXISTS (SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
                 WHERE p.proname = 'uuidv7' AND n.nspname = 'public') THEN
    FOR r IN
      SELECT n.nspname AS sch, c.relname AS tbl, a.attname AS col
      FROM pg_attrdef ad
      JOIN pg_class c       ON c.oid = ad.adrelid
      JOIN pg_namespace n   ON n.oid = c.relnamespace
      JOIN pg_attribute a   ON a.attrelid = ad.adrelid AND a.attnum = ad.adnum
      WHERE pg_get_expr(ad.adbin, ad.adrelid) ILIKE '%uuidv7%'
    LOOP
      EXECUTE format('ALTER TABLE %I.%I ALTER COLUMN %I SET DEFAULT uuidv7()', r.sch, r.tbl, r.col);
    END LOOP;
    DROP FUNCTION public.uuidv7();
  END IF;
END $$;
```

After it runs, new inserts use native `uuidv7()` and the `public.uuidv7` function is gone. (Do not
`DROP FUNCTION public.uuidv7()` on its own - the column defaults depend on it, so a bare drop fails
and `CASCADE` would strip the defaults and break inserts. The re-point above is what makes the drop
safe.)

## AWS licensing

With no license the Server runs as Developer Edition. Subscribe to Enterprise on the
[AWS Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-nmwogdztiyh64), then set the
Marketplace product ID (`prod-…`) and grant the Server's service account access to AWS License
Manager with IRSA - `marketplace.productId` and `marketplace.serviceAccountAnnotations` in the
chart values.

## Air-gapped / private registries

To run from Amazon ECR instead of Docker Hub, mirror `agledger/agledger:1.4.0` into ECR, set
`image.repository` to the ECR repository and `image.pullSecrets` (or use the node role / IRSA for
ECR pull), and pin `image.digest`.

**Mirror the index, not one architecture.** A plain `docker pull` + `docker push` copies only the
image for the machine doing the mirroring, so the digest your registry then reports is a
per-architecture child, not the multi-arch index digest pinned above. Pin that child and you have
pinned one architecture, which will not schedule on nodes of the other. Copy the whole index
(`docker buildx imagetools create --tag <your-registry>/agledger:1.4.0 agledger/agledger:1.4.0`, or
`crane copy` / `skopeo copy --all`), then read the index digest back from your registry and pin
that. Verified on this path: an ECR mirror holding only the amd64 child runs correctly on amd64
nodes when you pin the child digest, and the index digest simply does not exist in that repository.
The cosign verification bundle ships with each GitHub release; see the install repository's
air-gap guide for the offline `cosign verify --new-bundle-format` flow to verify the mirrored
image first.