> Markdown version of https://agledger.ai/docs/install/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # Install AGLedger on Kubernetes This guide brings up a single AGLedger Server on a generic Kubernetes cluster using the published Helm chart, fronted by TLS, backed by your own PostgreSQL. Every command below is a literal transcript captured against a live Server. On OpenShift the same install takes one extra flag; see the OpenShift section below. > **Just evaluating? Start here instead.** This page is the production Kubernetes install. To get > a signed Server running locally in about five minutes with a single command - bundled > PostgreSQL, keys generated for you, no cluster - use the > [quick install on Docker Compose](/docs/install/compose/). > Image tag, chart version, and pinned digests updated to v1.4.0 on 2026-08-09. The Kubernetes > transcripts (keygen fingerprint, rollout, readiness gate) carry over from the v0.27.5 live > validation on 2026-06-04 (Kubernetes 1.35, Helm 3.20, cosign 3.0); they are version-stable and > were not re-run against a live cluster for this release. The v1.4.0 image index digest > (`sha256:75c9590f…`) and chart digest (`sha256:b70d8df5…`) are read from the published Docker Hub > registry after the release ran. > > **v1.4.0 adds a fourth migration** (`004_signing_agility.sql`), which carries the per-key > algorithm the engine now resolves signatures against. The upgrade stays a rolling image bump, but > it runs a schema step, so read *Version upgrades* in [Day-2 operations](/docs/operations/day-2/) > before you bump. v1.3.4 added the third (`003_export_cap_and_checkpoint_immutability.sql`). > > **The runtime base is Red Hat UBI 10** (`ubi10/nodejs-24-minimal`) as of v1.4.0. Nothing in this > guide changes; the container runs as the same non-root uid with the same hardening. > > **Release images are multi-arch** (`linux/amd64` and `linux/arm64`). Pulling by tag > selects your platform automatically. If you pin by digest, pin the index digest above rather than > a per-architecture child digest, or you will pin one architecture. **Verified on Graviton > (2026-07-27):** the index digest resolves per-node, the arm64 runtime is native (`process.arch` = > `arm64`, `aarch64` kernel, no emulation), migrations apply, and signatures produced on arm64 verify > against the offline verifier on x86. The [API reference](/api/) is the OpenAPI document the Server serves at `/openapi.json`. This guide does not restate request or response schemas; it links to them. ## What you provide A Server is durable only as far as its database and its signing key. You bring both: - **A PostgreSQL database.** AGLedger writes a hash-chained, Ed25519-signed record chain to it. Use any PostgreSQL you can reach over a direct connection (pg-boss needs `LISTEN`/`NOTIFY`, so no transaction-mode pooler). This guide runs one inside the cluster; production should use a managed or HA PostgreSQL. Three role requirements are worth knowing before you provision it: the role that runs migrations must be a superuser (the schema installs an event trigger, and event triggers require one), and the role the Server runs as must be named `agledger_app` unless it also owns the schema objects, because migrations grant table privileges to that name. A non-owner runtime role named anything else gets no privileges and the Server fails at boot with `permission denied for table ...`. If your naming standard forbids `agledger_app`, make your role a member of it after migrating: `GRANT agledger_app TO "your_role" WITH INHERIT TRUE`. Spell out the `INHERIT` clause, because PostgreSQL 16 and later default it to the member role's own setting, so a `NOINHERIT` role receives nothing from a bare grant. Membership carries the privilege set the schema intends, including the append-only revokes that leave `audit_vault` and its siblings insert-only and the `ALTER DEFAULT PRIVILEGES` that keep later migrations reachable. Do not substitute a blanket `GRANT ... ON ALL TABLES IN SCHEMA public`: it also hands the runtime role `UPDATE` and `DELETE` on `audit_vault`, which is the write the chain is protected against. If `agledger_app` does not exist at all, the migrating role could not create it and the schema granted nothing to anyone; grant your role `USAGE` on schema `public`, `SELECT, INSERT, UPDATE, DELETE` on all its tables and the matching `ALTER DEFAULT PRIVILEGES`, then re-apply the append-only revokes. Separately from anything the migration grants, the runtime role needs `CREATE` on the database (`GRANT CREATE ON DATABASE agledger TO agledger_app`): pg-boss keeps its queue tables in a schema it installs on first start, and the Server starts pg-boss, so without it the Server exits at boot with `permission denied for database ...`. Creating the schema by hand does not substitute, because pg-boss reads the absence of its own version table as "not installed" and issues `CREATE SCHEMA IF NOT EXISTS` regardless, which PostgreSQL refuses without the database-level privilege. The grant can be revoked after the first successful boot; later pg-boss upgrades stay inside the schema it owns by then. You do not have to get this right from memory: on the external-database path the chart runs both checks as a `pre-install`/`pre-upgrade` hook, after migrating and before any workload starts. A role that cannot serve fails `helm install` before the API or Worker is created, and the hook Job's log names the role it connected as and the exact `GRANT` to run. Read it with `kubectl logs -n --tail=-1 -l app.kubernetes.io/instance=,app.kubernetes.io/component=preflight` (the instance selector matters if you run more than one release in a namespace). Apply the grant, then re-run. Migrations have already been applied at that point and are unaffected, but the release keeps its name after a failed install, so clear it with `helm uninstall ` first if `helm list` still shows it. - **An Ed25519 vault signing key.** The Server signs every record with it. Generate it once, store it like any other root secret, and reuse it across upgrades - the published verification key is derived from it. ## Prerequisites - A Kubernetes cluster (1.27+) and `kubectl` pointed at it - `helm` 3.x and `cosign` 3.0+ - An ingress controller and a way to issue TLS certificates. This guide uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://cert-manager.io/); any controller and certificate source work. - A usable `StorageClass` - only if you let the chart run PostgreSQL in-cluster (see step 3). Bundled PostgreSQL provisions a PersistentVolumeClaim against the cluster's default class, or an explicit `postgres.bundled.storageClassName`. With neither, the claim stays `Pending` and the install hangs. Modern EKS (1.30+) ships no default class - set `postgres.bundled.storageClassName` to your provisioner's class, e.g. `gp3`, with the EBS CSI driver installed. ## 1. Verify the release Releases are keyless-signed: GitHub Actions OIDC -> Sigstore/Fulcio -> the public Rekor transparency log. There is no static public key to fetch. A valid signature binds to the GitHub Actions workflow in agledger-ai/agledger-api that built the artifact, verifiable with no access to the source repository. Requires cosign 3.0+. Verify both the image and the chart before you install. ``` $ cosign verify \ --certificate-identity-regexp '^https://github\.com/agledger-ai/agledger-api/\.github/workflows/.+@refs/tags/v.+$' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ agledger/agledger:1.4.0 $ cosign verify \ --certificate-identity-regexp '^https://github\.com/agledger-ai/agledger-api/\.github/workflows/.+@refs/tags/v.+$' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ registry-1.docker.io/agledger/agledger-chart:1.4.0 ``` Expected - each prints `Verification for ... --` and a JSON block. The certificate identity in the output is the GitHub Actions workflow that built the artifact, and the image digest in the output (`sha256:75c9590f…`) is the digest you pin in step 3. To additionally check build provenance, verify the SLSA Build L3 attestation with `slsa-verifier verify-image` (full recipe in the install repo's `SECURITY.md` at https://github.com/agledger-ai/install). If you install with the scripts rather than by hand, set `AGLEDGER_REQUIRE_VERIFY=true`. The scripts verify signatures when cosign is present and continue with a warning when it is not, which suits an evaluation; this variable makes a run that cannot verify refuse to install instead. It is the control that makes verification mandatory, so set it on production hosts and in CI. ``` $ AGLEDGER_REQUIRE_VERIFY=true ./scripts/install.sh --version 1.4.0 ``` The lever in the other direction, `--skip-verify`, exists for local development. Neither it nor `AGLEDGER_SKIP_VERIFY=true` belongs on a host that serves anything. ## 2. Generate the vault signing key Generate the Ed25519 key with the Server image itself, then keep the private key safe. ``` $ kubectl run agledger-keygen --restart=Never --image=agledger/agledger:1.4.0 \ --command -- /nodejs/bin/node dist/scripts/generate-signing-key.js $ kubectl logs agledger-keygen VAULT_SIGNING_KEY= Public key: MCowBQYDK2VwAyEAPInxjno26azIT9i6GVqJag9QuCJFMoDG96iljTd8fHo= Fingerprint: dcd0573755f045c6 $ kubectl delete pod agledger-keygen ``` The fingerprint (`dcd0573755f045c6`) is the key id the Server later publishes at `/v1/verification-keys`. Confirming they match (step 5) is how you prove the Server is signing with the key you provided. ## 3. Bring up PostgreSQL and install Create the namespace and a PostgreSQL the Server can reach. Then install the chart, pointing `database.externalUrl` at it and passing the signing key. > Simpler path for evaluation: let the chart run PostgreSQL. Set `postgres.bundled.enabled: true` > in `values.yaml` and omit `database.externalUrl` - the chart provisions an in-cluster > `postgres:18-alpine` with a 10Gi PersistentVolumeClaim. Bundled PostgreSQL is not recommended for > production. It requires a usable `StorageClass`: with no cluster default and no explicit > `postgres.bundled.storageClassName`, the claim stays `Pending` and the install hangs. On EKS 1.30+ > (no default class) set the class explicitly. > > ```yaml > postgres: > bundled: > enabled: true > password: # also used in the generated database URL > storageClassName: gp3 # required on EKS 1.30+; "" uses the cluster default > ``` > > Confirm the claim binds before waiting on the workloads - `kubectl get pvc -n agledger` should show > the `-pgdata` claim `Bound`, not `Pending`. Choose one database path, not both; the steps below use > an external PostgreSQL. ``` $ kubectl create namespace agledger $ kubectl apply -f postgres.yaml # your PostgreSQL Deployment + Service $ kubectl rollout status deploy/agledger-postgres -n agledger --timeout=120s deployment "agledger-postgres" successfully rolled out ``` `values.yaml`: ```yaml image: digest: "sha256:75c9590f353c063735728229f07f941fd063b1ab242caee900406ddc1eb5d20c" # 1.4.0 database: # sslmode is required under nodeEnv=production. Prefer sslmode=verify-full against a TLS-enabled # database with config.nodeExtraCaCerts pointing at its CA — `require` encrypts but does NOT # validate the server certificate (the CA bundle is only consulted under verify-ca/verify-full). # sslmode=disable only suits a local, non-TLS PostgreSQL like the in-cluster one here. externalUrl: "postgresql://agledger:@agledger-postgres.agledger.svc.cluster.local:5432/agledger?sslmode=disable" config: externalUrl: "https://agledger.k8s.example" # the Server's signed issuer identity; use your host ingress: enabled: true className: nginx annotations: cert-manager.io/cluster-issuer: agledger-selfsigned hosts: - host: agledger.k8s.example paths: [{ path: /, pathType: Prefix }] tls: - secretName: agledger-tls hosts: [agledger.k8s.example] ``` Write the key to a file rather than passing it inline. `--set` puts the value in helm's argv, where `ps` shows it to every other user on the machine for the length of the install, and the vault signing key is the private key every record is signed with. ``` $ umask 077 && printf %s '' > vault-key $ helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \ --version 1.4.0 --namespace agledger \ --values values.yaml --set-file secrets.vaultSigningKey=vault-key NAME: agledger STATUS: deployed REVISION: 1 $ rm vault-key ``` The chart runs schema migrations as a pre-install hook, then starts the API and worker. A fresh install applies every shipped migration in a single Job; re-running the Job against a database already at that head applies nothing and reports a migration count of `0`, so a repeated `helm upgrade` is safe. On the external-database path the migrate Job is a pre-install/pre-upgrade hook, so a migration pod that cannot schedule fails the whole install or upgrade. If your API pods are pinned to a node class (architecture, taints), set `migrate.nodeSelector` / `migrate.tolerations` / `migrate.affinity` to match. Each is empty by default and falls back to the matching `api.*` value, so an install that pins only the API is already consistent. ### Sizing an upgrade window A migration that adds a column and builds an index does both in one transaction, which holds `ACCESS EXCLUSIVE` on the table until it commits: reads as well as writes stop for the duration. The cost is driven by how wide your `criteria` documents are, not by row count alone, and each migration file's header carries the measured numbers for that migration. Read it before upgrading a mature install. `migrate.extraEnv` delivers the knobs to the migration Job (it is a separate list from the top-level `extraEnv`, which goes to the API and worker): ```yaml migrate: extraEnv: # Default 15min. Raise past your expected index build time; if it fires, # the migration rolls back cleanly and you retry with a larger value. - name: MIGRATION_STATEMENT_TIMEOUT value: "60min" # Unset by default, meaning the server's own value. Raising it shortens # the window rather than just permitting a longer one. PostgreSQL takes # this per maintenance operation, so a large value can push a small # containerized PostgreSQL into the OOM killer. - name: MIGRATION_MAINTENANCE_WORK_MEM value: "256MB" ``` Do not set `DATABASE_URL` here if you also use `secrets.databaseUrlMigrate`: the chart refuses to render rather than let one silently override the other. On Compose the same knobs are read from `.env`; see the migration section of `.env.example`. ``` $ kubectl rollout status deploy/agledger-agledger-chart-api -n agledger --timeout=180s deployment "agledger-agledger-chart-api" successfully rolled out $ kubectl rollout status deploy/agledger-agledger-chart-worker -n agledger --timeout=120s deployment "agledger-agledger-chart-worker" successfully rolled out ``` ## 4. Create the platform API key The platform key is the first credential; you use it to provision organizations and agents. It is printed once. ``` $ kubectl exec deploy/agledger-agledger-chart-api -n agledger -- \ /nodejs/bin/node dist/scripts/init.js --non-interactive ✓ API_KEY_SECRET found in environment ✓ VAULT_SIGNING_KEY found in environment ✓ AGLEDGER_FEDERATION_SIGNING_KEY generated (Ed25519) ✓ AGLEDGER_FEDERATION_ENCRYPTION_KEY generated (X25519) ⚠ Read-only filesystem: .env could not be written. 3 secret value(s) already set in this environment are shown by name only. The secrets printed below were generated by this run and exist nowhere else. Capture them. --- .env content --- ... DATABASE_URL= API_KEY_SECRET= VAULT_SIGNING_KEY= AGLEDGER_FEDERATION_SIGNING_KEY= AGLEDGER_FEDERATION_ENCRYPTION_KEY= --- ✓ Platform API key created (ID: ...) ✓ Database query OK │ Platform Key: agl_plt_ ``` The chart runs the container with a read-only root filesystem, so `init.js` cannot write a `.env` and prints one to stdout instead. Everything you supplied in step 2 is shown by name only: your database password, `API_KEY_SECRET` and `VAULT_SIGNING_KEY` are already in your Secret and are not reprinted. **Two things in that output are secret and exist nowhere else:** the platform key, and the federation keypair this run generated. Federation keys are optional. Capture them into your Secret as `AGLEDGER_FEDERATION_SIGNING_KEY` and `AGLEDGER_FEDERATION_ENCRYPTION_KEY` if you intend to federate; otherwise ignore them, and a later run generates a fresh pair. Either way, do not run this step in a CI job whose logs are retained. ## 5. The readiness gate: up and signing A Server is ready when it is healthy and signing with your key. ``` $ kubectl port-forward svc/agledger-agledger-chart -n agledger 3001:80 & $ curl -s http://localhost:3001/health {"status":"ok","version":"1.4.0","timestamp":"..."} $ curl -s http://localhost:3001/health/ready {"status":"ready","version":"1.4.0","timestamp":"..."} ``` Confirm the active signing key is the one you generated in step 2 - same fingerprint: ``` $ curl -s http://localhost:3001/v1/verification-keys { "data": [{ "keyId": "dcd0573755f045c6", "algorithm": "Ed25519", "publicKey": "MCowBQYDK2VwAyEAPInxjno26azIT9i6GVqJag9QuCJFMoDG96iljTd8fHo=", "status": "active" }], "envelope": "COSE_Sign1", "signatureAlgorithm": "Ed25519" } ``` `keyId` matches the keygen fingerprint: the Server is signing with your key, COSE_Sign1 / Ed25519. With no license applied the Server runs as Developer Edition (all features enabled): ``` $ curl -s -H "Authorization: Bearer " http://localhost:3001/v1/admin/license {"validity":"unlicensed","tier":"developer", "features":["custom_schemas","expression_engine","delegation_chains","audit_export", "compliance_reports","encrypted_mode","entity_references","proposals","federation"], "source":"none"} ``` ## 6. TLS With ingress-nginx and cert-manager, the chart's Ingress requests a certificate and serves it. A self-signed ClusterIssuer is portable and needs no public DNS; for production use an ACME (Let's Encrypt) or your-CA issuer. ``` $ kubectl get certificate -n agledger NAME READY SECRET AGE agledger-tls True agledger-tls ... $ curl -sk https://agledger.k8s.example/health # through the ingress {"status":"ok","version":"1.4.0","timestamp":"..."} ``` ## External database The in-cluster PostgreSQL above is the simplest start. To use a managed PostgreSQL, point `database.externalUrl` at it with `sslmode=verify-full` and set `config.nodeExtraCaCerts` to its CA bundle (`require` encrypts without validating the server certificate). External-database licensing is per database instance. For Amazon Aurora / RDS specifically, see the [AWS install guide](/docs/install/aws/). **The migration role needs superuser.** The schema installs an event trigger, `agledger_block_audit_drop`, which is what stops the audit chain being dropped, and `CREATE EVENT TRIGGER` is superuser-only in PostgreSQL. A managed database hands you an owner role with `CREATEDB` and not this, so migrations fail with `permission denied to create event trigger` unless you grant it: | Provider | Grant | |---|---| | Amazon RDS / Aurora | `GRANT rds_superuser TO ;` | | Google Cloud SQL | `GRANT cloudsqlsuperuser TO ;` | | Azure Database for PostgreSQL | `GRANT azure_pg_admin TO ;` | | Self-managed | `ALTER ROLE SUPERUSER;` | To keep the running Server least-privilege, grant it to a separate role and give that role to migrations only: set `secrets.databaseUrlMigrate` (Helm) or `DATABASE_URL_MIGRATE` (Compose) to its URL, and leave `database.externalUrl` / `DATABASE_URL` as the DML role. `install.sh` checks this before it pulls anything and names the grant for your provider. ## OpenShift Add `--set openshift.enabled=true` to the `helm install` above. Everything else on this page is unchanged. `openshift.enabled` was introduced in chart 1.4.0, the version this page pins. On an earlier chart the key does not exist, and Helm accepts an unknown `--set` key without complaint, so the flag is silently ignored, the pod-level uid request stays, and you get exactly the admission error below. ```bash helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \ --version 1.4.0 --namespace agledger --create-namespace \ --set openshift.enabled=true \ --values values.yaml ``` Without it, admission rejects every pod before it starts: ``` unable to validate against any security context constraint: runAsUser: Invalid value: 65532: must be in the ranges: [1000700000, ...] ``` OpenShift's default `restricted-v2` SCC assigns each namespace its own uid range and admits pods with `MustRunAsRange`. The chart's stock pod security context asks for uid/gid 65532, which is outside that range on essentially every cluster. The flag drops that block from all three workloads (the api Deployment, the worker Deployment, and the migrate Job) and lets the platform assign a uid instead. The bundled PostgreSQL needs no equivalent: it requests no security context at all, which is exactly what `restricted-v2` admits. What goes is `runAsNonRoot: true` alongside the two uid/gid keys, so non-root stops being asserted in the manifest. It still holds, by a different guarantee: on OpenShift `restricted-v2` enforces it directly along with a default seccomp profile, and elsewhere it rests on the image's own `USER 65532:65532`. Every container-level control is untouched either way, so each container keeps its read-only root filesystem, no privilege escalation, and all capabilities dropped. The image supports an arbitrary uid: it declares `USER 65532` for plain Kubernetes and Docker, but nothing in it is owned by or hardcoded to that uid, its files are world-readable, and it needs no passwd entry. `./scripts/helm-install.sh` sets the flag for you when it sees the `security.openshift.io` API group on the cluster, unless you already set `openshift.enabled` yourself, whether by `--set` or in a values file it was passed. The chart also ships `values-openshift.yaml`, which sets the same flag for anyone who prefers a values file, though reaching it means `helm pull --untar` first: ```bash helm pull oci://registry-1.docker.io/agledger/agledger-chart --version --untar helm install agledger ./agledger-chart -n agledger \ -f ./agledger-chart/values-openshift.yaml \ -f values.yaml ``` > **What was measured.** Under the constraints `restricted-v2` imposes on a container > (`--user 1000700000:0 --read-only --cap-drop ALL --security-opt no-new-privileges`), the shipped > image applies its migrations, boots healthy, serves the unauthenticated discovery surfaces, mints > a platform key, and notarizes a record, so signing works under an assigned uid. The rendered > manifests were checked directly: the flag removes the pod-level uid request from all three > workloads and leaves container hardening identical to a stock render. Admission itself was not > exercised on a live OpenShift cluster; that part follows from the SCC rules. ## Run more than one Server You can run more than one Server and link them so chains reference records across Servers (we call linking Servers federation). Each Server is a full, independent install of this guide with its own database, database role, and signing key; linking is configured after both are healthy. If you host several Servers on one PostgreSQL cluster, give each its own role: PostgreSQL roles are cluster-wide, so Servers sharing the `agledger_app` role share one credential across all of their databases. ## Air-gapped install Nothing in install or runtime depends on agledger.ai, Docker Hub, or npm at runtime. For a restricted network, mirror the image (`agledger/agledger:1.4.0`) and chart (`agledger/agledger-chart:1.4.0`) into your internal registry, the cosign verification bundle ships with each GitHub release; see the install repository's air-gap guide for the offline `cosign verify --new-bundle-format` flow, and set `image.repository` (and `image.pullSecrets`) to your registry. The Server makes no outbound calls except the webhook and federation endpoints you configure. ## Uninstall ``` $ helm uninstall agledger -n agledger release "agledger" uninstalled ``` This leaves your database and signing key intact, so a reinstall against the same database and key resumes the same chain. ## Next Keep the Server healthy over time with [Day-2 Operations](/docs/operations/day-2/), and set up a [backup](/docs/operations/backup/) and [recovery](/docs/operations/recovery/) routine.