Install AGLedger on Kubernetes

This guide brings up a single AGLedger Server on a generic Kubernetes cluster using the published Helm chart, fronted by TLS, backed by your own PostgreSQL. Every command below is a literal transcript captured against a live Server. On OpenShift the same install takes one extra flag; see the OpenShift section below.

Just evaluating? Start here instead. This page is the production Kubernetes install. To get a signed Server running locally in about five minutes with a single command - bundled PostgreSQL, keys generated for you, no cluster - use the quick install on Docker Compose.

Image tag, chart version, and pinned digests updated to v1.4.0 on 2026-08-09. The Kubernetes transcripts (keygen fingerprint, rollout, readiness gate) carry over from the v0.27.5 live validation on 2026-06-04 (Kubernetes 1.35, Helm 3.20, cosign 3.0); they are version-stable and were not re-run against a live cluster for this release. The v1.4.0 image index digest (sha256:75c9590f…) and chart digest (sha256:b70d8df5…) are read from the published Docker Hub registry after the release ran.

v1.4.0 adds a fourth migration (004_signing_agility.sql), which carries the per-key algorithm the engine now resolves signatures against. The upgrade stays a rolling image bump, but it runs a schema step, so read Version upgrades in Day-2 operations before you bump. v1.3.4 added the third (003_export_cap_and_checkpoint_immutability.sql).

The runtime base is Red Hat UBI 10 (ubi10/nodejs-24-minimal) as of v1.4.0. Nothing in this guide changes; the container runs as the same non-root uid with the same hardening.

Release images are multi-arch (linux/amd64 and linux/arm64). Pulling by tag selects your platform automatically. If you pin by digest, pin the index digest above rather than a per-architecture child digest, or you will pin one architecture. Verified on Graviton (2026-07-27): the index digest resolves per-node, the arm64 runtime is native (process.arch = arm64, aarch64 kernel, no emulation), migrations apply, and signatures produced on arm64 verify against the offline verifier on x86.

The API reference is the OpenAPI document the Server serves at /openapi.json. This guide does not restate request or response schemas; it links to them.

What you provide

A Server is durable only as far as its database and its signing key. You bring both:

Prerequisites

1. Verify the release

Releases are keyless-signed: GitHub Actions OIDC -> Sigstore/Fulcio -> the public Rekor transparency log. There is no static public key to fetch. A valid signature binds to the GitHub Actions workflow in agledger-ai/agledger-api that built the artifact, verifiable with no access to the source repository. Requires cosign 3.0+. Verify both the image and the chart before you install.

$ cosign verify \
    --certificate-identity-regexp '^https://github\.com/agledger-ai/agledger-api/\.github/workflows/.+@refs/tags/v.+$' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    agledger/agledger:1.4.0
$ cosign verify \
    --certificate-identity-regexp '^https://github\.com/agledger-ai/agledger-api/\.github/workflows/.+@refs/tags/v.+$' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    registry-1.docker.io/agledger/agledger-chart:1.4.0

Expected - each prints Verification for ... -- and a JSON block. The certificate identity in the output is the GitHub Actions workflow that built the artifact, and the image digest in the output (sha256:75c9590f…) is the digest you pin in step 3. To additionally check build provenance, verify the SLSA Build L3 attestation with slsa-verifier verify-image (full recipe in the install repo's SECURITY.md at https://github.com/agledger-ai/install).

If you install with the scripts rather than by hand, set AGLEDGER_REQUIRE_VERIFY=true. The scripts verify signatures when cosign is present and continue with a warning when it is not, which suits an evaluation; this variable makes a run that cannot verify refuse to install instead. It is the control that makes verification mandatory, so set it on production hosts and in CI.

$ AGLEDGER_REQUIRE_VERIFY=true ./scripts/install.sh --version 1.4.0

The lever in the other direction, --skip-verify, exists for local development. Neither it nor AGLEDGER_SKIP_VERIFY=true belongs on a host that serves anything.

2. Generate the vault signing key

Generate the Ed25519 key with the Server image itself, then keep the private key safe.

$ kubectl run agledger-keygen --restart=Never --image=agledger/agledger:1.4.0 \
    --command -- /nodejs/bin/node dist/scripts/generate-signing-key.js
$ kubectl logs agledger-keygen
VAULT_SIGNING_KEY=<base64 Ed25519 private key>
Public key: MCowBQYDK2VwAyEAPInxjno26azIT9i6GVqJag9QuCJFMoDG96iljTd8fHo=
Fingerprint: dcd0573755f045c6
$ kubectl delete pod agledger-keygen

The fingerprint (dcd0573755f045c6) is the key id the Server later publishes at /v1/verification-keys. Confirming they match (step 5) is how you prove the Server is signing with the key you provided.

3. Bring up PostgreSQL and install

Create the namespace and a PostgreSQL the Server can reach. Then install the chart, pointing database.externalUrl at it and passing the signing key.

Simpler path for evaluation: let the chart run PostgreSQL. Set postgres.bundled.enabled: true in values.yaml and omit database.externalUrl - the chart provisions an in-cluster postgres:18-alpine with a 10Gi PersistentVolumeClaim. Bundled PostgreSQL is not recommended for production. It requires a usable StorageClass: with no cluster default and no explicit postgres.bundled.storageClassName, the claim stays Pending and the install hangs. On EKS 1.30+ (no default class) set the class explicitly.

postgres:
  bundled:
    enabled: true
    password: <change-me>       # also used in the generated database URL
    storageClassName: gp3       # required on EKS 1.30+; "" uses the cluster default

Confirm the claim binds before waiting on the workloads - kubectl get pvc -n agledger should show the -pgdata claim Bound, not Pending. Choose one database path, not both; the steps below use an external PostgreSQL.

$ kubectl create namespace agledger
$ kubectl apply -f postgres.yaml          # your PostgreSQL Deployment + Service
$ kubectl rollout status deploy/agledger-postgres -n agledger --timeout=120s
deployment "agledger-postgres" successfully rolled out

values.yaml:

image:
  digest: "sha256:75c9590f353c063735728229f07f941fd063b1ab242caee900406ddc1eb5d20c"  # 1.4.0
database:
  # sslmode is required under nodeEnv=production. Prefer sslmode=verify-full against a TLS-enabled
  # database with config.nodeExtraCaCerts pointing at its CA — `require` encrypts but does NOT
  # validate the server certificate (the CA bundle is only consulted under verify-ca/verify-full).
  # sslmode=disable only suits a local, non-TLS PostgreSQL like the in-cluster one here.
  externalUrl: "postgresql://agledger:<db-password>@agledger-postgres.agledger.svc.cluster.local:5432/agledger?sslmode=disable"
config:
  externalUrl: "https://agledger.k8s.example"   # the Server's signed issuer identity; use your host
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: agledger-selfsigned
  hosts:
    - host: agledger.k8s.example
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: agledger-tls
      hosts: [agledger.k8s.example]

Write the key to a file rather than passing it inline. --set puts the value in helm's argv, where ps shows it to every other user on the machine for the length of the install, and the vault signing key is the private key every record is signed with.

$ umask 077 && printf %s '<vault-key>' > vault-key
$ helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \
    --version 1.4.0 --namespace agledger \
    --values values.yaml --set-file secrets.vaultSigningKey=vault-key
NAME: agledger
STATUS: deployed
REVISION: 1
$ rm vault-key

The chart runs schema migrations as a pre-install hook, then starts the API and worker. A fresh install applies every shipped migration in a single Job; re-running the Job against a database already at that head applies nothing and reports a migration count of 0, so a repeated helm upgrade is safe.

On the external-database path the migrate Job is a pre-install/pre-upgrade hook, so a migration pod that cannot schedule fails the whole install or upgrade. If your API pods are pinned to a node class (architecture, taints), set migrate.nodeSelector / migrate.tolerations / migrate.affinity to match. Each is empty by default and falls back to the matching api.* value, so an install that pins only the API is already consistent.

Sizing an upgrade window

A migration that adds a column and builds an index does both in one transaction, which holds ACCESS EXCLUSIVE on the table until it commits: reads as well as writes stop for the duration. The cost is driven by how wide your criteria documents are, not by row count alone, and each migration file's header carries the measured numbers for that migration. Read it before upgrading a mature install.

migrate.extraEnv delivers the knobs to the migration Job (it is a separate list from the top-level extraEnv, which goes to the API and worker):

migrate:
  extraEnv:
    # Default 15min. Raise past your expected index build time; if it fires,
    # the migration rolls back cleanly and you retry with a larger value.
    - name: MIGRATION_STATEMENT_TIMEOUT
      value: "60min"
    # Unset by default, meaning the server's own value. Raising it shortens
    # the window rather than just permitting a longer one. PostgreSQL takes
    # this per maintenance operation, so a large value can push a small
    # containerized PostgreSQL into the OOM killer.
    - name: MIGRATION_MAINTENANCE_WORK_MEM
      value: "256MB"

Do not set DATABASE_URL here if you also use secrets.databaseUrlMigrate: the chart refuses to render rather than let one silently override the other.

On Compose the same knobs are read from .env; see the migration section of .env.example.

$ kubectl rollout status deploy/agledger-agledger-chart-api -n agledger --timeout=180s
deployment "agledger-agledger-chart-api" successfully rolled out
$ kubectl rollout status deploy/agledger-agledger-chart-worker -n agledger --timeout=120s
deployment "agledger-agledger-chart-worker" successfully rolled out

4. Create the platform API key

The platform key is the first credential; you use it to provision organizations and agents. It is printed once.

$ kubectl exec deploy/agledger-agledger-chart-api -n agledger -- \
    /nodejs/bin/node dist/scripts/init.js --non-interactive
  ✓ API_KEY_SECRET found in environment
  ✓ VAULT_SIGNING_KEY found in environment
  ✓ AGLEDGER_FEDERATION_SIGNING_KEY generated (Ed25519)
  ✓ AGLEDGER_FEDERATION_ENCRYPTION_KEY generated (X25519)

  ⚠  Read-only filesystem: .env could not be written.
     3 secret value(s) already set in this environment are shown by name only.
     The secrets printed below were generated by this run and exist nowhere else. Capture them.

  --- .env content ---
  ...
  DATABASE_URL=<already set in this environment; not reprinted>
  API_KEY_SECRET=<already set in this environment; not reprinted>
  VAULT_SIGNING_KEY=<already set in this environment; not reprinted>
  AGLEDGER_FEDERATION_SIGNING_KEY=<printed in full>
  AGLEDGER_FEDERATION_ENCRYPTION_KEY=<printed in full>
  ---

  ✓ Platform API key created (ID: ...)
  ✓ Database query OK
  │ Platform Key: agl_plt_<store-this-securely>

The chart runs the container with a read-only root filesystem, so init.js cannot write a .env and prints one to stdout instead. Everything you supplied in step 2 is shown by name only: your database password, API_KEY_SECRET and VAULT_SIGNING_KEY are already in your Secret and are not reprinted.

Two things in that output are secret and exist nowhere else: the platform key, and the federation keypair this run generated. Federation keys are optional. Capture them into your Secret as AGLEDGER_FEDERATION_SIGNING_KEY and AGLEDGER_FEDERATION_ENCRYPTION_KEY if you intend to federate; otherwise ignore them, and a later run generates a fresh pair. Either way, do not run this step in a CI job whose logs are retained.

5. The readiness gate: up and signing

A Server is ready when it is healthy and signing with your key.

$ kubectl port-forward svc/agledger-agledger-chart -n agledger 3001:80 &
$ curl -s http://localhost:3001/health
{"status":"ok","version":"1.4.0","timestamp":"..."}
$ curl -s http://localhost:3001/health/ready
{"status":"ready","version":"1.4.0","timestamp":"..."}

Confirm the active signing key is the one you generated in step 2 - same fingerprint:

$ curl -s http://localhost:3001/v1/verification-keys
{
  "data": [{ "keyId": "dcd0573755f045c6", "algorithm": "Ed25519",
             "publicKey": "MCowBQYDK2VwAyEAPInxjno26azIT9i6GVqJag9QuCJFMoDG96iljTd8fHo=",
             "status": "active" }],
  "envelope": "COSE_Sign1", "signatureAlgorithm": "Ed25519"
}

keyId matches the keygen fingerprint: the Server is signing with your key, COSE_Sign1 / Ed25519.

With no license applied the Server runs as Developer Edition (all features enabled):

$ curl -s -H "Authorization: Bearer <platform-key>" http://localhost:3001/v1/admin/license
{"validity":"unlicensed","tier":"developer",
 "features":["custom_schemas","expression_engine","delegation_chains","audit_export",
             "compliance_reports","encrypted_mode","entity_references","proposals","federation"],
 "source":"none"}

6. TLS

With ingress-nginx and cert-manager, the chart's Ingress requests a certificate and serves it. A self-signed ClusterIssuer is portable and needs no public DNS; for production use an ACME (Let's Encrypt) or your-CA issuer.

$ kubectl get certificate -n agledger
NAME           READY   SECRET         AGE
agledger-tls   True    agledger-tls   ...
$ curl -sk https://agledger.k8s.example/health   # through the ingress
{"status":"ok","version":"1.4.0","timestamp":"..."}

External database

The in-cluster PostgreSQL above is the simplest start. To use a managed PostgreSQL, point database.externalUrl at it with sslmode=verify-full and set config.nodeExtraCaCerts to its CA bundle (require encrypts without validating the server certificate). External-database licensing is per database instance. For Amazon Aurora / RDS specifically, see the AWS install guide.

The migration role needs superuser. The schema installs an event trigger, agledger_block_audit_drop, which is what stops the audit chain being dropped, and CREATE EVENT TRIGGER is superuser-only in PostgreSQL. A managed database hands you an owner role with CREATEDB and not this, so migrations fail with permission denied to create event trigger unless you grant it:

| Provider | Grant | |---|---| | Amazon RDS / Aurora | GRANT rds_superuser TO <role>; | | Google Cloud SQL | GRANT cloudsqlsuperuser TO <role>; | | Azure Database for PostgreSQL | GRANT azure_pg_admin TO <role>; | | Self-managed | ALTER ROLE <role> SUPERUSER; |

To keep the running Server least-privilege, grant it to a separate role and give that role to migrations only: set secrets.databaseUrlMigrate (Helm) or DATABASE_URL_MIGRATE (Compose) to its URL, and leave database.externalUrl / DATABASE_URL as the DML role. install.sh checks this before it pulls anything and names the grant for your provider.

OpenShift

Add --set openshift.enabled=true to the helm install above. Everything else on this page is unchanged.

openshift.enabled was introduced in chart 1.4.0, the version this page pins. On an earlier chart the key does not exist, and Helm accepts an unknown --set key without complaint, so the flag is silently ignored, the pod-level uid request stays, and you get exactly the admission error below.

helm install agledger oci://registry-1.docker.io/agledger/agledger-chart \
  --version 1.4.0 --namespace agledger --create-namespace \
  --set openshift.enabled=true \
  --values values.yaml

Without it, admission rejects every pod before it starts:

unable to validate against any security context constraint:
runAsUser: Invalid value: 65532: must be in the ranges: [1000700000, ...]

OpenShift's default restricted-v2 SCC assigns each namespace its own uid range and admits pods with MustRunAsRange. The chart's stock pod security context asks for uid/gid 65532, which is outside that range on essentially every cluster. The flag drops that block from all three workloads (the api Deployment, the worker Deployment, and the migrate Job) and lets the platform assign a uid instead. The bundled PostgreSQL needs no equivalent: it requests no security context at all, which is exactly what restricted-v2 admits.

What goes is runAsNonRoot: true alongside the two uid/gid keys, so non-root stops being asserted in the manifest. It still holds, by a different guarantee: on OpenShift restricted-v2 enforces it directly along with a default seccomp profile, and elsewhere it rests on the image's own USER 65532:65532. Every container-level control is untouched either way, so each container keeps its read-only root filesystem, no privilege escalation, and all capabilities dropped.

The image supports an arbitrary uid: it declares USER 65532 for plain Kubernetes and Docker, but nothing in it is owned by or hardcoded to that uid, its files are world-readable, and it needs no passwd entry.

./scripts/helm-install.sh sets the flag for you when it sees the security.openshift.io API group on the cluster, unless you already set openshift.enabled yourself, whether by --set or in a values file it was passed. The chart also ships values-openshift.yaml, which sets the same flag for anyone who prefers a values file, though reaching it means helm pull --untar first:

helm pull oci://registry-1.docker.io/agledger/agledger-chart --version <ver> --untar
helm install agledger ./agledger-chart -n agledger \
  -f ./agledger-chart/values-openshift.yaml \
  -f values.yaml

What was measured. Under the constraints restricted-v2 imposes on a container (--user 1000700000:0 --read-only --cap-drop ALL --security-opt no-new-privileges), the shipped image applies its migrations, boots healthy, serves the unauthenticated discovery surfaces, mints a platform key, and notarizes a record, so signing works under an assigned uid. The rendered manifests were checked directly: the flag removes the pod-level uid request from all three workloads and leaves container hardening identical to a stock render. Admission itself was not exercised on a live OpenShift cluster; that part follows from the SCC rules.

Run more than one Server

You can run more than one Server and link them so chains reference records across Servers (we call linking Servers federation). Each Server is a full, independent install of this guide with its own database, database role, and signing key; linking is configured after both are healthy. If you host several Servers on one PostgreSQL cluster, give each its own role: PostgreSQL roles are cluster-wide, so Servers sharing the agledger_app role share one credential across all of their databases.

Air-gapped install

Nothing in install or runtime depends on agledger.ai, Docker Hub, or npm at runtime. For a restricted network, mirror the image (agledger/agledger:1.4.0) and chart (agledger/agledger-chart:1.4.0) into your internal registry, the cosign verification bundle ships with each GitHub release; see the install repository's air-gap guide for the offline cosign verify --new-bundle-format flow, and set image.repository (and image.pullSecrets) to your registry. The Server makes no outbound calls except the webhook and federation endpoints you configure.

Uninstall

$ helm uninstall agledger -n agledger
release "agledger" uninstalled

This leaves your database and signing key intact, so a reinstall against the same database and key resumes the same chain.

Next

Keep the Server healthy over time with Day-2 Operations, and set up a backup and recovery routine.