> Markdown version of https://agledger.ai/docs/guides/federation/ > Full index of this site for AI assistants: https://agledger.ai/llms.txt # Federating Two Servers Federation links two AGLedger Servers so that each sees the outcome of the other's shared records. The Servers stay separate: - **Separate storage and keys.** Each keeps its own database and its own signing key. - **Signed messages.** Every message between them is signed by the Server that sent it. - **Structure only.** Only the structure of a shared record crosses, never its content. The examples link Server A (`https://agledger.meridian.example.com`) and Server B (`https://agledger.cascade.example.com`), run by two different companies. ## 1. What each side needs first On each Server: - **An instance id.** `AGLEDGER_INSTANCE_ID` must be a UUID, and the installer sets one. It is the id the Server presents to peers, and it must never change. - **A federation signing key.** Generate one with the script that ships in the install folder, then recreate the containers: ```bash ./scripts/generate-federation-keys.sh >> compose/.env (cd compose && docker compose up -d --force-recreate) ``` The script prints the public half as a comment (`# Signing public key: MCowBQ...`): that is what you hand the other operator. On Kubernetes, put `AGLEDGER_FEDERATION_SIGNING_KEY` in the release's Secret. - **A URL the other Server can reach.** It must be `https://`. A Server that terminates TLS at a load balancer, or peers over a private network, also sets `SSRF_ALLOW_PLAIN_HTTP=true`. A peer at a private address also needs its range in `SSRF_ALLOW_CIDRS`. - **Two Compose installs on one host** do not share a network. Do not attach one to the other's Compose network: both have a service named `postgres`, and the API would then reach the other install's database. Create one network for federation and join both installs' API and worker to it in each `compose/docker-compose.override.yml`, which survives container recreates where a `docker network connect` does not: ```bash docker network create agledger-federation ``` ```yaml networks: agledger-federation: { external: true } services: agledger-api: { networks: [default, agledger-federation] } agledger-worker: { networks: [default, agledger-federation] } ``` Then `(cd compose && docker compose up -d)` on each side. Address the peer by its API container's name (`agledger-b-agledger-api-1`), not by `agledger-api`, which both installs answer to on the shared network. - **A platform key.** Minting tokens, listing peers, the DLQ, revoke and delete are all platform-only. An org-admin key can only read `GET /federation/v1/admin/instance`. Check that the Server is ready: ```bash curl -s "$AGLEDGER_API_URL/federation/v1/admin/instance" -H "Authorization: Bearer $PLATFORM_KEY" ``` ```json { "instanceId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "configured": true, "signingPublicKey": "MCowBQYDK2VwAyEAHZUt8dp5Muq0xgO2LGOtdsrQhorMLJg6KVgy8MYZgAE=" } ``` `configured: false` means the instance id or the signing key is missing. ## 2. Peer the two Servers A handshake registers one Server on the other, so peering takes two handshakes, one in each direction. For each direction the operators exchange, out of band (a secrets manager or encrypted email): | From the calling Server's operator, first | From the receiving Server's operator, in return | |---|---| | its `instanceId` | a peering token minted for that `instanceId` | | its signing public key | | | its URL | | **Direction 1: register A on B.** B's operator mints a token for A's `instanceId`, naming the B org that A's shared records project into (`GET /v1/admin/orgs` lists it): ```bash curl -s -X POST "https://agledger.cascade.example.com/federation/v1/admin/peering-tokens" \ -H "Authorization: Bearer $B_PLATFORM_KEY" -H 'Content-Type: application/json' \ -d '{"peerHubId":"'"$A_INSTANCE_ID"'","boundOrgId":"'"$B_ORG_ID"'","label":"Meridian Components, handshake 2026-09"}' ``` ```json { "peeringToken": "aO36wq...", "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "boundOrgId": "01a0d9f2-5e61-7c1e-9d0a-3f5b2c8e4a17", "expiresAt": "2026-10-03T05:16:15.315Z", "label": "Meridian Components, handshake 2026-09" } ``` The token is single-use, expires after seven days, and admits only the `instanceId` it was minted for. A's operator then calls B. This call takes no API key: the token is the credential. ```bash curl -s -X POST "https://agledger.cascade.example.com/federation/v1/peer" \ -H 'Content-Type: application/json' \ -d '{ "peeringToken": "'"$TOKEN_FROM_B"'", "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "peerUrl": "https://agledger.meridian.example.com", "signingPublicKey": "MCowBQYDK2VwAyEAHZUt8dp5Muq0xgO2LGOtdsrQhorMLJg6KVgy8MYZgAE=" }' ``` ```json { "peered": true, "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "status": "active", "serverHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf", "serverSigningPublicKey": "MCowBQYDK2VwAyEA..." } ``` The response is B's half of the reverse handshake: `serverHubId` is B's `instanceId`, which A's operator names when minting B's token, and `serverSigningPublicKey` is B's key. Its `nextSteps` (trimmed here) spell out that reverse step. What the body carries: - **`peerHubId`** is A's `instanceId`, the id the token was minted for. Any other value is refused (422) and leaves the token unused. - **No org.** The token carries the B org that A's records project into, so the caller cannot choose it. **Direction 2: register B on A.** Repeat with the Servers swapped: A's operator mints the token for B's `instanceId`, and B's operator calls A's `/federation/v1/peer` with its own signing key. Confirm on each side: ```bash curl -s "https://agledger.meridian.example.com/federation/v1/admin/peers" -H "Authorization: Bearer $A_PLATFORM_KEY" ``` ```json { "data": [ { "peerHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf", "peerUrl": "https://agledger.cascade.example.com", "status": "active", "lastDeliveryAt": null, "lastDeliveryError": null, "consecutiveDeliveryFailures": 0 } ] } ``` Handshake errors: | Situation | Answer | |---|---| | Token unknown or minted on another Server; or reused, expired or voided, for an `instanceId` not registered here | 401, "Peering token is unknown, already consumed, or expired", with a `recoveryHint` to ask for a fresh one. The token is checked first, so nothing else about the request is answered without one. A handshake that peers an `instanceId` voids every other token minted for it, so a spare left over after that peer is revoked and deleted answers this 401. | | Live token minted for a different `instanceId` | 422; the token stays unused | | Token minted for an `instanceId` that is now registered | 409, with a `recoveryHint` naming the revoke (if the peer is active) or the delete (if it is revoked) to run before a fresh token can peer it | | Minting a token for an `instanceId` already registered | 409 on the mint, naming the revoke and delete to run first | | The receiving Server has no `AGLEDGER_FEDERATION_SIGNING_KEY` | 409, with a `recoveryHint` saying its operator has to enable federation there; the token is not spent, so the handshake can be run again once they have | | Minting a token on a Server with no `AGLEDGER_FEDERATION_SIGNING_KEY` | 409 on the mint, naming the steps to enable federation first | To change a peer's URL or key, revoke it, delete it, and run the handshake again with a fresh token. ## 3. Share a record A record is sent to every active peer when it reaches a terminal state (fulfilled, failed, recorded, cancelled, rejected, expired), or when its Settlement Signal webhook fires. The share gate decides whether it goes, and the most specific setting wins: 1. **On the record:** `"share": true` or `false` on `POST /v1/records`. 2. **On the type:** `defaultShare` on `POST /v1/schemas`. 3. **Server-wide:** `AGLEDGER_DEFAULT_SHARE` (default `true`). A record created with `"share": false` stays on its own Server. The peer answers 404 for it, and the record reads `sharedToPeers: []`. Its own chain still records everything: sharing is a decision about the wire, not about the chain. **Register shared types identically on both Servers.** Every federation message names the record's contract type with a digest of its registration. A receiver that has the same type and version registered with different content refuses the message (422, not retried). On the sending side, each record reports its delivery in `federationStatus`: `pending`, `delivered`, `partial` (some peers delivered, others not yet or not at all) or `failed`: ```json { "federationStatus": "delivered", "sharedToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"], "settlementSignal": { "recommendation": "SETTLE", "deliveredToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"], "pendingToPeers": [], "failedToPeers": [] } } ``` **What crosses, and what never does:** | Crosses | Never crosses | |---|---| | The signed state and the Settlement Signal payload, with its `reasonCode` and failing rule ids | Criteria values, and the signal's free-text reason (a principal's verdict notes) | | The agent ids and the contract type | Completion evidence | | Lineage and correlation ids (`correlationId`, `projectRef`, `externalTaskId`, `platformRef`, `parentRecordId`, `rootRecordId`, `chainDepth`) | The originating Server's audit entries and signed statements | | `riskClassification`, and `euAiActDomain` when the class is `high` | Human-oversight text, dispute evidence | ## 4. What arrives on the other side On B, the shared record appears under the same id as a read-only projection: ```bash curl -s "https://agledger.cascade.example.com/v1/records/$RECORD_ID" -H "Authorization: Bearer $B_ADMIN_KEY" ``` ```json { "id": "01a0dc1a-876a-7189-95b1-541f46334838", "status": "FULFILLED", "type": "principal-gate-generic-v1", "correlationId": "PO-88120", "criteria": {}, "federationStatus": "inbound", "settlementSignal": { "recommendation": "SETTLE", "source": "inbound", "receivedFrom": { "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "receivedAt": "2026-09-26T05:05:23.357Z" } } } ``` **How B sees it:** - **The chain.** B's chain for the record carries its own entries: `FEDERATION_STATE_SYNC` for the state, and `FEDERATION_SIGNAL_RECEIVED` for the signal. - **Records.** Record search has no filter for projections (`?federationStatus=` is refused as an unknown parameter). Subscribe to the webhook events below, or read `federationStatus: "inbound"` on each record. - **Agents.** A's agents appear on B the first time a message names them. No agent directory is exchanged. A new record from A that names one of B's own agents, or another peer's, is refused (404) and nothing is written. **Changes stay with the originating Server.** B's copy is a view of A's record. Every call that would change it on B (cancel, transition, PATCH, a verdict, a new dispute) is refused for B's org-admin and platform keys with 422, `allowedActions: []` and a `recoveryHint` naming A, and the record's `nextSteps` on B offer only reads. A counterparty that disagrees disputes the record on A. B's own compliance records and AI impact assessments on the projection stay open. **Webhooks on the receiving side.** Subscribe B's systems with an org-admin key: ```bash curl -s -X POST "https://agledger.cascade.example.com/v1/webhooks" \ -H "Authorization: Bearer $B_ADMIN_KEY" -H 'Content-Type: application/json' \ -d '{"url":"https://erp.cascade.example.com/agledger","eventTypes":["federation.record.state_changed","federation.settlement.signal","signal.received","federation.dispute"]}' ``` ```json { "type": "federation.settlement.signal", "data": { "recordId": "01a0dc1a-876a-7189-95b1-541f46334838", "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "recommendation": "SETTLE", "outcome": "accept", "reasonCode": "PRINCIPAL_ACCEPT", "failingRuleIds": null, "outcomeHash": "b933c34d66d58f0388649cae48e9902101e03723bd63fe9c0ca7f89f7b7caf14", "validUntil": "2026-09-27T05:05:21.415Z", "counterSignature": null } } ``` - **`federation.record.state_changed`** carries the state, the agents and the lineage. Per-state events also exist, such as `record.federation_fulfilled`. - **`signal.received`** carries the same signal as `federation.settlement.signal`. - **Acting on it.** As on a single Server, treat a delivery as the prompt to read the record, not as the state itself. **Disputes.** A dispute opened on A reaches B: ```json { "status": "FULFILLED", "hasDispute": true, "disputeStatus": "EVIDENCE_WINDOW", "disputeId": "01a0dc1a-a50f-7c04-a544-a1a9f978bcd4", "disputeCount": 1 } ``` - **The status is not flipped.** B keeps its projected status, so "FULFILLED at time T" stays true on B's books. - **Only the structure crosses.** `grounds`, the dispute status, the initiator's role and the outcome cross; evidence stays on A. - **B cannot act on it.** `GET /v1/records/{id}/dispute` on B answers 404, because the dispute runs on A. - **Webhooks and opt-out.** Each step fires `federation.dispute` on B, with `action` `opened`, `resolved` or `withdrawn`. A type registered with `federateDisputes: false` keeps disputes local. ### Counter-signed settlement (co-sign) When a Settlement Signal payload should carry both companies' signatures, register the type with `"coSignRequired": true`, identically on both Servers: ```bash curl -s -X POST "$AGLEDGER_API_URL/v1/schemas" \ -H "Authorization: Bearer $ADMIN_KEY" -H 'Content-Type: application/json' \ -d '{"type":"supplier-payment-release-v1","coSignRequired":true,"defaultGateMode":"principal", "recordSchema":{"type":"object","required":["summary"],"properties":{"summary":{"type":"string"},"amount":{"type":"number"},"poRef":{"type":"string"}}}, "completionSchema":{"type":"object","required":["summary"],"properties":{"summary":{"type":"string"}}}}' ``` When a record of that type settles on A: 1. A asks B to counter-sign, and B checks the outcome against its own projection before signing it. 2. A attaches B's counter-signature to the signal it sends. A's record then reads: ```json { "coSignRequired": true, "coSignStatus": "succeeded", "settlementSignal": { "deliveredToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"], "coSignPeers": [ { "peerHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf", "status": "succeeded", "counterSignature": "7125b25a6adc572a..." } ] } } ``` - **Both chains prove it.** A's chain carries `FEDERATION_CO_SIGN_REQUESTED` and `FEDERATION_CO_SIGN_COUNTERSIGNED`, and B's carries `FEDERATION_CO_SIGN_RESPONDED`. Either chain alone proves the bilateral commitment offline. - **B holds A to it.** A signal that reaches B without a counter-signature, for a record whose type B registered with `coSignRequired: true`, is refused (422, not retried) and nothing is recorded. B reads the record's own type and its latest registration of it; a `schemaRef` on the signal naming another type, or an earlier version that needed no co-sign, does not waive it. That happens when A's registration of the type lacks `coSignRequired`: register it identically on both Servers. - **A peer that does not have the type.** `AGLEDGER_FEDERATION_COSIGN_PEER_POLICY` decides: - `lenient`, the default: deliver the plain signal and record a `FEDERATION_CO_SIGN_WAIVED` entry. - `strict`: fail the delivery to that peer. Use `strict` when the two keys belong to different companies. ## 5. When deliveries fail **A peer that is down.** Deliveries retry on their own. While B was stopped, A reported: ```json { "federationStatus": "pending", "settlementSignal": { "deliveredToPeers": [], "pendingToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"] } } ``` `GET /federation/v1/admin/peers` showed the cause: ```json { "status": "active", "lastDeliveryError": "getaddrinfo ENOTFOUND agledger-b-agledger-api-1", "consecutiveDeliveryFailures": 2 } ``` A federation leg that fails is retried 8 times, at roughly doubling gaps, over roughly 42 to 85 minutes, and then dead-letters. When B came back inside that window, the record was delivered on the next retry, and `federationStatus: "delivered"` followed with no operator action. Until a leg dead-letters, `system-health` reads `healthy`; `consecutiveDeliveryFailures` on the peer listing is what shows a peer failing in the meantime. **Legs that exhaust their retries** go to the federation DLQ: ```bash curl -s "$AGLEDGER_API_URL/federation/v1/admin/dlq" -H "Authorization: Bearer $PLATFORM_KEY" curl -s -X POST "$AGLEDGER_API_URL/federation/v1/admin/dlq/recover" \ -H "Authorization: Bearer $PLATFORM_KEY" -H 'Content-Type: application/json' -d '{"dryRun":true}' ``` - **Recovery.** A background sweep retries DLQ jobs every 10 minutes, for 24 hours after a leg dead-letters. `recover` replays them now, and is the only way past that horizon (it reaches back 30 days). Fix the cause first: the peer's URL, its registration, a missing co-sign type. - **An empty DLQ does not mean nothing failed.** A leg the peer refused permanently never enters the DLQ. It shows on the record as `federationStatus: "failed"`, with the peer in `failedToPeers` and a `FEDERATION_DELIVERY_FAILED` chain entry naming the reason. **Key and clock problems:** | Symptom | Cause | |---|---| | `lastDeliveryError: "401 from peer (not retryable): ... Peer hub is revoked"` | The peer revoked its registration of this Server. Its operator deletes the revoked row and mints a new token; then run the handshake again. | | `lastDeliveryError: "401 from peer (not retryable): ... Unknown peer hub"` | The peer has no registration for this Server's `instanceId`: it was deleted, or the handshake used the wrong `peerHubId`. Run the handshake again. | | 400 `Federation timestamp too old (skew: ...)` | The two clocks differ by more than the allowed window (`AGLEDGER_CLOCK_SKEW_MAX_MS`, 5 minutes by default; a Server refuses to start with more than 15). Fix NTP on the Server whose clock is off. | | 401 `Missing federation auth headers` | The request was not sent by a peer Server. Federation endpoints accept only signed Server-to-Server calls. | ## 6. Revoke and remove a peer Revoke stops the peer from authenticating immediately: ```bash curl -s -X POST "https://agledger.cascade.example.com/federation/v1/admin/peers/$A_INSTANCE_ID/revoke" \ -H "Authorization: Bearer $B_PLATFORM_KEY" -H 'Content-Type: application/json' \ -d '{"reason":"Supplier agreement ended 2026-09-30"}' ``` ```json { "revoked": true } ``` **After a revoke:** - **Already shared records stay.** Projections on B remain readable, including their dispute state, and the agents they name still resolve. Those agents stay marked as A's even after a delete, so they never become B's own. - **New records stop arriving.** On A, a record that terminates after the revoke reads `federationStatus: "failed"`, and A's peer row shows `lastDeliveryError: "401 from peer (not retryable): ... Peer hub is revoked"` (`Unknown peer hub` once B deletes the row). - **B stops sending to A as well.** A Server sends only to peers it lists as active. - **A still lists B as active** until A's operator revokes B too. Each side revokes its own registration of the other. Delete removes a revoked peer so that it can register again from scratch with a fresh token minted for its `instanceId`: ```bash curl -s -X DELETE "https://agledger.cascade.example.com/federation/v1/admin/peers/$A_INSTANCE_ID" \ -H "Authorization: Bearer $B_PLATFORM_KEY" ``` ```json { "deleted": true, "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c" } ``` Deleting an active peer is refused (409): revoke it first. Field shapes for every call above are in the [API reference](/api/). ## Air-gapped Two Servers on one private network federate the same way, with no call to anything outside it: - **Tokens, instance ids and public keys** move between the operators out of band. - **Each Server allows the other's address.** Put its range in `SSRF_ALLOW_CIDRS`. - **Plain `http://`** needs `SSRF_ALLOW_PLAIN_HTTP=true`, which is meant for TLS terminated at a load balancer.