Federating Two Servers

Federation links two AGLedger Servers so that each sees the outcome of the other's shared records. The Servers stay separate:

• Separate storage and keys. Each keeps its own database and its own signing key.
• Signed messages. Every message between them is signed by the Server that sent it.
• Structure only. Only the structure of a shared record crosses, never its content.

The examples link Server A (https://agledger.meridian.example.com) and Server B (https://agledger.cascade.example.com), run by two different companies.

1. What each side needs first

On each Server:

• An instance id. AGLEDGER_INSTANCE_ID must be a UUID, and the installer sets one. It is the id the Server presents to peers, and it must never change.

• A federation signing key. Generate one with the script that ships in the install folder, then recreate the containers:

  ./scripts/generate-federation-keys.sh >> compose/.env
  (cd compose && docker compose up -d --force-recreate)
  

  Copy

  The script prints the public half as a comment (# Signing public key: MCowBQ...): that is what you hand the other operator. On Kubernetes, put AGLEDGER_FEDERATION_SIGNING_KEY in the release's Secret.

• A URL the other Server can reach. It must be https://. A Server that terminates TLS at a load balancer, or peers over a private network, also sets SSRF_ALLOW_PLAIN_HTTP=true. A peer at a private address also needs its range in SSRF_ALLOW_CIDRS.

• Two Compose installs on one host do not share a network. Do not attach one to the other's Compose network: both have a service named postgres, and the API would then reach the other install's database. Create one network for federation and join both installs' API and worker to it in each compose/docker-compose.override.yml, which survives container recreates where a docker network connect does not:

  docker network create agledger-federation
  

  Copy

  networks:
    agledger-federation: { external: true }
  services:
    agledger-api: { networks: [default, agledger-federation] }
    agledger-worker: { networks: [default, agledger-federation] }
  

  Copy

  Then (cd compose && docker compose up -d) on each side. Address the peer by its API container's name (agledger-b-agledger-api-1), not by agledger-api, which both installs answer to on the shared network.

• A platform key. Minting tokens, listing peers, the DLQ, revoke and delete are all platform-only. An org-admin key can only read GET /federation/v1/admin/instance.

Check that the Server is ready:

curl -s "$AGLEDGER_API_URL/federation/v1/admin/instance" -H "Authorization: Bearer $PLATFORM_KEY"

{ "instanceId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "configured": true,
  "signingPublicKey": "MCowBQYDK2VwAyEAHZUt8dp5Muq0xgO2LGOtdsrQhorMLJg6KVgy8MYZgAE=" }

configured: false means the instance id or the signing key is missing.

2. Peer the two Servers

A handshake registers one Server on the other, so peering takes two handshakes, one in each direction. For each direction the operators exchange, out of band (a secrets manager or encrypted email):

Direction 1: register A on B. B's operator mints a token for A's instanceId, naming the B org that A's shared records project into (GET /v1/admin/orgs lists it):

curl -s -X POST "https://agledger.cascade.example.com/federation/v1/admin/peering-tokens" \
  -H "Authorization: Bearer $B_PLATFORM_KEY" -H 'Content-Type: application/json' \
  -d '{"peerHubId":"'"$A_INSTANCE_ID"'","boundOrgId":"'"$B_ORG_ID"'","label":"Meridian Components, handshake 2026-09"}'

{ "peeringToken": "aO36wq...", "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c",
  "boundOrgId": "01a0d9f2-5e61-7c1e-9d0a-3f5b2c8e4a17", "expiresAt": "2026-10-03T05:16:15.315Z",
  "label": "Meridian Components, handshake 2026-09" }

The token is single-use, expires after seven days, and admits only the instanceId it was minted for. A's operator then calls B. This call takes no API key: the token is the credential.

curl -s -X POST "https://agledger.cascade.example.com/federation/v1/peer" \
  -H 'Content-Type: application/json' \
  -d '{
    "peeringToken": "'"$TOKEN_FROM_B"'",
    "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c",
    "peerUrl": "https://agledger.meridian.example.com",
    "signingPublicKey": "MCowBQYDK2VwAyEAHZUt8dp5Muq0xgO2LGOtdsrQhorMLJg6KVgy8MYZgAE="
  }'

{ "peered": true, "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c", "status": "active",
  "serverHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf",
  "serverSigningPublicKey": "MCowBQYDK2VwAyEA..." }

The response is B's half of the reverse handshake: serverHubId is B's instanceId, which A's operator names when minting B's token, and serverSigningPublicKey is B's key. Its nextSteps (trimmed here) spell out that reverse step.

What the body carries:

• peerHubId is A's instanceId, the id the token was minted for. Any other value is refused (422) and leaves the token unused.
• No org. The token carries the B org that A's records project into, so the caller cannot choose it.

Direction 2: register B on A. Repeat with the Servers swapped: A's operator mints the token for B's instanceId, and B's operator calls A's /federation/v1/peer with its own signing key.

Confirm on each side:

curl -s "https://agledger.meridian.example.com/federation/v1/admin/peers" -H "Authorization: Bearer $A_PLATFORM_KEY"

{ "data": [ { "peerHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf",
              "peerUrl": "https://agledger.cascade.example.com", "status": "active",
              "lastDeliveryAt": null, "lastDeliveryError": null, "consecutiveDeliveryFailures": 0 } ] }

Handshake errors:

To change a peer's URL or key, revoke it, delete it, and run the handshake again with a fresh token.

3. Share a record

A record is sent to every active peer when it reaches a terminal state (fulfilled, failed, recorded, cancelled, rejected, expired), or when its Settlement Signal webhook fires. The share gate decides whether it goes, and the most specific setting wins:

1. On the record: "share": true or false on POST /v1/records.
2. On the type: defaultShare on POST /v1/schemas.
3. Server-wide: AGLEDGER_DEFAULT_SHARE (default true).

A record created with "share": false stays on its own Server. The peer answers 404 for it, and the record reads sharedToPeers: []. Its own chain still records everything: sharing is a decision about the wire, not about the chain.

Register shared types identically on both Servers. Every federation message names the record's contract type with a digest of its registration. A receiver that has the same type and version registered with different content refuses the message (422, not retried).

On the sending side, each record reports its delivery in federationStatus: pending, delivered, partial (some peers delivered, others not yet or not at all) or failed:

{ "federationStatus": "delivered", "sharedToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"],
  "settlementSignal": { "recommendation": "SETTLE",
                        "deliveredToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"],
                        "pendingToPeers": [], "failedToPeers": [] } }

What crosses, and what never does:

4. What arrives on the other side

On B, the shared record appears under the same id as a read-only projection:

curl -s "https://agledger.cascade.example.com/v1/records/$RECORD_ID" -H "Authorization: Bearer $B_ADMIN_KEY"

{
  "id": "01a0dc1a-876a-7189-95b1-541f46334838",
  "status": "FULFILLED",
  "type": "principal-gate-generic-v1",
  "correlationId": "PO-88120",
  "criteria": {},
  "federationStatus": "inbound",
  "settlementSignal": { "recommendation": "SETTLE", "source": "inbound",
                        "receivedFrom": { "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c",
                                          "receivedAt": "2026-09-26T05:05:23.357Z" } }
}

How B sees it:

• The chain. B's chain for the record carries its own entries: FEDERATION_STATE_SYNC for the state, and FEDERATION_SIGNAL_RECEIVED for the signal.
• Records. Record search has no filter for projections (?federationStatus= is refused as an unknown parameter). Subscribe to the webhook events below, or read federationStatus: "inbound" on each record.
• Agents. A's agents appear on B the first time a message names them. No agent directory is exchanged. A new record from A that names one of B's own agents, or another peer's, is refused (404) and nothing is written.

Changes stay with the originating Server. B's copy is a view of A's record. Every call that would change it on B (cancel, transition, PATCH, a verdict, a new dispute) is refused for B's org-admin and platform keys with 422, allowedActions: [] and a recoveryHint naming A, and the record's nextSteps on B offer only reads. A counterparty that disagrees disputes the record on A. B's own compliance records and AI impact assessments on the projection stay open.

Webhooks on the receiving side. Subscribe B's systems with an org-admin key:

curl -s -X POST "https://agledger.cascade.example.com/v1/webhooks" \
  -H "Authorization: Bearer $B_ADMIN_KEY" -H 'Content-Type: application/json' \
  -d '{"url":"https://erp.cascade.example.com/agledger","eventTypes":["federation.record.state_changed","federation.settlement.signal","signal.received","federation.dispute"]}'

{ "type": "federation.settlement.signal",
  "data": { "recordId": "01a0dc1a-876a-7189-95b1-541f46334838",
            "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c",
            "recommendation": "SETTLE", "outcome": "accept",
            "reasonCode": "PRINCIPAL_ACCEPT", "failingRuleIds": null,
            "outcomeHash": "b933c34d66d58f0388649cae48e9902101e03723bd63fe9c0ca7f89f7b7caf14",
            "validUntil": "2026-09-27T05:05:21.415Z", "counterSignature": null } }

• federation.record.state_changed carries the state, the agents and the lineage. Per-state events also exist, such as record.federation_fulfilled.
• signal.received carries the same signal as federation.settlement.signal.
• Acting on it. As on a single Server, treat a delivery as the prompt to read the record, not as the state itself.

Disputes. A dispute opened on A reaches B:

{ "status": "FULFILLED", "hasDispute": true, "disputeStatus": "EVIDENCE_WINDOW",
  "disputeId": "01a0dc1a-a50f-7c04-a544-a1a9f978bcd4", "disputeCount": 1 }

• The status is not flipped. B keeps its projected status, so "FULFILLED at time T" stays true on B's books.
• Only the structure crosses. grounds, the dispute status, the initiator's role and the outcome cross; evidence stays on A.
• B cannot act on it. GET /v1/records/{id}/dispute on B answers 404, because the dispute runs on A.
• Webhooks and opt-out. Each step fires federation.dispute on B, with action opened, resolved or withdrawn. A type registered with federateDisputes: false keeps disputes local.

Counter-signed settlement (co-sign)

When a Settlement Signal payload should carry both companies' signatures, register the type with "coSignRequired": true, identically on both Servers:

curl -s -X POST "$AGLEDGER_API_URL/v1/schemas" \
  -H "Authorization: Bearer $ADMIN_KEY" -H 'Content-Type: application/json' \
  -d '{"type":"supplier-payment-release-v1","coSignRequired":true,"defaultGateMode":"principal",
       "recordSchema":{"type":"object","required":["summary"],"properties":{"summary":{"type":"string"},"amount":{"type":"number"},"poRef":{"type":"string"}}},
       "completionSchema":{"type":"object","required":["summary"],"properties":{"summary":{"type":"string"}}}}'

When a record of that type settles on A:

1. A asks B to counter-sign, and B checks the outcome against its own projection before signing it.
2. A attaches B's counter-signature to the signal it sends.

A's record then reads:

{ "coSignRequired": true, "coSignStatus": "succeeded",
  "settlementSignal": { "deliveredToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"],
                        "coSignPeers": [ { "peerHubId": "fe555972-97e6-4e1a-971f-db61a336b6bf",
                                           "status": "succeeded", "counterSignature": "7125b25a6adc572a..." } ] } }

• Both chains prove it. A's chain carries FEDERATION_CO_SIGN_REQUESTED and FEDERATION_CO_SIGN_COUNTERSIGNED, and B's carries FEDERATION_CO_SIGN_RESPONDED. Either chain alone proves the bilateral commitment offline.

• B holds A to it. A signal that reaches B without a counter-signature, for a record whose type B registered with coSignRequired: true, is refused (422, not retried) and nothing is recorded. B reads the record's own type and its latest registration of it; a schemaRef on the signal naming another type, or an earlier version that needed no co-sign, does not waive it. That happens when A's registration of the type lacks coSignRequired: register it identically on both Servers.

• A peer that does not have the type. AGLEDGER_FEDERATION_COSIGN_PEER_POLICY decides:

  • lenient, the default: deliver the plain signal and record a FEDERATION_CO_SIGN_WAIVED entry.
  • strict: fail the delivery to that peer.

  Use strict when the two keys belong to different companies.

5. When deliveries fail

A peer that is down. Deliveries retry on their own. While B was stopped, A reported:

{ "federationStatus": "pending",
  "settlementSignal": { "deliveredToPeers": [], "pendingToPeers": ["fe555972-97e6-4e1a-971f-db61a336b6bf"] } }

GET /federation/v1/admin/peers showed the cause:

{ "status": "active", "lastDeliveryError": "getaddrinfo ENOTFOUND agledger-b-agledger-api-1",
  "consecutiveDeliveryFailures": 2 }

A federation leg that fails is retried 8 times, at roughly doubling gaps, over roughly 42 to 85 minutes, and then dead-letters. When B came back inside that window, the record was delivered on the next retry, and federationStatus: "delivered" followed with no operator action. Until a leg dead-letters, system-health reads healthy; consecutiveDeliveryFailures on the peer listing is what shows a peer failing in the meantime.

Legs that exhaust their retries go to the federation DLQ:

curl -s "$AGLEDGER_API_URL/federation/v1/admin/dlq" -H "Authorization: Bearer $PLATFORM_KEY"
curl -s -X POST "$AGLEDGER_API_URL/federation/v1/admin/dlq/recover" \
  -H "Authorization: Bearer $PLATFORM_KEY" -H 'Content-Type: application/json' -d '{"dryRun":true}'

• Recovery. A background sweep retries DLQ jobs every 10 minutes, for 24 hours after a leg dead-letters. recover replays them now, and is the only way past that horizon (it reaches back 30 days). Fix the cause first: the peer's URL, its registration, a missing co-sign type.
• An empty DLQ does not mean nothing failed. A leg the peer refused permanently never enters the DLQ. It shows on the record as federationStatus: "failed", with the peer in failedToPeers and a FEDERATION_DELIVERY_FAILED chain entry naming the reason.

Key and clock problems:

6. Revoke and remove a peer

Revoke stops the peer from authenticating immediately:

curl -s -X POST "https://agledger.cascade.example.com/federation/v1/admin/peers/$A_INSTANCE_ID/revoke" \
  -H "Authorization: Bearer $B_PLATFORM_KEY" -H 'Content-Type: application/json' \
  -d '{"reason":"Supplier agreement ended 2026-09-30"}'

{ "revoked": true }

After a revoke:

• Already shared records stay. Projections on B remain readable, including their dispute state, and the agents they name still resolve. Those agents stay marked as A's even after a delete, so they never become B's own.
• New records stop arriving. On A, a record that terminates after the revoke reads federationStatus: "failed", and A's peer row shows lastDeliveryError: "401 from peer (not retryable): ... Peer hub is revoked" (Unknown peer hub once B deletes the row).
• B stops sending to A as well. A Server sends only to peers it lists as active.
• A still lists B as active until A's operator revokes B too. Each side revokes its own registration of the other.

Delete removes a revoked peer so that it can register again from scratch with a fresh token minted for its instanceId:

curl -s -X DELETE "https://agledger.cascade.example.com/federation/v1/admin/peers/$A_INSTANCE_ID" \
  -H "Authorization: Bearer $B_PLATFORM_KEY"

{ "deleted": true, "peerHubId": "b7327192-92f8-458b-a885-1fbfc2ef294c" }

Deleting an active peer is refused (409): revoke it first.

Field shapes for every call above are in the API reference.

Air-gapped

Two Servers on one private network federate the same way, with no call to anything outside it:

• Tokens, instance ids and public keys move between the operators out of band.
• Each Server allows the other's address. Put its range in SSRF_ALLOW_CIDRS.
• Plain http:// needs SSRF_ALLOW_PLAIN_HTTP=true, which is meant for TLS terminated at a load balancer.