A Simulated Quarter, Measured and Verified Offline

• 353,861 API calls in 86 minutes - 70 calls/sec sustained - with zero client-visible failures. The mixed claims workload: notarize-only writes, bulk batches, engine gates, principal verdicts, all at once, on two API pods (one t3.medium node each) against a database deliberately capped at 4 ACU - about $0.50 an hour of Aurora - that ran at 100% CPU for most of the run. The plateau is that cap's floor, not the platform's ceiling. Every state-changing call carried an Idempotency-Key.
• Engine verdicts in 32 ms median on a healthy database (p99 124 ms; 16-ACU rerun on the published 1.3.3 image), and still 347 ms median with the capped database saturated under full load, measured from the signed state-transition timestamps on the audit chain itself, not client clocks.
• The full corpus - 264,956 records, 544,991 signed chain entries - verified offline in 56.6 seconds (~4,700 records/s) on a desktop CPU (Intel i7-13700) inside a loopback-only network namespace, against signing keys fetched out-of-band beforehand. Export took 16.7 minutes through the paginated API or 46.5 seconds via the vault-dump script.
• Zero forks across nearly one million records under restart chaos (1.0.0 stamp). 999,006 records notarized at ~108/sec on two pods through 240 forced pod-kills. Every export page hash-chain-valid.
• 2.1 complete gated lifecycles/sec at 24-way concurrency, 3.1/sec at 48-way, zero failures at every concurrency tested. One lifecycle is the full six-call two-party consent flow: create, propose, performer accept, activate, completion, principal verdict.
• Federation delivered 50 of 50 co-signed records in every run on a 3-server mesh, with verdict-to-peer-projection propagation at 1.1–2.4 s median and the bilateral co-sign round-trip at 0.3–0.8 s median.

40,395 insurance claims driven end-to-end through a claims-processing recipe by 25 concurrent writer agents with their own keys, three principal agents, and one admin orchestrator: intake, coverage check, damage assessment (single-call notarize per step), fraud scoring batched through the bulk endpoint in 100-record batches, an engine-decided authority-band gate on every claim, and principal-gated settlement decisions for the roughly 10% that exceeded authority. That is a quarter at ~2,700 records a day for a mid-size deployment, compressed into one sustained session.

The corpus is synthetic in content but real in mechanics: real EKS, ALB, and Aurora, the published customer image, real keys, real signatures. At export time the instance held 264,956 records and 544,991 signed audit-chain entries, with every count cross-checkable between the admin API, the run manifest, and the export file.

The corpus mix models a claims operation where amounts are drawn so ~90% fall within the configured authority ceiling. In that mix, of 243,492 claims-flow records:

• 81.9% were notarize-only: recorded in one signed call, no gate at all.
• Of the 40,472 engine-gated completions, 90.0% were auto-cleared by the rules engine with no human involvement; 10.0% failed the authority check and escalated.
• 1.66% of all records waited on a human. 3,700 accepted, 146 rejected, 158 deliberately left unanswered past their deadline so the expiry sweep would time them out on the record, and 41 abandoned partway (4,045 in all).

Those shares are a property of this simulated mix and its gate policy, not a product constant; your split depends on where you set the authority bands. What is a product property is the overhead. On a healthy database (16-ACU cap, published 1.3.3 image, identical protocol), engine verdicts landed in 32 ms median (p95 48 ms, p99 124 ms), the principal-verdict floor sat at 90 ms median (p95 136 ms), and the verdict on a held completion was recorded 57 ms median after the hold (a scripted principal answering immediately - the mechanical floor). During the quarter run itself, where the database was deliberately capped and saturated, the engine still rendered verdicts in 347 ms median under the full 70 calls/sec load (p99 1.25 s) and the principal floor was 669 ms. In production the principal span is dominated by how long the human takes to decide, which is not an API property; the platform adds tens of milliseconds of median overhead around the human, and still under a second even with the database pinned. Latency spans were computed from createdAt on the signed chain entries, so anyone with the export can recompute them.

AGLedger has two flows with different cost profiles, and the numbers below fence to their conditions:

Performance numbers for an evidence system only matter if the evidence checks out, so the quarter ended the way an audit would. The tenant chain export pulled all 544,991 entries (1.21 GB) through the cursor-paginated API in 16.7 minutes; the shipped vault-dump script produced the same corpus in 46.5 seconds. Verification then ran on a desktop machine (Intel i7-13700, 47 GB RAM) inside a loopback-only network namespace - the only interface present was lo, down - against signing keys fetched out-of-band before going offline.

The published verifier library walked every record chain: payload hash, hash-chain linkage, Ed25519 over each COSE_Sign1 envelope. 264,956 records and 544,991 entries verified in 56.6 seconds with zero failures. A 50-record random sample exported through the per-record audit-export endpoint and checked with the published CLI (agledger-verify --require-out-of-band-keys) passed 50 for 50, every entry attributed to the out-of-band keys rather than anything embedded in the file. That is the property the whole product rests on: a regulator or counterparty can take the export and check it without trusting us, our server, or their network.

The chaos result from 1.0.0 still stands as the integrity experiment: ~1,000,000 notarizations attempted while a chaos loop force-deleted an API or worker pod every ~45 seconds. 993,663 committed; the only failures were 6,337 transient 503s (0.63%), every one during a forced restart, with zero corruption-class errors. (The vault's 999,006 notarized records at sweep time add the setup and side-run records that sat outside the two load drivers.) The vault sweep found zero duplicate (recordId, chainPosition) pairs, zero payloadHash collisions, zero position anomalies; 9,992 of 9,992 export pages hash-chain-valid. One precision note: AGLedger's chain is per-record lineage (each notarization is a COSE_Sign1-signed root; delegation children extend a lineage), not one global linear chain - “zero forks” means those vault-wide invariants held. Distributed-systems work like Jepsen has shown how often invariants like “no record loss under back-pressure” fail once actually measured; we hold this one open as a soak invariant the testbed re-checks rather than a once-and-done claim.

Cross-organization work was measured the way a customer runs it: co-sign-required gated records driven through the full consent lifecycle on server A, results read from server B's projection and A's chain, on three bilaterally-peered servers. Steady state: verdict on A to FULFILLED projection on B in 1.1–2.4 s median (p95 2.0–3.2 s); the bilateral co-sign round-trip in 0.3–0.8 s median. Delivery was 50 of 50 in every run, including one run that started while the outbound queue was still draining a ~3,500-record backlog from the bulk probes - p95 stretched to 36 s there, but delivery stayed complete. Under backlog the mesh degrades in latency, not in completeness. Every delivered settlement signal carried the counterparty's Ed25519 counter-signature.

These numbers are a floor, not a ceiling. The envelope was chosen for cost - API pods capped at 1 vCPU / 512 MiB on t3.medium nodes, a database capped at 4 ACU, roughly $0.50 an hour of Aurora - and post-session analysis showed the database cap was the binding constraint: CloudWatch has it at 100% CPU for about 70 of the sustained phase's 86 minutes. The API pods averaged a quarter of their CPU cap and peaked at 193 MiB memory. Zero client-visible failures and every integrity result above held with the database pinned, and nothing degraded as the corpus grew from zero to 265k records.

Give the database headroom and the same protocol moves: with the cap raised to 16 ACU (peak observed 11.5), a single pod went from ~36 to 164–194 single-record rec/s and from ~115 to 217–235 bulk rec/s, with 1.2.0 and 1.3.3 interleaved at parity. A methodology lesson this batch paid for: absolute numbers measured on a capped, saturated database are floor artifacts - verify database headroom before trusting any of them, ours included. HA and clustering remain a vertical-scale motion (bigger DB, more API replicas behind the same Server), not a horizontal-shard motion, and Aurora Serverless v2 scales to 128 ACU before read replicas enter the picture.

v1.3.4 is the first multi-arch release (linux/amd64 and linux/arm64, one index digest), so we benched the architectures against each other: the same released artifact on m7g.large (Graviton3) and m7i.large (Xeon Platinum 8488C), 2 vCPU / 8 GiB each, one pod per node, shared Aurora at 16 ACU, nothing else on the cluster, warmup discarded, four rounds interleaved with alternating lead.

arm64 runs the released image natively, is 13–18% slower per core, costs 19% less, and lands at parity to modestly better per dollar. Single-record median 289.5 rec/s on arm64 against 334.0 on amd64 (-13.3%); bulk 87.8 against 107.6 (-18.4%). Variance was low and the arms never overlapped across rounds. On us-west-2 on-demand pricing read from the AWS pricing API ($0.0816 vs $0.1008/hr, 19.0% cheaper), that works out to 3,548 vs 3,313 single records per dollar-hour (+7.1% for arm64) and 1,076 vs 1,067 bulk (+0.8%).

Native means native: pods on the Graviton node resolve the same index digest per-node, process.arch reports arm64 on an aarch64 kernel with no emulation, migrations apply, the SDK and CLI floors pass (104/104, 24/24), and Ed25519 signatures produced on arm64 verify against the published offline verifier running on x86.

A methodology trap this run nearly fell into: never bench an architecture on burstable instances (t2/t3/t4g). Our first attempt used t4g.medium against t3.medium and reported arm64 +38.6% on bulk - wrong sign, wrong magnitude. Both instances were sitting at a CPUCreditBalance of about 0.5 with utilization above the 20% baseline: throttled, and not symmetrically. The tell was that single-record sat at parity while only bulk diverged; a real per-core difference moves both. The m7 numbers above are from fixed-performance instances.

A benchmark that finds nothing wrong wasn't looking. This batch filed five findings upstream. Four have since shipped (the export-cap flag, the ALB drain, and vault-dump streaming in v1.3.4; bulk idempotency in v1.3.2); the CLI full-dump verify cap remains open. As found:

• A plain rolling restart behind the ALB is a ~15-second hard outage: the chart ships without ALB pod-readiness gates or a preStop drain, and kubectl rollout status reports success mid-outage.
• The compliance-export packet silently capped at the newest 10,000 records with no truncation flag and no total; v1.3.4 made a truncated export say so, with the full match count. Packet numbers from that surface wait for a re-run against the fixed release.
• The documented vault-dump command (exec into the API pod) OOM-kills the serving pod at this vault size; the 46.5-second dump above ran in a dedicated pod instead.
• The published CLI's full-dump verify mode reads the dump as one string and dies past Node's ~512 MB cap. The verifier library streams fine - the 56.6-second full verification used it - but the CLI convenience path does not yet.
• Bulk create ignored Idempotency-Key (single-record create dedups correctly), and 100-record shared batches could time out at the gateway while still committing server-side. Fixed in v1.3.2 and re-hardened in v1.5.0; the bulk-plus-federation numbers wait for a re-run.

One lesson where the platform was right: an early leg run without idempotency keys lost ~330 steps to a rate-limit and rollout window - the duplicates were correctly refused, with a recovery hint naming the allowed actions. The restarted leg put a key on every state-changing call and ran 353,861 calls without losing one. The documented replay-safe pattern held at scale.

Quarter session (2026-07-21): published image 1.2.0, digest-pinned, running image verified; EKS 1.36 us-west-2, Aurora Serverless v2 PostgreSQL 18.3, shared ALB with TLS; API pods at the chart's testbed envelope (1 vCPU / 512 MiB cap), one pod for the quiescent latency condition, two pods (one per t3.medium node) for the sustained phase. Load: 25 concurrent writer agents with their own keys, 3 principal agents, every state-changing call carrying an Idempotency-Key. Verification hardware: Intel i7-13700 (20 threads), 47 GB RAM, Linux, inside a loopback-only network namespace with keys fetched out-of-band beforehand. Latency spans derive from signed chain-entry timestamps; every published number is recomputable from the export file plus the run manifests. Post-session analysis found the 4-ACU database cap saturated during these runs (100% database CPU for ~70 of the 86 sustained minutes, 62–94% during the quiescent pass), so the healthy-database latency and throughput figures above come from a same-day rerun of the identical protocol (400 claims, 4 writers) at a 16-ACU cap on the published 1.3.3 image (digest-pinned, validated the same day); a same-day interleaved A/B measured 1.2.0 and 1.3.3 at throughput parity, so those figures are not version-bound.

Mesh session (2026-07-12): the published 1.2.0 image on a 3-server P2P federation mesh, one server per t3.medium node, per-server databases on a shared Aurora cluster, all servers bilaterally peered before measurement; rate-limit exemption applied to the two bench agents so concurrency reached the append path. Out of the box, the default customer rate limit is 200 requests per 60 seconds.

Chaos run (1.0.0, 2026-06-10): EKS 1.36, Aurora PostgreSQL 17.9 at 4 ACU, both API pods and the worker on a single t3.medium, two drivers at 500k notarizations each, a chaos loop force-deleting a pod every ~45 seconds, 240 kills total.

Still unpublished: a controlled same-corpus scale-out A/B, bulk-plus-federation throughput, and compliance-export packet timings (the blocking findings shipped; the re-runs against the fixed releases have not happened yet). Principal decision latency in production is human think-time and is not an API property; we publish only the mechanical floor around it.

A two-pod deployment on compute that costs cents per hour sustained a quarter's operation in an afternoon without a single client-visible failure, on a database capped at $0.50 an hour and pinned at 100% CPU. The load profile of an LLM agent fleet is small relative to modern databases; nothing in this run suggests AGLedger becomes the bottleneck that forces a second instance or a more expensive database tier.

The number to keep is the last one: in this mix, 98.3% of records never waited on a human, the engine cleared the gated ones in 32 ms median (a third of a second even with the database saturated under full load), and the entire signed history of the quarter fit through an export and a one-minute offline verification on a desktop CPU. Agents doing real work at speed, with evidence anyone can check without trusting the operator - that is the point of the exercise.