Testing to Learn: Deterministic CI, Probabilistic Agents, and the MCP Proxy We Killed

Testing has two jobs. The first is the one every team builds: catch regressions before customers do. The second is the one that shaped this product: falsify ideas while they are still cheap. AGLedger’s stack looks the way it does because a test told us our first architecture could not work, one month after we built it.

The deterministic layer

Every build runs more than 2,900 individual test cases across 195 files in the API repository: state-machine transitions, RBAC matrices, schema validation, cryptographic primitives, and integration tests against a real PostgreSQL instance, not an in-memory stand-in. This is the wide base of Fowler’s practical test pyramid, and it is aligned with NIST’s minimum standards for developer verification.

Above it sits an independent testbed: 39 deployed-infrastructure suites that run against a live deployment. Real EKS cluster, real Aurora database, real ALB and WAF, the same Docker image customers pull, fresh namespace per run. No mocks, no stubs. The testbed never re-runs what CI already proves; it exists for behavior that only manifests on deployed infrastructure or a customer-facing surface. A separate stress harness records pg_stat_statements snapshots per release, which is how we caught and fixed a query pattern burning 70 seconds of database CPU every 5 minutes.

Every assertion in the testbed produces one of four outcomes: PASS, FAIL, SKIP, or KNOWN_ISSUE. The last one is the interesting one. Without it, an author who discovers a real bug faces a bad choice: mark it PASS with a comment (hides it) or FAIL (makes the suite cry wolf about things already being fixed). KNOWN_ISSUE references a numbered finding (F-NNN), stays visible in every run, and starts passing the moment the underlying bug is fixed, so we notice. Three rules hold for every author:

This layer is allowed to be boring. Its job is to keep the next layer honest.

The probabilistic layer

Deterministic tests verify the system we built. They cannot tell you whether an agent that has never seen your product can actually use it, or whether the architecture you committed to matches how agents behave in the wild. For that we run real LLM agents: Claude Haiku, Gemini Flash, GPT-4o-mini, and Amazon Nova, deliberately mixed along two axes.

The first axis is the provider, because every model carries different priors and fails differently. The second axis is product awareness: at the bottom tier an agent gets tool definitions only, no documentation, no examples; higher tiers get progressively more context, up to scripted SDK baselines that establish what a fully-informed caller achieves. Comparing tiers is the point. The gap between “tool definitions only” and “fully informed” is a direct measurement of how much teaching the product still owes its users.

The agents run randomized business scenarios: procurement, analysis, coordination, infrastructure. Hundreds of distinct simulated customer scenarios, each executed dozens of times across releases. The actors are nondeterministic; the scoring is not. We measure discovery rate, error recovery, steps to completion, and where each provider gets stuck, under a hard cap on tool calls. Research on code generation found 84–89% correctness on synthetic benchmarks vs. 25–34% on real-world tasks (arXiv 2510.26130); we see the same synthetic-vs-real gap in API discovery, which is why we only trust the numbers from the bottom tier.

The proxy that testing killed

In January we built AGLedger’s first architecture around an MCP proxy. The idea was elegant: sit in the agent’s tool path, observe every tool call as it happens, and notarize automatically as a side effect. No integration work for the agent at all.

Agent simulations falsified it within weeks, on two fronts. First, coverage: a proxy in the MCP path sees MCP traffic and nothing else. The moment an agent calls an external service over plain HTTP, the proxy is blind, and capable agents reach for direct API calls constantly. You can lock an agent down and force everything through MCP, but that brought the second finding: our testing measured a massive token cost for MCP-mediated calls compared to the same work over pure HTTP - an overhead Anthropic’s own engineering team has since written about. And in the same simulations, unforced agents were already voting with their tool calls: direct API and CLI usage dominated MCP. Two independent signals, one verdict.

In February we pivoted the entire stack. The API became primary: an agent reports what it intends and what it did, authenticated by its own key, and our engine signs what was reported. The MCP server and CLI were rebuilt as thin wrappers over that API. We still ship @agledger/mcp-server; it is an interface now, not an interception point.

We lost a month instead of a year, and got a better product: one that works however the agent connects, instead of only when the agent is forced through our path.

The pivot left us with a doctrine. MCP used to be the tool you used to teach agents to use an API. Now the API has to teach agents directly: error messages written as directives, response fields that say what to do next, hints in the payload itself. That doctrine became its own discipline, and its own post.

Your test agents are not blind

The most expensive lesson from hundreds of hours and millions of tokens of agent testing is about the testing itself. Claude subagents are not blind. No frontier model is. They arrive with strong priors: REST conventions, common endpoint shapes, pattern-matches against every API in their training data. An agent that “discovers” your API in three calls may be demonstrating your design, or just its own memory.

The subtler leak is the harness. The scaffolding you build to manage test agents - system prompts, tool descriptions, orchestration context - quietly coaches them toward the happy path. It is very easy to believe you are testing discovery while handing the agent the correct answers before it starts.

We do not claim to have eliminated this; we control for it. Multiple providers with different priors. Randomized scenarios instead of fixed scripts. Hard tool-call caps. And the awareness tiers above: rather than assuming any single run is blind, we measure the delta between tiers, because a contaminated baseline shows up as an implausibly small gap.

What the scale looks like

Catch bugs, test ideas

If you are evaluating AGLedger, this is the part worth internalizing: the deterministic layer exists so customers never see a regression, and the probabilistic layer exists so we never spend a year on an architecture a month of testing could falsify. The proxy pivot was the largest correction the loop has produced, but it runs continuously; agent failures have rewritten our error messages, our tool descriptions, and our response payloads.

We sell signed evidence of what automated work actually did. Testing is where we apply the same standard to ourselves: not what we hoped the architecture would do, but what it measurably did.