← Blog

2026-06-10

Engineering

We Bought Our Own Product for a Penny: Four Bugs Our Unit Tests Blessed

By Michael Cooper · Founder

Our AWS Marketplace licensing integration had sixteen passing unit tests and had never once been run against the real AWS API. The function in question is the one a customer's cluster calls at startup to prove it bought AGLedger Enterprise — CheckoutLicenseagainst AWS License Manager. We were tracking the gap as an open launch blocker. If the code was wrong, every paying customer's first boot would fail to validate. It was wrong. Four times, in one ~30-line function, and the test suite was green the whole time.

This is the story of finding that out for one cent, an hour and a quarter of an AI agent's time, and no console clicks. The reference material it leans on — which service owns which operation, the change-set lifecycle, the error-to-cause table — lives in a separate guide we keep current: Listing, buying, and licensing a product on AWS Marketplace programmatically. This post is what happened when we ran it for real.

The problem: you cannot test a license check without a license

A License Manager integration validates against an entitlement, and entitlements only exist as the result of a real Marketplace purchase. There is no sandbox. Our listing was at Limited visibility, pre-launch, with zero subscribers — so the checkout path had zero chances to have ever run. The unit tests mocked the AWS responses, and the mocks were written by reading the same documentation the code was written from. Which means the tests did not check our assumptions; they enshrined them.

AWS's own documentation suggests the way out, almost in passing: a seller can subscribe to its own Limited-visibility product, and the documented test pattern is to temporarily reduce the price, buy it, then restore. So that became the plan: drop every rate card to $0.01 with one UpdatePricingTerms change set, buy our own product, run the real checkout against the real entitlement, and see what breaks.

Buying it: the part that only recently became possible

Until this spring, “buy it” meant clicking through the console. The AWS Marketplace Discovery API (April 2026) and Agreements API (May 2026) changed that: read the offer and its terms with Discovery, then purchase with CreateAgreementRequest → AcceptAgreementRequest. Our installed aws-cli was too old to know either service existed, so the agent used the npm SDK clients instead. The full four-step flow with request shapes is in the guide.

CreateAgreementRequest returns a chargeSummary showing the exact charge before you commit. An agent that is about to spend your money should read the quote out loud. Ours did: "newAgreementValue": "0.01". Then it accepted, the agreement went ACTIVE, and the entitlement reached PROVISIONED about two minutes later. We were now our own customer.

The four bugs

All four lived in the same small function. All four were invisible to the mocked suite. Each one was found the same way: run the real call, read the real error, inspect the real issued license.

Bug 1 — an idempotency token built to lock customers out for a day

The code derived its ClientTokenfrom the product code plus the UTC date — “one checkout per day per instance,” someone had reasoned. But License Manager accepts a given token exactly once, then rejects every reuse for the rest of the UTC day, across processes: ValidationException: Invalid request client token. So a single transient failure at boot (a network blip, IAM still propagating) meant no clean checkout until midnight UTC. And because the token had no per-instance component, HA replicas collided with each other. An idempotency token that is stable for a day is not idempotency; it is a 24-hour lockout. The fix is a fresh randomUUID() per attempt.

Bug 2 — the mock asserted the bug was correct

The code sent CheckoutType: 'PERPETUAL'. Real Marketplace contract licenses of our shape are floating — the issued license carries a ProvisionalConfiguration with a 60-minute TTL, and only PROVISIONAL checkouts are valid against it. PERPETUAL returns NoEntitlementsAllowedException. This is the bug that makes the whole story worth telling, because of what the test suite did with it: a unit test asserted that PROVISIONAL checkouts must be rejected. The mock was written from the same wrong reading of the docs as the code. It did not miss the bug — it certified the bug as the specification. All sixteen tests passed; zero customers could ever have activated a license.

Bug 3 — ProductSKU is the product ID, not the product code

The environment variable, the Helm docs, and our own listing's usage instructions all passed the product code (the eq9fe4… string buyers see). The license AWS issues carries the product ID (prod-…) in its ProductSKU field, and CheckoutLicense matches on it. Wrong identifier, NoEntitlementsAllowedException, forever, for every customer. AWS does document this — in one parenthetical: “the application's Product SKU (Product ID).” The agent found it not by re-reading the docs but by calling GetLicense on the real issued license and diffing its fields against the request we were sending.

Bug 4 — the counted entitlement and the 60-minute hold

The entitlement on the license is counted — Unit: 'Count', max one, value required — and the code sent Unit: 'None' with no value. After fixing that, a subtler one surfaced: a successful checkout consumes the counted unit for the full 60-minute provisional TTL. The agent proved it by running a second checkout while the first token was outstanding — it failed. In production that is a container restart inside the hour, or a second HA replica, mysteriously unlicensed. The fix follows from what the license actually is: not node-bound, by AWS's own definition. So we treat checkout as an entitlement probe, not a seat lock — validate, then CheckInLicense immediately, releasing the unit for the next replica or restart.

A bonus papercut on the way out: the checkout response's Expirationis the consumption-token TTL — checkout time plus 60 minutes — not the contract end date. Our code mapped it to “licensed through,” so every install would have proudly displayed a license expiring today. The contract's real validity lives in the license's Validity field, a year out.

What closed the blocker

With the four fixes in, the agent re-ran the full validation matrix against live AWS: the happy path (boot, checkout, check-in, immediate re-boot); no entitlement before purchase; unreachable License Manager in several variants; an environment-supplied license taking precedence with zero License Manager calls; and a lapsed fallback after breaking credentials post-boot. All five scenarios behaved, and every failure path landed fail-open — which is the licensing philosophy we ship: a licensing-system outage must never take down a customer's audit infrastructure. The unit tests were rewritten to pin the live-verified shapes, so the mocks now encode what AWS does rather than what we once assumed. Real pricing was restored the same way it was dropped: one UpdatePricingTerms change set, re-applying the saved original terms verbatim, because that operation replaces pricing wholesale.

Total marketplace spend for all of the above: one cent.

The same method, applied to the listing itself

A day after the penny purchase we staged v1.0.0 and submitted UpdateVisibility → Public— the step that hands your listing to a manual Seller-Ops review of up to 37 days and locks the product against all other changes while it runs. Before the audit began, we put the buyer-facing usage instructions through the same discipline that found the four bugs: read them as a buyer would, which means execute them.

They failed. The instructions told buyers to run our installer with --image …/agledger:1.0.0, and the installer composes the image reference as image:version — producing the doubled tag …/agledger:1.0.0:1.0.0, which Docker rejects as an invalid reference. A buyer following our instructions verbatim would have hit a hard failure on their first command. AWS's reviewers verify that usage instructions work end-to-end, so they likely would have caught it too — as a failed audit, weeks later. We cancelled the in-flight Public submission inside the window where that is still possible, fixed the instructions and hardened the installer against the doubled-tag input, and resubmitted. As of publication the listing is in Seller-Ops review.

The worst bug in the code and the worst bug in the listing fell to the same move: stop reading what you wrote and run what the customer will run.

What we took away

Mocked tests are a transcript of your assumptions. They are useful for pinning behavior you have already verified, and actively dangerous as a substitute for verifying it — our suite asserted the opposite of AWS's real contract and stayed green. Three of the four bugs contradicted a plausible reading of the documentation; the API's refusals were the only ground truth on offer. The $0.01 self-purchase pattern makes that ground truth almost free, and the new Agreements API makes it scriptable end-to-end. There is not much excuse left for shipping a Marketplace integration that has never met a live entitlement.

One more observation, because error design is our business. Nothing in this exercise was harder to diagnose than the fact that NoEntitlementsAllowedExceptionmeans five different things — wrong identifier, wrong unit, wrong checkout type, exhausted count, not-yet-propagated — behind one opaque string. Five causes, one error, no hint of which. We hold our own API to the opposite standard: error responses carry the recovery path, because the caller, increasingly an agent, has only what the response gives it. The error-to-cause table in the guide is the map we wish that exception had been.

Sources & Further Reading

Related