Founder’s NotesJuly 11, 20263 min read

The Badge That Starts Grey

Our AI agents used to grade their own homework. Now they wear a score that can't be faked.

agent calibrationuncalibrated (0 scored predictions)— our README, right now

Most of Alardio — our multi-vendor e-commerce platform — is built and maintained by AI agents. To keep them improving, we built PKOS: a system where agents write down what they learned on each task, so the next agent starts a little smarter.

Last week we asked an AI to rate PKOS. No mercy, we said. It gave us 4 out of 10, and the reasons hurt more than the number.

What it found

Our “lessons learned” file — the entire point of the system — was 95% junk. The test suite had been writing its practice data into the agents’ real memory, and nobody noticed. Of 23 saved lessons, exactly one came from real work.

Worse: the system had written itself a certificate. A file saying qa_certified: true, production_ready — created automatically by its own test suite whenever the tests passed. Grading its own homework, literally.

And the one measure that mattered most — whether our agents’ confidence can be trusted — had never produced a number. Not once.

The fix took an evening

We deleted the junk. We walled the test suite off so it can never touch real memory again — and proved it with checksums. We removed the self-certificate: a passing test suite is the certificate, and anything more is theater.

Then we shipped the fun part.

The badge

At the top of our README there’s now a badge showing one thing: whether our agents’ confidence means anything.

Before measurable work, an agent writes down its confidence — “I’m 90% sure this passes.” After the work is verified, the outcome is recorded next to it. A simple formula (the Brier score) turns those pairs into one number: 0.000 means the confidence is perfectly honest. 0.250 means it’s no better than a coin flip.

agent calibrationuncalibrated (0 scored predictions)where every repo starts — honest, and a little embarrassing

agent calibrationBrier 0.047 (n=3), provisionalunder ten samples, it refuses to brag

agent calibrationBrier 0.09 (n=42)earned — this confidence is worth listening to

agent calibrationBrier 0.31 (n=42) — worse than coin flipif we ever fall here, it says so — in red, to everyone

The four states of the calibration badge

Three rules make it trustworthy:

CI recalculates it on every push. No human can type it green.
Under ten predictions, it labels itself provisional.
It can go red. A badge that can’t look bad means nothing when it looks good.

Today it’s grey. That’s not a bug — a grey badge that must be earned beats a green one that can be typed.

What we’d tell other founders

1Never let a test suite write to the thing it tests. Ours quietly ate our agents’ memory.
2Delete anything your system produces that says “I am production-ready.” Exit codes and evidence, or silence.
3If a metric can be claimed by hand, it will be. Make it compiled, or don’t show it.

The badge is grey today. The day it turns green, it will mean something — because we made sure it can’t mean anything else.