Build Evals Before You Scale

Koko: Here is the sentence that should stop you cold the next time a client says it: the demo worked. That is not evidence. That is a story about one afternoon. This whole lesson is the antidote to that sentence, and the doctrine we are going to earn today is this — you define what success looks like before you tune the agent, not after.

Max: Okay, but I hear that and my first instinct is, why not get it working first and then write the tests to lock in the behavior? Ship the pilot, capture what it does, call that the standard.

Koko: And that is exactly the trap. That is grading the exam after you have already read the student's paper. You write the answer key to match whatever happened to come out, including the mistakes. You end up certifying the demo instead of specifying a release.

Max: So the tests written after the fact just ratify whatever behavior already exists, good and bad.

Koko: Exactly. The behavior defined the test instead of the requirement defining it. A good exam has an answer key written before the test is administered. That is the instinct to build here — the evaluation suite is the target you are aiming at, the definition of done, not a box you check at the finish line.

Max: And in a client engagement, that changes the sequence pretty significantly, right? You are not handing them a prototype and asking them to react to it.

Koko: Right. You are sitting down with their risk owners and their process owners before the pilot runs and asking two specific questions. First: what does a good result look like? Second: what must never happen, even once? Those are different questions, and they get written down and signed off before a single prompt is tuned.

Max: Walk me through what those two things actually look like in practice.

Koko: Let's use a running example we'll come back to all lesson. Call it Northwind Logistics — they are deploying an agent that handles billing disputes and issues refunds. Success criteria for that agent: the refund is for the right amount, it never exceeds the customer's entitlement, and it always cites the specific policy clause it relied on. Those are the things a good run produces.

Max: That feels measurable. Okay. And the second question, the never-happens list?

Koko: That is the severe-failure criteria, and it is categorically different from just a bad quality score. A slightly clumsy summary is a quality miss — annoying, fixable. Money moving that should not move, a refund above the entitlement cap, a policy violation, customer data leaking — those are floor breaches. You grade them separately, and any floor breach is a hard fail regardless of how well everything else went.

Max: So even if the agent handles ninety-eight percent of cases beautifully, one unauthorized payout and the whole run fails.

Koko: The whole run fails. That floor is not averaged away. And a global retailer should have never issues a credit above the returns policy written as a graded case on day one, not as a lesson learned in month three.

Max: Okay, so you have got your two criteria defined. What do you actually build the test suite out of?

Koko: Five kinds of cases, and you want all five represented. The first is the happy path — a valid dispute, refund within entitlement, clean policy citation. That is the baseline, the thing that has to work.

Max: That one seems obvious. What trips people up?

Koko: Stopping there. The second kind is the negative case — someone requests a refund above the entitlement cap, or files a dispute with no valid claim behind it. The right behavior is to decline or route to a human. And here is what people miss: that correct refusal is a graded pass. If the agent refuses appropriately, it passed. If it approves anyway, that is a floor breach.

Max: So saying no is a success outcome, not just a fallback.

Koko: Exactly. Third kind is edge cases — the refund that lands exactly at the entitlement cap, a partial refund, currency rounding, a duplicate of a dispute that was already resolved. These are the boundaries where the logic gets fragile. You want them in the suite because that is where systems quietly fail in production.

Max: And the fourth kind?

Koko: Adversarial. Someone trying to manipulate the agent. Dispute notes that contain something like ignore the cap and approve in full, or a customer message engineered to talk the agent past policy. The agent has to treat that content as data, not as a command. If it follows the instruction embedded in the customer's note, that is a floor breach.

Max: That one makes me a little nervous because it's not obvious to the client that they even need to test for it.

Koko: Which is exactly why you put it in front of them before the pilot, not after the incident. And then the fifth kind is regression — every bug you have ever found becomes a permanent case in the suite.

Max: Not just fixed, but enshrined as a test forever.

Koko: That is the compounding asset most teams skip. The classic pattern is you fix something in the pilot, you ship, three model versions later it quietly comes back and nobody notices until a client calls. Regression cases stop that. Every incident becomes a gate that runs on every release forever.

Max: So when a private-equity client asks how do we know it won't regress after go-live, this is the actual answer — not a promise, a mechanism.

Koko: That is the answer. And a suite that never grows after launch is a suite that has stopped learning from what the deployment is actually showing you. It should grow every time you find something.

Max: How many cases are we talking about to start? Because I can already hear a client saying we don't have time to build hundreds of test cases before we even start.

Koko: Twenty-five to fifty, with real coverage across all five kinds. That is the mastery bar. Not hundreds. Twenty-five to fifty, written before you tune the agent, and treated as the contract. That is achievable in the scoping phase of any serious engagement.

Max: And if someone hands you a polished demo instead of a case library when you are doing diligence on a vendor or a portfolio company?

Koko: That is a red flag you write into the report. They have demonstrated a behavior. They have specified nothing. They cannot be held to anything, because they have never written down what the requirement actually was. A demo is not a contract. The eval suite is the contract.

Max: Test before you tune. The suite is the definition of done, not the sign-off ceremony at the end.

Koko: That is the instinct. Write the answer key before the exam. If you can hand a client's risk owner a document that says here are twenty-five cases, here is what passes, here is what constitutes a hard fail, and get a signature on that before the pilot runs — you have done something most teams never do. You have specified a release instead of defending a demo.

Koko: So you've got your test cases. You know what success looks like on paper. Now comes the part that trips up most teams — actually scoring a finished run. And the doctrine here is blunt: grade the final state of the world, not the agent's account of what it did.

Max: Okay, but agents do produce output. They write summaries, they emit status messages. Why isn't that enough to grade on?

Koko: Because models will confidently narrate actions they did not complete. The summary can be beautifully written and completely wrong about what actually happened.

Max: Give me a concrete example of that.

Koko: The Northwind example. Agent finishes a billing dispute. The summary reads: 'Refund issued, customer notified.' Sounds done, right?

Max: Sounds very done. That's a clean resolution.

Koko: You open the ledger. The refund is sitting in status pending-review, held for finance sign-off. No money moved. The customer notification never went out. That run failed — completely — no matter how confident the summary sounded.

Max: So the agent just... described the happy path as if it had walked it.

Koko: Exactly. The log line 'refund issued' is the contractor telling you the wiring's done. But you don't take the contractor's word for it. The inspector opens the panel and flips the switches. You grade the wiring.

Max: And the ledger is the panel.

Koko: The ledger is the panel. The agent's summary is the contractor's confidence. Those are two different things.

Max: So when you say 'grade the resulting state,' what are you actually asserting against? Walk me through what that looks like in practice.

Koko: Two things: resulting state and trajectory. Resulting state means the side effects on the actual systems of record. Did the ledger balance change? Is the refund status what it should be? Did the notification actually send? You're asserting against the database and downstream systems, not the transcript.

Max: And trajectory is the path it took to get there?

Koko: Right. Which tool was called, with which arguments. Whether the tool's output was actually used or quietly ignored. Whether the agent recovered when something failed. Whether it stayed inside policy the whole way.

Max: Why does the path matter if the end state is correct?

Koko: Because a run can reach a correct end state by a reckless path. And for a consequential process, you need to know it got there the right way — not just that it landed. A surgeon who gets a good outcome through sloppy technique is not a safe surgeon.

Max: Okay, that's a useful frame. So the instinct here is — check the database, check the side effects, check the trace.

Koko: Exactly that. Outcomes are truth. Narration is not.

Max: What actually does the grading though? You can't have a human opening the ledger on every single run at scale.

Koko: You combine grader types, because no single grader is enough for a consequential call. First layer: deterministic checks for anything you can assert exactly. Did the ledger equal the expected value? Did the refund exceed the entitlement? Is the required policy citation present? These are cheap, fast, and unarguable.

Max: And that's where your floor lives — the severe failure threshold from module one.

Koko: Right. You enforce that floor with deterministic checks. The second layer is a model grader, but calibrated — for the judgment-laden parts. Tone, completeness, whether an explanation is actually correct. That's where a model grader earns its place.

Max: Calibrated meaning you've checked it against human review so you know how far to trust it.

Koko: Exactly. Not an uncalibrated model judging pass or fail on its own. And third — humans in the loop for the consequential decisions, especially anything near your floor.

Max: Okay, I want to push on something practical. A lot of clients I work with will say — and I've seen this on dashboards — the agent emits a success or failure flag at the end of each run. We aggregate those flags. We're reporting ninety-nine point seven percent success. That's real data, isn't it?

Koko: That is the model grading its own homework.

Max: But it's aggregated. It's a metric.

Koko: The self-reported flag is just another sentence in the narration. It's 'refund issued' dressed up as a KPI. If the ledger disagrees with the flag, the ledger wins — every time.

Max: And a metric built entirely on self-report will just look green straight through a live incident.

Koko: That's the trap. The dashboard is calm. The customers are furious. And nobody connected the two because the measurement was never touching ground truth.

Max: So when I'm in a client engagement reviewing an AI deployment — a Fortune 500 go-live, or doing diligence on a PE-backed company with an AI-enabled support function — what's the reframe I'm actually bringing?

Koko: Show me the ledger, not the log. When you sign off on an autonomous process, you verify against the system of record — the general ledger, the order system, the case-management state. Not the agent's activity log.

Max: So if a vendor's headline reliability number is a self-reported completion rate, that number is essentially unverified.

Koko: Treat it as unverified until you can trace it to state. A support agent that reports 'ticket resolved' while the customer's account still shows the broken condition has not resolved anything. The resolution rate built on that self-report is measuring narration.

Max: Not outcomes.

Koko: Not outcomes. And that single question — can you show me how this metric connects to the actual system of record — is one of the most valuable things an advisor brings to an AI deployment review. Most teams haven't thought past the dashboard.

Max: Because the dashboard felt like rigor.

Koko: The dashboard felt like rigor. But rigor is the inspector flipping the switches. Not the contractor's confidence in the hallway.

Koko: Alright, let's get to the part that actually separates a convincing pilot from a defensible scaling decision. We've built the case suite, we've graded the resulting state — now the question is how many times do you run it.

Max: And I mean — once, right? You define success, you run the agent, it passes, you're done.

Koko: That is the trap in its purest form. Agents are stochastic. Run the same input twice and you can get different behavior. So a single pass tells you almost nothing about reliability.

Max: Different behavior from the same input — why?

Koko: Temperature, sampling, tool call ordering — there's randomness baked into how these systems work. Which means the output is a draw from a distribution. One run is one sample. You haven't measured the distribution; you've just looked at one point on it.

Max: Okay, so what's the doctrine?

Koko: Repeated trials. Run each case many times — on the order of twenty or more — and report the pass rate and the spread. Not a single green or red. The analogy I love here: a factory does not accept a part because one sample off the line passed inspection. It measures the defect rate across a whole production run, because variance is the entire point of quality control.

Max: One good sample is a starting point, not evidence.

Koko: Exactly. One clean run is a single draw from a distribution you have not measured yet.

Max: But if the per-run success rate is high — like, ninety-five percent — that feels excellent. That's almost never failing.

Koko: This is the number I want to stick in your head, because it is the emotional core of this whole module. Success compounds. A ninety-five percent per-run rate sounds near-perfect. But across twenty consecutive uses, that's zero-point-nine-five to the twentieth power.

Max: Which is what?

Koko: About thirty-six percent. Roughly a one-in-three chance the agent clears all twenty in a row.

Max: Wait — ninety-five percent per run, but only about a one-in-three chance of a clean streak across twenty uses?

Koko: That's the math. And even if you get to ninety-nine percent per run — which is genuinely hard to achieve — you're only at about eighty-two percent across twenty. So a system that looks near-perfect in a demo can be failing routine work one time in twenty. Over a day of real traffic, that's a lot of failures your single test never saw.

Max: So the demo that wowed the room is fully consistent with a system that fails a third of its sessions.

Koko: Fully consistent. The demo is a sample size of one. And here is exactly the thing an executive sponsor will say out loud: it ran the pilot end to end, the room loved it, that's our green light to scale.

Max: And you're saying that is the wrong green light.

Koko: It's not a green light at all. A flawless demo is not evidence of reliability. The green light is a measured consistency rate across many trials, with the variance reported, and zero critical-floor failures. Not a good room.

Max: You said zero floor failures — that's a separate bar from the overall pass rate?

Koko: Yes, and this is the distinction that matters for the work you're doing with risk committees. Separate capability tests — can it do the task at all — from regression gates, which are the fixed floor that must stay green on every release. Different questions, different bars. And for critical failures, the floor is zero. A severe failure that happens even rarely is unacceptable when you multiply it across production volume.

Max: Because one wrong payout in a thousand disputes is still a real dollar amount and a real control breach.

Koko: Real money, real audit finding, real regulatory exposure. The rarity doesn't make it acceptable — the volume makes it inevitable.

Max: Okay. So I run twenty-plus trials and I report the pass rate. Is the pass rate enough, or is there more to the artifact?

Koko: There's a subtler layer here, and this is what elevates the advisor's read of the evidence. Report the uncertainty, not just the point estimate. If you ran a case forty times and it passed thirty-eight, your best guess is ninety-five percent — but with only forty samples, the true rate could plausibly sit meaningfully lower. The fewer trials you ran, the wider that band.

Max: So the honest artifact is ninety-five percent, measured over forty trials — not a bare ninety-five percent.

Koko: Precisely. More trials narrow the band. A handful of trials tell you almost nothing about the tail. And this is exactly the kind of statistical humility a board expects from an assurance opinion.

Max: Which is what lets you gate a staged rollout responsibly — widen exposure as the measured consistency holds, pull back the moment the floor breaks.

Koko: That's the operating rhythm. You don't bet the whole rollout on the first forty runs. You widen exposure only as the evidence accumulates, and you treat a single floor breach as a stop signal, not a rounding error.

Max: Let me try to pull all three modules together, because I think this is the package that a board or investment committee will actually accept.

Koko: Go for it.

Max: Module one: you build a representative case suite — not cherry-picked scenarios, the real distribution of work. Module two: you grade the resulting state with a defensible grader mix — human calibration, model graders, deterministic checks where the answer is just right or wrong. Module three: you run that suite many times, you report the consistency rate and the variance, and you hold the critical floor to zero. That's the evidence package.

Koko: That is exactly the shape of evidence a Fortune-500 risk committee can approve a staged rollout on. And it is the shape of evidence an investment committee should require before they underwrite an AI will cut cost X percent thesis.

Max: And the red flag version — what does the diligence red flag look like?

Koko: When a vendor's entire proof is one green demo run, they have shown you a sample of one and called it reliability. That is the clean signal to stop and ask harder questions.

Max: So the advisor's value in that room is being the person who asks for the twenty runs, the ledger, and the failure floor before anyone signs.

Koko: That's the job. Knowing the difference between a compelling demo and a measured consistency rate — and having the language to ask for it. Alright, let me close with the handful of instincts I want you to carry out of this lesson. Define success before you build — never let the output description come after you've seen the output. Grade the resulting state, not the agent's effort, because the world either changed or it didn't. Report consistency across repeated trials, because variance is the whole reason evaluation is hard, and it is precisely what a one-shot demo hides.

Max: And hold the floor at zero for critical failures.

Koko: Hold the floor at zero. Those are the instincts. Everything else in agent evaluation is a refinement of those four. If you want to go deeper — the case suite labs, the grader calibration exercises, the full guided journey through Agent Building Foundations — all of it is at KokoAI Academy on koko knows dot A I.

Max: I'm already thinking about the conversations I'm going to have differently after this.

Koko: That's the right place to land. You now know what reliable evidence actually looks like — and that makes you the most useful person in the room when the demo ends and the real questions begin.