Choose the Right Thing

Sam: Here is what I actually want from this lesson. I want something I can use on Monday, in a room where nobody has agreed what AI even means yet, and where someone is about to say we should just pick a model and start.

Koko: That is exactly the right want. And here is the uncomfortable thing sitting underneath it: the model you pick today will probably change twice before what you are building reaches production. Model choice is not the stable thing. So if you build your method around it, you are building on sand.

Sam: Which means the stable thing has to be something else.

Koko: The questions. The vocabulary. The artifacts. Those are what travel from project to project and from model generation to model generation. The one instinct I want you to carry out of this module: the questions are the deliverable. The letters — and we do use a five-letter structure — are only how you remember them.

Sam: Say more about the five phases, because I want to understand what each one is actually asking.

Koko: Align is first. What outcome matters, what is the baseline today, and why is this the smallest sufficient solution? Ground is second. How should the work itself change, who decides what, and what evidence does the system actually need to do its job? Equip is third — that is where you define what the system may know, decide, and call. Narrow is fourth, and we are going to come back to that one because it is the one that most methods miss. And fifth is Trace and Tune: is it reliable, observable, affordable, and supportable over time?

Sam: Align through to Trace and Tune. So the letters are A, G, E, N, T.

Koko: They are. And before you ask — yes, that is deliberate, and yes, it does make it easier to remember. But the letters are decoration. The questions behind them are the work.

Sam: Let me test that against something concrete. The ask I keep getting right now is some version of: put AI on the deductions queue. What does Align actually do to that ask?

Koko: It turns a technology instruction into an outcome with a number. So you ask: how many claims? Twelve hundred a month. How much touch time? Eighty-nine minutes each, on average. What is the financial exposure sitting unexamined? Four point one million written off last year. And who is the named owner whose targets actually improve if this works? You have answered none of those by saying put AI on it. But once you have answered all of them, the conversation has already changed — and you have not written a single line of code.

Sam: That is the part that I think surprises people. The early phases are doing real work even though there is nothing built yet.

Koko: Think of it like a builder. A builder does not start with the material. They start with what the room is for, who walks through it, and which walls hold the roof up. You do not pick the timber before you know whether the wall is load-bearing. Same principle.

Sam: And the consequence of skipping to Equip — to the what-may-it-know-and-call phase — is that you end up rebuilding.

Koko: Consistently. Two thirds of the work in this method lives in Align, Ground, and Narrow — the three phases with no code in them. The teams that treat those as overhead are the teams that are back in the room six months later with a system that technically works and operationally does not.

Sam: The version of this I keep hearing on steering committees is: this is just discover, design, build, test, run — you have renamed the boxes. And I have never had a completely clean answer to that.

Koko: The clean answer is Narrow. Standard delivery lifecycles fold controls into design and into testing. They treat controls as a design constraint and a test criterion. That works for deterministic systems. For a probabilistic system — one that will produce a different output given the same input under different conditions — the controls cannot live only inside the model. They have to be enforced outside it, and they need a separate owner, separate artifacts, and a separate phase. That is what Narrow is. It asks: what must be prevented, approved, detected, reversed, or escalated — and by whom, by what mechanism, independently of whatever the model does. There is no equivalent phase in a standard software lifecycle because standard software does not need one. That structural difference is why the fourth letter exists at all.

Sam: So the misconception is not that the phases sound familiar. It is that familiar-sounding phases make you think you can handle everything inside existing gates.

Koko: Exactly. And the cost of that misconception is a system that is live, that has passed all your standard gates, and that has no enforced mechanism for what happens when the model is confidently wrong. Which, with a probabilistic system, it will be.

Sam: Right. The model does not know it is wrong. That is the whole problem.

Koko: So: the method is the stable thing. The questions are the deliverable. And Narrow is the phase that makes this structurally different from any lifecycle you have used before. That is the foundation everything else in this lesson builds on.

Koko: So the ladder. Seven rungs, and the rule is simple: you test a candidate from the bottom up and you stop at the first rung that reliably delivers the outcome. Rung zero is rules and automation — if a rule does it, you ship a rule. Rung one is predictive ML, a number or a class from your own history. Rung two is a single model call. Rung three is a retrieval-grounded assistant. Rung four is a coded workflow — chain, router, parallel, evaluator-optimizer, all of it written by a human. Rung five is a bounded single agent. Rung six is multi-agent across an organizational boundary.

Sam: The framing I use with teams is: every rung you go up is a cost you are choosing to pay, not a prize you are earning.

Koko: That is exactly it. And that framing sets up the most important structural point in this whole course, which is that the ladder has exactly one real break in it — between rung four and rung five.

Sam: Say more about what makes that break different from the others.

Koko: Below rung five, a human wrote the sequence of steps. So you can test the sequence. You know step one leads to step two, the router sends intent A down path A, you can trace it. Above rung five the model chooses part of the sequence at runtime. You can only test the outcome and the bounds — you cannot test the path, because you do not know the path in advance.

Sam: And that single line is where everything hard about agentic delivery actually lives.

Koko: Everything. Governance, rollback, audit trails, failure modes — all of it changes the moment you cross that line.

Koko: Here is the worked example. You are triaging twenty-two thousand customer emails a month into eight known intents, and then running the documented sequence for whichever intent it is. That is a router plus a chain. Rung four, testable end to end.

Sam: Because someone sat down and wrote every branch. The path exists on paper before a single email arrives.

Koko: Right. Now change the problem. You are investigating a customer short-payment where the evidence differs on almost every claim and the contract is sometimes ambiguous. Nobody can write the sequence in advance, because what you look at next depends on what you just found.

Sam: That crossed the line the moment the path stopped being knowable.

Koko: Rung five. And the analogy that makes it stick for me: a recipe versus a cook improvising with what is in the fridge. A recipe you can check step by step — did you add the flour before the eggs, yes or no. A cook improvising you can only judge by what comes out, and by what you refused to let them touch.

Sam: Which is why the governance question at rung five is always about bounds — what can the agent reach, not what will it do.

Koko: Exactly. So the takeaway to carry into the room: name the rung, and say out loud why the rung below it will not do. Not just 'we need an agent' — 'we need an agent because no one can enumerate the sequence in advance, and here is why.' And if the honest answer is rung zero, say rung zero. The credibility you earn by declining to over-engineer is what buys you the mandate when you genuinely need the hard stuff.

Sam: I have seen that pay off. Saying 'this is a rule' in a room full of people who came to talk about agents lands better than you'd expect.

Koko: Because it signals that you are optimizing for outcome, not for impressiveness.

Koko: Two misconceptions here, and I want to name both clearly. First one: the ladder is not a maturity model. Rung six is not better than rung zero. Climbing is a cost you are paying, not an achievement you are unlocking. An organization sitting at rungs zero and four, doing it cleanly, may be considerably more mature than one running agents everywhere because agents felt exciting.

Sam: The version of this I keep seeing on teams is a kind of ambient pressure to reach for something newer. The ladder language helps because it reframes the question — not 'are we being ambitious enough' but 'are we paying the right cost for this outcome.'

Koko: And the second misconception is about what naming the top rung actually means. The ladder is cumulative, not exclusive. A real rung five system almost always contains rung zero validators inside it, rung two extraction, rung three retrieval. Naming the top rung tells you the governance regime — it tells you where the hardest accountability question lives. It does not mean everything is done by an agent.

Sam: So when you say 'this is a rung five,' you are saying the agent is the thing that needs the most careful bounds-setting, not that rules and retrieval have disappeared.

Koko: Right. The rungs below are still doing real work inside the system. You are just naming where control shifts from a written sequence to a model-chosen path — because that is the line where your delivery approach has to change.

Koko: Before a candidate gets a number — before anyone argues about impact scores or complexity ratings — there are five questions. And a candidate that fails one of them does not get a score. It gets a conversation.

Sam: Which is actually the right outcome at that point. Scoring something you are not allowed to build, or that has no owner, is just a more precise way of wasting the room's time.

Koko: Exactly. So the five. Permitted: are we allowed to do this, at this autonomy tier, on this step? Owner: is there a named person whose objectives improve if it works and who will actually change how the work is done? Baseline: can we measure the current state within two weeks, without building a new system first? Right to use: do we have the right to use this data, for this purpose, in this system, everywhere this business operates — and can we prove it? And cheaper rung: would documenting the rule, fixing the upstream data, or just removing the step deliver most of the value anyway?

Sam: That cheaper rung question is the one I see skipped most often. Teams get to a real problem, they want to solve it, and they jump past the question of whether a simpler intervention would already get them eighty percent of the way there.

Koko: And that skip is expensive. So let's make this concrete. Picture a longlist of eight candidates. One of them is prior-authorization triage — eighteen thousand requests a month, real turnaround pain, a regulatory response-time standard breathing down the team's neck. Looks great on paper.

Sam: And it fails Right to Use.

Koko: It fails Right to Use as written. The evidence you need to triage those requests is protected health data, and nothing in the current data agreements covers processing it in an AI environment. Not every jurisdiction, not every use.

Sam: So what happens to it on the longlist? Does it get cut?

Koko: It gets parked with a named condition — route it to privacy and the data owner, note what would need to be true before it re-enters, and move on. It is not a decline, it is a route. But it does not get a score today, and it does not sit there eating everyone's attention.

Sam: That distinction matters. Parked with a condition is a real status. It is not the same as killed, and it is not the same as live.

Koko: Right. Now the analogy. A building inspector does not rank your kitchen against your bathroom until they have checked that you are allowed to build at all. The gate is not pedantry — it is the thing that makes the ranking mean something.

Sam: And if you run these five questions at about ten minutes per candidate, on a longlist of eight, that is less than ninety minutes of structured conversation before the scoring model even opens.

Koko: And they typically remove a third to a half of the longlist. A third to a half. That does more for the quality of your portfolio than any scoring model, at a fraction of the effort.

Sam: The takeaway I carry from this is: the five questions are the filter that makes the score honest. If you score everything on the longlist you are not prioritizing, you are just ranking noise.

Koko: That is it exactly. Now — Permitted is first on the list for a specific reason, and it is worth naming. The other four questions change what you build or when. Permitted is the only one that can end the conversation entirely. If it fails, you are not adjusting scope. You are not lowering a score. You are withdrawing.

Sam: The version of this I keep seeing on teams is: someone looks at the permitted question, feels uncertain, decides they are probably fine, and carries the candidate through scoring, through design, sometimes through early build — and then the answer comes back no from legal or compliance. At that point it is not a delay. It is a withdrawal, and everything since the uncertainty was sunk cost.

Koko: The failure mode is discovering it late. And the specific thing that enables discovering it late is concluding yourself from a slide rather than routing the question to the function that actually owns the answer. If Permitted belongs to legal, compliance, or a data owner — route it there before scoring. Do not infer from what you have in the room. Get the answer from the person who can give it.

Koko: So the scorecard has twelve weighted factors — six on value, six on complexity — and the weights are not equal. On the value side: financial impact carries thirty percent, volume times frequency twenty, cycle time and service fifteen, risk and control fifteen, strategic and experience ten, reusability ten. Complexity side: data readiness twenty-five, variability and exceptions twenty, integration and write surface twenty, risk and regulatory exposure fifteen, change and role impact ten, evaluability ten.

Sam: And every one of those factors is scored against a written anchor, not a gut feel. That's the whole mechanism — the anchor is what gives the number a meaning that two different people can agree on.

Koko: Exactly. And before we go further there is one thing in the complexity axis that I want to name outright, because it is consistently misread. Complexity here is delivery difficulty and risk. It is not a measure of how sophisticated the AI is.

Sam: Which means a technically trivial solution can be high complexity.

Koko: Right. If it writes to the general ledger, it is high complexity — full stop. And a sophisticated model that produces a draft for a human to read is low complexity, because the delivery risk is low. The sophistication of the model and the difficulty of the delivery are two different axes.

Sam: That distinction matters enormously when you're prioritizing. You can have something that looks impressive technically and ranks low on complexity — and something that looks simple and ranks high — and both of those are correct scores.

Koko: Now let me make this concrete. You have a room, eight candidates, and you ask people to score them on a one-to-five feel with no written anchors. What happens?

Sam: Everything lands around three and a half. Nothing separates. The exercise was supposed to produce a ranking and it produces a cluster.

Koko: Which is the one thing the exercise exists not to do. Now anchor one factor — financial impact, say. 'A credible number with a stated basis, but not one the owner reports externally' is a three. 'Moves a figure the owner reports' is a five. Suddenly two people who gave the same candidate different scores can find out why in one sentence.

Sam: Because the anchor externalises the reasoning. The disagreement becomes visible and resolvable instead of just sitting there as noise.

Koko: Think of it like a wine score out of a hundred with no tasting notes. That number tells you something about the taster. The notes are what let you argue.

Sam: And what you're arguing about is calibration, not taste. You're not fighting over opinions, you're aligning on what the words mean.

Koko: So the instinct to carry into the room: agree the weights with the sponsor in week one and then freeze them. Because the moment someone re-weights mid-exercise to move a candidate up the list, the model loses its authority — and everyone in the room has watched that happen.

Sam: The version of this I keep seeing on teams is the re-weighting happens quietly, between sessions, and by the time anyone notices, the ranking has changed and no one can explain why. The model looks like it's doing the work but it's just ratifying a decision that was already made.

Koko: Which is a good segue into where things actually go wrong late. Two factors on the complexity axis have a habit of surfacing after initial scoring, and both of them are expensive to discover late. The first is write access.

Sam: Write access turns a pilot into a controls project almost immediately. The read-only version of nearly any candidate is dramatically cheaper and faster, and it captures most of the value.

Koko: So if you discover late that the candidate writes to a system of record, you are not repricing a feature — you are repricing the whole engagement. The second one is evaluability. If there is no agreed definition of what a correct answer looks like, and no ground truth to test against, you are not scoring a candidate. You are scoping a definition exercise.

Sam: And no amount of engineering downstream rescues that. You can build the most capable pipeline in the world and still have no way to know if it's working.

Koko: The misconception here is that a three in the middle of the scale is a considered judgment. Often it is an avoidance. If the score is not anchored to a written description, a three just means the scorer did not want to commit. And that ambiguity does not resolve itself — it travels downstream into the build.

Koko: Last module. And I want to start with the shape of the whole method, because this is where it either clicks or it doesn't. There are five moves, and they go in strict order. Eliminate, simplify, standardize, determinize, delegate. AI is move five. It is also the most expensive move on the list.

Sam: And expensive here means not just cost to build — it means evaluation overhead, controls design, monitoring, the whole apparatus you have to stand up around a model before you can trust it in production.

Koko: Exactly. Moves one through four need none of that. No model, no evaluation suite, no controls design. So let's name them fast. Eliminate: ask whether the output has a consumer who actually acts on it, whether the check has a cause that could be fixed upstream, whether the approval ever declines. If the answer is no, the step goes.

Sam: Simplify next — that's removing handoffs, approval layers, duplicate entry, queue time.

Koko: Then standardize: agree one way of doing it, write the rule down, reduce the variant count. Then determinize: whatever is now a written rule, implement it as code. A routing table, a threshold, a formula. And only then — delegate. What is left after all four of those passes, the judgment on unstructured evidence, the exceptions that cannot be enumerated — that goes to whatever rung suits it.

Sam: And the rung might be a human, a rules engine, or a model. The method doesn't prejudge that. It just ensures the thing you're delegating is the irreducible residue, not the whole original mess.

Koko: That's a good way to put it. Let's make this concrete with the deductions process we've been building on across this lesson. You run the five-move sequence. What do you find?

Sam: The biggest elapsed-time win is an approval-threshold change — the sort of approvals that were always rubber-stamped because they came in under a certain amount. Eliminate that step. Second biggest is routing: once you write the routing rule down, it turns out to be deterministic. You implement it as code. Neither of those is AI.

Koko: Right. A team that jumped straight to an agent for the judgment step — the genuinely ambiguous deductions — would have delivered the smallest share of the total benefit. And they would have carried all of the control burden for the privilege.

Sam: The version of this I keep seeing on teams is that they scope the AI component first because it's the interesting part, and then they design the surrounding process to feed it — which means all the waste and variance goes in with the work.

Koko: And here's the analogy that I think makes the mechanism stick. Before you hire a specialist to handle your correspondence, you cancel the subscriptions you don't read, stop printing what nobody files, and put the recurring ones on a standing instruction. What's left is worth a specialist. You wouldn't hire the specialist and then hand them the unsorted pile.

Sam: Because the specialist ends up spending most of their time on the subscriptions.

Koko: And charging specialist rates for it. So the takeaway — the instinct to carry into the room — is this: moves one through four need no model and no evaluation suite. Do them first, and what remains for move five is small enough to actually prove. That's the goal. A provably small AI component, not a large one.

Sam: Small enough to evaluate cleanly. Small enough to monitor. Small enough that when something goes wrong, you know where to look.

Koko: Now the failure mode — and this one has a name. It is the difference between reimagining a process and automating one. A team that goes straight to move five builds a system that has to absorb the variance, the duplicate entry, the undocumented regional rule differences — all of it. And then they discover that the variance is precisely why it cannot pass evaluation. The model can't generalize cleanly because the inputs were never standardized. It can't route reliably because the routing rule was never written down.

Sam: So the evaluation failure is actually a process design failure that showed up late.

Koko: Late and expensively. The order of the five moves is not a preference. It is what makes the AI component small enough to work.

Sam: And if you've done this lesson in order, you've now got the full stack: value identification, the prioritization gates, the effort and benefit scoring, the process moves — eliminate through delegate — and why the sequence matters.

Koko: That is the stack for Part One. Part Two is where we get into the harness — the controls, the evidence you collect, and the number that governs all of it once you hand something to a model. That's next.

Sam: Worth knowing before you build anything.

Koko: The method, the prioritization gates, the scoring model, the workflow moves, and the sorting lab from earlier in the lesson — all of it is on the course page at KokoAI Academy, at koko knows dot A I. Go do this on a real process. That's where it gets good.