Choose the Right Thing

Chris: Here is what I actually want from this: something I can walk into a room with on Monday, where nobody has agreed on what AI even means, and still move the conversation forward. That is the thing.

Koko: And that is exactly the premise. The model you pick will change twice before the thing you build reaches production. Almost none of the reasons these projects fail are model reasons. So the thing that has to be stable is the method — the questions, the vocabulary, the artifacts. Platform specifics come later. Right now we are building the skeleton that holds regardless of what the model underneath turns out to be.

Chris: Which means if someone swaps the underlying model six weeks in, the method does not fall over.

Koko: Exactly. And the one instinct I want you to carry out of this module: the questions are the deliverable. The letters we are about to give you are only how you remember them. Do not get attached to the acronym — get attached to the questions.

Chris: Okay, so walk me through the structure.

Koko: Five phases. Align, Ground, Equip, Narrow, Trace and Tune. Align asks: what outcome matters, what is the baseline, and why is this the smallest sufficient solution? Ground asks: how should the work itself change, who decides what, and what evidence does the system actually need to do its job? Equip asks: what may the system know, decide and call? Narrow asks: what must be prevented, approved, detected, reversed or escalated — and critically, outside the model? And Trace and Tune asks: is it reliable, observable, affordable and supportable over time?

Chris: So Align is where you stop someone from handing you a solution and make them hand you a problem instead.

Koko: That is the right way to put it. Let me make it concrete. A team gets asked to put AI on the deductions queue. That is the brief. Full stop. What does Align do with that?

Chris: It turns the phrase into a number. So you would find out — what is the volume, what is the cost of the current state, and who owns the outcome?

Koko: Right. Twelve hundred claims a month. Eighty-nine minutes of touch time each. Four point one million dollars written off unexamined last year. And a named owner whose targets actually improve if this works. Nothing has been built. No model has been chosen. And the conversation in the room has already changed completely, because now everyone is talking about the same problem.

Chris: That is doing a lot of work before a single line of code.

Koko: That is the point. Think of it this way: a builder does not start with the material. They do not walk onto a site and say, right, I have got timber, what do I do with it? They start with what the room is for, who walks through it, and which walls are holding the roof up. You have to know that before you touch anything.

Chris: And if you skip that, you end up with a beautiful wall in the wrong place.

Koko: Or you knock out a load-bearing one. Which is approximately what happens when teams jump straight to Equip — what can we give the AI access to — before they have done Align and Ground. Two thirds of the real work is in Align, Ground and Narrow. Those are the phases with no code in them. Teams that skip to Equip are the teams that rebuild.

Chris: The version of this I keep seeing on steering committees is: this is just discover, design, build, test, run — you have just given the phases new names. And I want to be able to answer that cleanly.

Koko: The answer is Narrow. That is the structural difference, and it is why the fourth letter exists at all. Standard delivery lifecycles fold controls into design and testing. You figure out what could go wrong, you test for it, you ship. For a probabilistic system that is not enough. The controls have to be a separate phase with a separate owner, because they have to be enforced outside the model. The model does not prevent itself from doing the wrong thing — something else does. That separation does not exist in a standard software delivery method, and if you do not build it in deliberately, it does not get built.

Chris: So Narrow is not a checklist inside design. It is its own conversation with its own accountable person.

Koko: Exactly. And that is the thing to say in the room when someone tells you this is just renamed Agile. You can run Agile sprints inside any of these phases. The method is not competing with your delivery framework — it is sitting above it, making sure the right questions get asked before the sprints start. The questions are the deliverable. Hold onto that.

Koko: So the ladder. Seven rungs, and the rule is simple: you test a candidate from the bottom up and you stop at the first rung that reliably delivers the outcome. Rung zero, rules and automation. Rung one, predictive machine learning — a number or a class from your own historical data. Rung two, a single model call. Rung three, a retrieval-grounded assistant. Rung four, a coded workflow — chain, router, parallel, evaluator-optimizer. Rung five, a bounded single agent. Rung six, a multi-agent system operating across an organizational boundary.

Chris: And the thing I keep coming back to is that most real systems I've scoped land at rung four. People come in expecting to build an agent and it turns out someone can write the steps down, so you just write the steps down.

Koko: That is exactly right, and it connects to the most load-bearing idea in this whole course. The ladder has exactly one real break in it — between rung four and rung five. Below that line, a human wrote the sequence of steps, which means you can test the sequence. Above it, the model is choosing part of the sequence at runtime, which means you can only test the outcome and the constraints you put around it. Everything that is genuinely hard about agentic delivery flows from that single line.

Chris: So the break is not about capability, it's about observability. Below rung five I can trace every step. Above it I'm tracing inputs, outputs, and guardrails — not the path.

Koko: Precisely. Here is the scenario that makes it concrete. Triaging twenty-two thousand customer emails a month into eight known intents and then running the documented sequence for whichever intent fires — that is a router plus a chain, rung four, and you can test it end to end. Now take a different problem: investigating a customer short-payment where the evidence conflicts on almost every claim and the contract is sometimes ambiguous. Nobody can write the sequence in advance, because what you look at next depends on what you just found. That is rung five, and it crossed the line the moment the path stopped being knowable ahead of time.

Chris: The discriminator is whether the path is knowable before runtime, not whether the task is complex. A complex task with a knowable path is still rung four.

Koko: Say that at every design review. The analogy I use is a recipe and a kitchen. A recipe you can check step by step — you can audit every instruction before anything goes in the oven. A cook improvising with what's in the fridge you can only judge by what comes out and by what you refused to let them use. The guardrails are the pantry lock, not the recipe.

Chris: And the takeaway for the room is name the rung and say out loud why the rung below it won't do.

Koko: That's it. And if the honest answer is rung zero, say rung zero. The credibility you earn by declining to over-engineer is what buys you the mandate when you genuinely need the harder thing.

Koko: Before we get to the failure modes, one more beat, because it decides most model choices in practice. A model is a probabilistic component with a token budget. Everything it reads spends from one context window. The same input can produce a different answer on two consecutive calls. Extra reasoning time buys accuracy, but at a real cost. And a hosted model can change on its provider's schedule without you asking it to.

Chris: Which is why we fix the outcome and the evaluation criteria first and then pick the cheapest model that passes.

Koko: Right. Fix what good looks like, run the evaluation, pick the cheapest model that clears the bar, and pin its version. When answers fall short, you pull three levers in order: better instructions and examples first, then retrieved evidence, then fine-tuning.

Chris: Why is fine-tuning last? I'd expect someone to reach for it earlier — it feels like the most direct way to shape behavior.

Koko: Because it is the most expensive way to get a stale answer. Training shapes behavior, it does not reliably add facts. By the time your fine-tuned model is deployed, the policy it memorized may already have changed. Better instructions and retrieval are live — they update when your documents update. Fine-tuning is a last resort, not a first move.

Chris: So the order of levers is basically cheapest-and-most-current first, most-expensive-and-hardest-to-update last.

Koko: Exactly that.

Koko: Okay, two misconceptions, and I want to name both of them clearly. First: the ladder is not a maturity model. Rung six is not better than rung zero. Climbing is a cost, not an achievement. An organization sitting steadily at rungs zero and four may be considerably more mature than one running agents everywhere, because the first organization chose the right tool and the second one is managing complexity it did not need.

Chris: The version of this I keep seeing on teams is someone presenting a rung five proposal and framing it as 'we're ready for the next level' — as if the rung number is a badge rather than a governance decision.

Koko: And the question that cuts through it is: what outcome fails if we use rung four? If there is no good answer to that, the proposal is climbing for the sake of climbing. Second misconception: the rungs are cumulative, not exclusive. A real rung five system almost always contains rung zero validators, rung two extraction calls, and rung three retrieval inside it. When you name the top rung, you are naming the governance regime — you're saying the model is making runtime path decisions and those decisions need to be bounded and audited. You are not saying everything inside the system is agentic.

Chris: So naming the rung tells you what kind of oversight you need, not what the internals look like.

Koko: That's the instinct to carry in. Name the highest rung present, say why the rung below it won't do, and then design your governance for that top rung — knowing the lower rungs are in there doing their jobs.

Koko: So before anything gets a score — before the matrix, before the prioritization conversation — there are five questions. And here is the rule: a candidate that fails any one of them does not get a number. It gets a conversation.

Chris: And that distinction matters. Not eliminated, not deprioritized — parked with a condition attached.

Koko: Exactly. So the five. First: Permitted — are we allowed to do this, at this autonomy tier, on this specific step? Second: Owner — is there a named person whose objectives improve if this works and who will actually change how the work gets done? Third: Baseline — can we measure the current state within two weeks, without standing up a new system? Fourth: Right to Use — do we have the right to use this data, for this purpose, in this system, everywhere this business operates — and can we evidence that? Fifth: Cheaper Rung — would documenting the rule, fixing the upstream data, or just removing the step deliver most of the value anyway?

Chris: That last one is the one that tends to surprise people. You are asking whether automation is even the right answer before you score how good an automation candidate it is.

Koko: Right. And that is the point. Think of it like a building inspector. They do not rank your kitchen against your bathroom until they have checked that you are permitted to build at all. The ranking conversation is downstream of the permission conversation.

Chris: Let me make that concrete. In a recent scoping exercise I was part of, we had eight candidates on a longlist. Prior-authorization triage was the most attractive one — around eighteen thousand requests a month, real turnaround pain, a hard regulatory response-time standard. On the surface it looked like the obvious top pick.

Koko: And then someone ran the five questions.

Chris: Right to Use stopped it cold. The evidence driving those triage decisions was protected health data, and nothing in the current agreements covered processing it in an AI environment. Full stop.

Koko: And here is the move: that is not a decline. That is a route. You park it with a named condition — this goes to privacy and to the data owner — and it stays on the longlist until that condition is resolved or it genuinely cannot be. You do not score it. You do not argue about it. You route it.

Chris: Which is different from what I see teams do, which is to note the concern and score it anyway with a risk flag attached. The flag disappears by the second review meeting and suddenly you are three months into design on something you were never cleared to build.

Koko: That is the failure mode, and it is worth naming clearly. A prohibited use found after design is not a delay — it is a withdrawal. You do not slow down. You stop. And the deeper version of this is where the delivery team concludes the answer themselves from a slide deck rather than routing it to the function that actually owns the decision. Permitted is not a box the project team checks. If the answer belongs to legal, to privacy, to compliance — it goes there before scoring, not after.

Chris: And Permitted is the only one of the five that can actually end the conversation. The other four — Owner, Baseline, Right to Use, Cheaper Rung — they change what you build or when. Permitted can close it entirely.

Koko: Which is exactly why it is first. So here is the takeaway. Ten minutes per candidate, five questions, and they remove a third to a half of a typical longlist before anyone has argued about scores. That is more impact on portfolio quality than the scoring model itself delivers, at a fraction of the effort. Run the gates first. Score what survives.

Koko: So we have twelve factors, six on value and six on complexity, each carrying a weight. On the value side: financial impact at thirty percent, volume times frequency at twenty, cycle time and service at fifteen, risk and control at fifteen, strategic and experience at ten, reusability at ten. Complexity mirrors that structure: data readiness at twenty-five percent, variability and exceptions at twenty, integration and write surface at twenty, risk and regulatory exposure at fifteen, change and role impact at ten, evaluability at ten. Now those weights only do anything if every score on every factor is tied to a written anchor — a sentence that describes what a three actually means, what a five actually means.

Chris: And without that, you get the thing I have watched happen in almost every prioritization room I have ever been in: eight candidates, everyone scores on feel, and everything lands around three and a half.

Koko: Which is the one outcome the exercise exists to prevent. If nothing ranks, you walked out of the room with the same list you walked in with.

Chris: So what does an anchor actually look like? Give me the financial impact factor, because that one gets argued the most.

Koko: Sure. A three on financial impact is: a credible number with a stated basis, but not one the owner reports externally. A five is: moves a figure the owner reports — revenue, cost, a regulatory capital line. Now two people who disagree can find out why in one sentence. One of them thinks the number is externally reported. The other does not. That is a resolvable argument.

Chris: Versus the unanchored version, where the argument is just 'I feel like this is more important than that.'

Koko: Right. And here is the analogy I keep coming back to. A wine score out of a hundred from someone with no tasting notes tells you about the taster. The notes are what let you argue. Without them you are just trading opinions about whose palate to trust.

Chris: That tracks. The anchor is the tasting note. It externalizes the reasoning so someone else can challenge it on the same terms.

Koko: Exactly. Now there is a specific thing I want to flag about the complexity axis, because it is the one that gets inverted most often in practice. Complexity in this model means delivery difficulty and risk — not how technically sophisticated the AI is.

Chris: That is a real distinction. I have seen teams score a rules-based automation as low complexity because it felt simple to build, when it was writing directly to the general ledger.

Koko: That is high complexity by this model, full stop. And the flip side: a candidate that uses a genuinely sophisticated model but only produces a draft for a human to read — that is low complexity, because delivery risk is contained. The human is the control.

Chris: So the axis is really about what can go wrong in production, not what is impressive to build.

Koko: Precisely. The instinct to carry into the room: agree the weights with the sponsor in week one and then freeze them. Because re-weighting mid-exercise to change a ranking is how a scoring model loses its authority. And everyone in that room has watched it happen — someone's preferred candidate starts losing and suddenly strategic alignment becomes thirty percent instead of ten.

Chris: The version of this I keep seeing on teams is that the re-weighting gets framed as refinement. Like, 'we learned something, so we updated the model.' But if the learning was 'our candidate is losing,' that is not refinement, that is retrofitting.

Koko: And once the room sees it happen once, the model is gone. Nobody trusts a score that can be adjusted after you see the results. Freeze the weights. Debate the anchors, debate the scores — but freeze the weights.

Chris: There are two complexity factors that I know from experience get discovered late and cost the most when they do. Write access and evaluability.

Koko: Both on the complexity axis, both capable of completely reframing a candidate. Write access first: the moment a pilot writes to a system of record, you are in a controls project. Audit trail, reconciliation, rollback, sign-off chains — the read-only version of almost any candidate is dramatically cheaper and faster, and it captures most of the value. That is almost always worth scoping explicitly before you commit.

Chris: And evaluability is the one that is hardest to recover from. If there is no agreed definition of what a correct answer looks like, and no ground truth to test against, you cannot validate anything downstream.

Koko: You are not scoring a candidate at that point — you are scoping a definition exercise. No amount of engineering fixes the absence of a ground truth. And here is where the anchor failure mode shows up most clearly: a three that is not tied to a written description is not a middle position, it is avoidance. Someone in the room does not want to say they do not know, so they write down three. The anchor forces the question into the open where it can actually be resolved.

Koko: Alright, last one, and it is the one that ties the whole lesson together. There are five moves, and they go in strict order. Eliminate, simplify, standardize, determinize, delegate. Delegation to AI is move five. It is the last move and the most expensive move.

Chris: And expensive in what sense? Cost to build, or cost to run, or both?

Koko: All of the above, but the one that really bites is the controls burden. The moment you delegate to a model you need an evaluation suite, a monitoring layer, a fallback path, a controls design conversation. That is real overhead. Moves one through four need none of that.

Chris: Right, because eliminate through determinize are just logic and governance work. Nobody is asking you to prove that an if-then statement generalizes.

Koko: Exactly. So let's walk the five moves fast. Eliminate: does the output have a consumer who actually acts on it? Does the check have a cause that could be fixed upstream? Does the approval ever decline? If the answer to those is no, the step should not exist.

Chris: The approval-that-never-declines is the one I keep having to surface for people. It feels like a control. It is just a delay.

Koko: It is a very official-looking delay, yes. Then simplify: remove handoffs, approval layers, duplicate entry, queue time. Then standardize: agree one way of doing it, write the rule down, reduce the variant count. Then determinize: whatever is now a written rule, implement it as code. And then, only then, delegate. What is left — judgment on unstructured evidence, exceptions that genuinely cannot be enumerated — goes to the rung that suits it.

Chris: The deductions process from earlier in the lesson is the cleaner of this, isn't it. Walk me through where the actual time came out.

Koko: So when the team did the analysis, the biggest elapsed-time win was an approval-threshold change and the determinization of the routing. Neither of those is AI. A team that had jumped straight to an agent for the judgment step would have delivered the smallest share of the benefit and taken on all of the control burden for it.

Chris: Smallest share of the benefit because the judgment step is actually a small slice of the total cycle time once you have cleared the upstream waste.

Koko: That is the mechanism. The judgment step looks large when the process is full of noise. Strip the noise out and the genuine ambiguous cases are a small residual. That is what move five is supposed to touch.

Chris: The analogy you have for this one is good. Use it.

Koko: Before you hire a specialist to handle your correspondence, you cancel the subscriptions you do not read, you stop printing what nobody files, and you put the recurring items on a standing instruction. What is left after that is actually worth a specialist's time. You have not handed a specialist a pile of junk and asked them to sort it.

Chris: And if you do hand them the junk pile, the cost of the specialist goes up and the output quality goes down.

Koko: Which is precisely the failure mode. The version of this I keep seeing on teams is they jump to move five, they build an agent, and then the agent has to absorb the variance, the duplicate entry, the undocumented regional differences that nobody cleaned up. And then they discover that the variance is exactly why the system cannot pass evaluation. It is not a coincidence. The disorder they skipped is now inside the model's problem.

Chris: So the order is not a stylistic preference. It is what makes the AI component small enough to actually prove.

Koko: That is the takeaway. Do moves one through four and what remains for move five is small enough to evaluate, small enough to monitor, small enough to govern. The order is not a preference. It is what makes the AI component tractable.

Chris: Alright. Let's close the lesson out. What are we carrying forward from today?

Koko: Five instincts. First: find the real constraint before you find the solution — clock the step, map the path, locate where time actually lives. Second: score for feasibility and value together, and let the gate protect the roadmap from work that is not ready. Third: understand the three workflow moves — remove, reroute, or augment — and know what the human role becomes under each one. Fourth: when you reach the sorting decision, grade the task on structure, reversibility, and consequence, and match the rung to the grade. And fifth: delegate last, and delegate less — eliminate, simplify, standardize, and determinize before you ever get to move five.

Chris: And the thing connecting all five is that the design work happens before you build, not while you build.

Koko: That is the whole lesson in one sentence. The method, the prioritization gates and scoring model, the workflow moves, and the sorting lab are all on the course page at KokoAI Academy — that is koko knows dot A I. Part Two picks up where this leaves off: the harness, the controls, the evidence, and the number that governs all of it. You have done the hard conceptual work today. The rest is applied.