Evaluation, Testing & Optimization

Sam: Here is what I actually want from this domain: an order of operations sharp enough to defend a decision that cost money. Not just 'we tested it' — something I can stand behind in a review when someone asks why we made the call we made.

Koko: And that is exactly what domain four is testing. Sixteen percent of the Professional exam, and it is the domain that decides whether everything else you built can be defended. An architecture with no measurement is an opinion. A modest architecture with a baseline, a fixed evaluation set, and a declared metric is a position you can hold.

Sam: The exam question version of that, the way I have seen it come up, is not 'was your evaluation thorough' — it is 'can this result actually distinguish the claim you are making from the claim sitting next to it.' That is the discriminating question every time.

Koko: Exactly right. Almost every wrong answer in this domain is someone reading a number that cannot support the conclusion drawn from it. So let's start with the fact — how evaluation criteria actually work according to Anthropic's guidance. The criteria have to be specific, measurable, achievable, and relevant, and they have to cover several dimensions at once. Fidelity, consistency, tone, privacy, latency, cost — those are not interchangeable. You need readings on each.

Sam: And safety gets a number too. That is the part teams underestimate. 'Safe' is not a criterion. 'Under one tenth of one percent of outputs flagged' is a criterion.

Koko: That is the precise version of it, yes. Quantify the soft criteria or you cannot compare runs. Now let me make this concrete. Here is the scenario. A team is building a clinical-intake summarizer. Before a line of code is written, they publish five criteria. First: factual coverage against a labeled set, with a target, a method, and an owner. Second: zero identifiers appearing outside the designated field — a hard binary. Third: ninety-fifth-percentile latency under six seconds. Fourth: a cost ceiling per summary. Fifth: a tone score of four out of five on a rubric. Each one has a target, a measurement method, and a named owner.

Sam: That last one is worth pausing on. Tone score of four on five. That is a soft criterion that got quantified. You cannot track drift on 'appropriate tone' — you can track drift on a score with a rubric behind it.

Koko: Right. And notice there are five separate criteria, not one. One accuracy number for a production system is almost always an under-specification. The analogy I use for this: 'bake until done' cannot be followed twice the same way. 'Bake until a skewer comes out clean' can. And no cake is judged on a single axis — rise, crumb, and color are separate readings. You do not average them into one cake number.

Sam: I like that because it also captures why criteria have to be set before you start. You decide what 'done' means before the oven goes on, not after you taste it.

Koko: Which brings us straight to the takeaway. The instinct to carry into the room: decide what you will measure before you build, then change one thing at a time against a set that does not move. Set your targets against benchmarks or your own prior experiments — not against frontier claims about what a model is supposed to do in general. And quantify the soft criteria instead of dropping them.

Sam: The 'change one thing at a time' part is doing real work there. If the evaluation set moves while the system is also changing, you cannot attribute any delta to anything.

Koko: Correct — and that is what makes a result defensible versus just a number. Now, the trap. And I want to name this one directly because it is the most expensive failure mode in this domain. The trap is building the system first and then working out how to evaluate it. Teams treat evaluation as a phase that comes after the build. It is not. Criteria are a design input. Here is why that order matters: in the clinical summarizer example, the criterion 'zero identifiers outside the designated field' rules out passing raw text to a third-party tool. If you discover that constraint after you have already integrated the tool, you are redesigning, not evaluating. The criteria constrain your architecture. Learning them late is the expensive order.

Sam: The version of this I keep seeing on teams is: 'we will figure out the metrics once we see what the model can do.' That sounds pragmatic. It is actually just deferring the design decision to the point where it costs the most to reverse.

Koko: That is the misconception precisely. Evaluation is not a checkpoint at the end. It is the specification you build toward. Set the criteria first, fix the evaluation set, quantify everything including the soft stuff, and then you have something you can defend.

Koko: So we just covered how to design your eval set. Now the question is: once you have those cases, how do you actually grade the answers? And the fact here is that the method is determined by the shape of the right answer. There are four. Exact match for categorical outputs — did it pick the right label, yes or no. Similarity measures for paraphrase consistency — does this restatement mean the same thing. Overlap measures like ROUGE for summarization, where you're comparing against a reference. And model-based grading on a Likert, ordinal, or binary rubric for anything subjective. Four distinct instruments, and which one you reach for depends entirely on what a correct answer even looks like.

Sam: Right, and the selection criterion is really about what varies in a correct response. If there's one right answer, exact match. If the space of correct answers is wide — tone, helpfulness, coherence — you need a rubric and a model to apply it.

Koko: Exactly. And Anthropic's guidance is explicit: prefer many automatically graded cases over few hand-graded ones. Larger samples are more reliable. That's not a convenience argument — it's a statistics argument.

Sam: Walk through the concrete scenario? I want to hear how the three methods actually sit alongside each other in one eval.

Koko: Perfect example for this. Imagine a support-routing system. You run exact match on the routing label — is this ticket tagged as billing, technical, or account — across a thousand real tickets. That's automatic, fast, and you get real statistical power from that sample size. Then for the summary field, you run a binary model-graded check: does this summary contain personal data, yes or no. Binary rubric, model as judge. And then on a stratified hundred tickets — deliberately including sarcasm, typos, empty inputs, and one unanswerable ticket — you run an ordinal tone rubric. Professional, neutral, or curt. Three different instruments, one pipeline.

Sam: The stratification detail matters a lot. That hundred isn't random — you're guaranteeing the hard cases are in there. Sarcasm, empty input, the edge the model is most likely to fail on.

Koko: Right. Mirror the real distribution of your traffic, then add the edge cases on purpose. Don't leave it to chance whether your sample happens to contain a typo-heavy ticket or an unanswerable one.

Sam: The analogy I use for this is a factory line. You run a go/no-go gauge on every single part off the line — that's your exact match at scale. You use a scale for weight checks. You bring in a trained eye for surface finish. No one instrument answers every question, and you don't inspect thirty parts by hand and call it rigorous.

Koko: That's the analogy exactly. The gauge runs on every part, not on thirty picked up by hand. The hand inspection is for the question the gauge can't answer — surface finish, tone — not as a replacement for volume.

Sam: One thing I want to pin down — when you use a model as judge on that rubric, is the grading model the same model you're evaluating?

Koko: No, and that's a structural point. You judge with a different model. If the same model grades its own outputs you get a systematic bias — it will find its own reasoning coherent even when it isn't. Separate the system under test from the grader.

Sam: The instinct I'd carry in: structure your questions so they grade automatically wherever possible, use the rubric-plus-model path only where the answer space is genuinely open, and keep your sample large enough to detect regressions.

Koko: That's the takeaway. Now the trap, because the exam will put this right next to the right answer. The misconception is that a small hand-graded set is more rigorous than a large auto-graded one. The reasoning sounds plausible — human judgment, careful review, domain expertise. But thirty cases are dominated by sampling noise. You cannot detect the regression you care about in thirty cases. Reliability comes from sample size, and auto-grading is how you get sample size. Human review is for calibrating the rubric, not for replacing volume.

Sam: The version of this I keep seeing on teams is someone saying they had five experts grade fifty outputs and that's their gold standard. And it feels careful. But if you change a prompt and you want to know whether routing accuracy dropped two points, fifty cases won't tell you.

Koko: Exactly. The gold is in the sample size. Method follows the shape of the answer, volume follows the need for reliability, and those are two separate decisions.

Koko: Before you touch a single word of your prompt, you need to know whether the prompt is actually the problem. Here is the fact: a wrong answer has at least five candidate causes, and the prompt is only one of them. The model could have received an ambiguous instruction. It could have fabricated a fact. A relevant passage might exist in your corpus but was never retrieved. A tool result might have come back and the model misread it. Or the model tier you are running is simply below the task. Each of those has a different fix, and only one of them is prompt work.

Sam: That list matters a lot in practice. The instinct on most teams is to open the system prompt the moment something goes wrong, and I have done it myself — edited three lines, re-ran the eval, and the defect was completely unchanged because the retrieval step had never surfaced the relevant document in the first place.

Koko: Exactly the scenario in the notes. Contract assistant, cites clauses that are not in the document. What do you check first?

Sam: You pull the retrieved passages for that query and you ask: was the clause even there? If it was not in the retrieved set, you have a chunking or retrieval problem, full stop. The prompt never saw the clause, so the prompt could not have cited it correctly.

Koko: And if the passage was present in the retrieved set and the answer was still wrong?

Sam: Then you are in a different situation — the model had the information and still got it wrong. That is when the documented structural remedies apply: permit the model to say it does not have enough information, require word-for-word quotes before any analysis, and build in a step where unsupported claims get retracted.

Koko: Those three moves are specific because they address what actually went wrong at that layer. Permitting the model to say it does not know removes the pressure to confabulate. Requiring verbatim quotes before analysis forces grounding — the model has to point at real text before it reasons about it. And retracting unsupported claims is a verification step baked into the output format itself.

Sam: The discriminator the exam is probably looking for: none of those are prompt-quality changes in the usual sense. They are structural changes to what inputs reach the model and what the model is required to do with them before it answers.

Koko: Right. Here is the analogy I use. The water tastes wrong. You do not immediately replace the tap. You taste it at each point along the pipe — mains supply, storage tank, the pipes themselves, and then the tap. You replace the tap only after you have confirmed the problem is at the tap. The tap is just what you can reach most easily, not necessarily where the fault is.

Sam: And in a RAG system the equivalent of tasting at each point is logging the inputs at every layer so you can answer four questions: was the relevant content even retrievable from the corpus, was it retrieved for this query, was it present in the context window, and did the model actually use it in its answer.

Koko: That is the takeaway, stated precisely: log the inputs at every layer so you can answer what the model saw, then work outward — retrievable, retrieved, in context, used — and fix the earliest failing layer first. That is the decision rule. Earliest failing layer first.

Sam: The version of this I keep seeing fail on teams is that they conflate all of this under the heading of hallucination and treat it as a prompt problem. You hear it in incident reviews: the model hallucinated, so we need a better prompt.

Koko: And that is the trap to name explicitly. The misconception is that hallucination is a prompt-quality problem. It is a grounding and verification problem, and it has structural remedies. Anthropic's own framing is that those structural remedies reduce hallucination without eliminating it — which means for high-stakes outputs, human validation is still required regardless of how well you have engineered the prompt. The exam option sitting next to the right answer will say something like: improve the system prompt to instruct the model not to fabricate. That is not wrong as far as it goes, but it is the wrong layer for a retrieval gap and it is insufficient on its own for anything where errors carry real consequences.

Sam: So the professional answer is: diagnose the layer, apply the structural fix at the earliest failing point, and do not mistake a retrieval defect for a prompting defect.

Koko: That is it. Taste the water before you replace the tap.

Koko: Here is the first fact, and it is the one that makes every other optimization practice meaningful: improvement only compounds if each iteration is attributable. One variable changes. The evaluation set stays fixed across releases. The deciding metric is declared before you look at any numbers. And the counter-metrics — the ones that must not degrade — are named in advance too.

Sam: The counter-metrics piece is where I see teams get loose. They pick a primary metric, the number goes up, they ship — and two weeks later they notice latency crept up or cost doubled. Those were counter-metrics that were never written down.

Koko: Exactly. And nondeterminism is what makes the written-in-advance rule non-negotiable. A point or two on a small set is noise, not a result. If you didn't write the decision rule before you saw the output, you are fitting to noise and calling it progress.

Sam: The worked example I keep coming back to is a retrieval change: two arms, same eight hundred cases, recall at five as the deciding metric, cost per query and ninety-fifth-percentile latency as counter-metrics. Prompt, model, and tool set identical across arms. The decision rule is written down before the first number is returned.

Koko: That is the whole architecture of a valid test right there. And the analogy that makes the mechanism stick — think of a split-field agricultural trial. Same seed, same weather, one fertilizer changed on one half of the field. The yield is weighed at harvest against a rule written at planting. Nobody decides by walking out and looking at six plants.

Sam: And changing the fertilizer and the irrigation and the planting depth at the same time tells you nothing.

Koko: Nothing. You harvested something, you just don't know why. The instinct the exam rewards: freeze everything but the variable, keep the set stable across releases, and record the run — variant, set version, metrics, decision — so a reader a year from now can tell why the system is the way it is.

Koko: Now the second fact, and it belongs in the same module because it is just the first fact applied to cost. When what you are optimizing is cost, the order of changes is prescribed: the cheap moves come first, because they cost no accuracy.

Sam: Cache the stable prefix first. Cache reads are a documented fraction of input price — that is a real, measurable saving with no quality tradeoff.

Koko: Then stop sending what is not read. Dead tool results sitting in context, unused tool definitions, guidance that belongs in a Skill and not in the prompt — all of that is tokens you are paying for and the model is not using.

Sam: The classifier rewrite I shipped followed that order exactly. Caching the instruction block and label taxonomy took the largest share of the savings. Deferring twenty-eight rarely used tool definitions was the next step. The model tier change came last.

Koko: And it came last for a specific reason. The smaller model is the only step in that sequence that can move quality. So you do it last, you hold it on the evaluation set, and the set is the referee. That way, if quality moves, you know what moved it — because nothing else changed.

Sam: Measure before you optimize. The token-counting endpoint, the cache-read and cache-write figures — look at what a request is actually made of before deciding where to cut.

Koko: The analogy here: seal the drafty windows before you replace the furnace, and read the meter at each step. Sealing the windows changes the bill and nothing else. The furnace is the one change that alters how the house feels. You do not start with the furnace.

Sam: And if you replace the furnace first and the house is still cold, you do not know if the problem was the furnace or the windows.

Koko: Right. The instinct: take the accuracy-neutral savings first, so that if quality moves when you finally touch the model tier, you know exactly what moved it. Now — the traps, and there are two.

Sam: The one the exam puts right next to the right answer: the option that says a better-looking sample of outputs is evidence. The version of this I keep seeing on teams is someone reads ten outputs, prefers one arm, and calls it validated.

Koko: Reading ten outputs and preferring one arm measures your expectations. What you are shipping is what moved the declared metric without moving a counter-metric the wrong way. Qualitative preference is useful for hypothesis generation — it is not a decision.

Sam: What is the second trap?

Koko: That offline numbers describe the running system. The misconception here is that once your evaluation set passes, you are done. An evaluation set measures the cases you thought of, when you thought of them. Production drifts underneath. Live monitoring is a separate obligation — and it comes from your platform stack, because the source material for this exam says almost nothing on observability design. Offline evaluation and production monitoring are two different things. Conflating them is how you ship a system that was great on the set and quietly degrades in production.

Sam: Alright, let me try to say the through-line the way I actually think about it in practice. Before anything else gets built, you write down what good looks like in a way you can measure. That happens first, and it shapes everything downstream — the architecture, the grading approach, the optimization sequence. If you can't measure it before the build, the build is already in trouble.

Koko: That's exactly right, and it's the one that's easiest to skip when a team is in a hurry. You end up building something you can't evaluate, and then you're retrofitting criteria that quietly get written to match what you already shipped.

Sam: Rule two follows directly: once you know what you're measuring, the shape of the answer tells you how to grade it. Closed answers — classification, extraction, anything with a right or wrong — you can automate the grading and buy volume. Open-ended or nuanced answers you grade with a model evaluator. And in both cases you want volume over hand-curation.

Koko: Right. A hundred carefully hand-labeled examples is not as useful as a thousand consistently graded ones. The grading method is a choice that follows from the answer type, not from what's convenient.

Sam: Then rule three: when something is failing, you isolate the layer before you touch anything. Is it retrieval? Is it the prompt? Is it the model? You don't edit until you know which one is responsible, because an edit in the wrong layer either does nothing or breaks something else.

Koko: That isolation step is where a lot of time gets saved or lost. One wrong assumption about the layer and you spend a week tuning a prompt that was never the problem.

Sam: Rule four is about how you run changes once you've found the layer: one variable, fixed evaluation set, metric declared before you start. Not two things at once. Not a different set each run. The discipline there is what makes the result mean something.

Koko: And rule five closes the loop on optimization sequencing. You take the accuracy-neutral savings first — caching, prompt compression, output length, anything that doesn't touch quality — and model swaps come last. Because a model swap changes everything, so you want to exhaust the lower-risk levers before you introduce that variable.

Sam: The version of this I keep seeing teams get backwards is reaching for a smaller or cheaper model the moment costs look high, before they've even looked at whether a caching layer would solve it. And then they're chasing a quality regression they didn't need to introduce.

Koko: Exactly. Accuracy-neutral first, model swap last. That sequence is the instinct to carry in.

Sam: One more thing worth naming for the exam specifically. When two options both look defensible, the question to ask is: which one could actually be measured? The measurable option is almost always the right one. That's not a trick — it's the actual principle.

Koko: And paired with that: when a question opens with a symptom — users are complaining, quality dropped, latency spiked — the exam is almost always testing whether you can locate the layer before you reach for a fix. The symptom is the setup. The skill being tested is diagnosis.

Sam: Which is why the five rules run in the order they do. Criteria, grading method, isolation, controlled change, then optimization sequence. That order is the diagnostic process, not just a list.

Koko: Well said. That's Domain Four. The drills and flashcards for all five modules — the evaluation criteria, the grading methods, the RAG versus prompt isolation logic, the optimization sequence — all of it is at KokoAI Academy, at kokoknows dot AI. Work the flashcards until the decision rules feel like instincts, not recall.

Sam: That's the goal, right? By exam day these shouldn't feel like rules you memorized. They should feel like how you think.

Koko: Exactly that. You've already done the hard part just by being here. Go show them.