Evaluation, Testing & Optimization

Koko: An architecture with no measurement is an opinion. That's the whole domain in one sentence. Evaluation, testing and optimization is sixteen percent of the Professional exam, and the reason it sits that high is that it's the only domain where you can defend every other decision you made. You can have beautiful prompt engineering, a clever retrieval pipeline, a well-chosen model — and none of it means anything if you can't show a number that distinguishes your design from the one sitting next to it on the slide.

Chris: That's exactly what I need coming out of this module — an order of operations sharp enough to defend a decision that cost money. Because that's the moment that matters. Not the build, the review after.

Koko: Right. And the order of operations starts before the build, which is the whole point. So let's start with the fact — how evaluation criteria actually work according to Anthropic's guidance.

Koko: The guidance is specific, measurable, achievable, and relevant — across several dimensions at once. Not one number. Several. We're talking fidelity, consistency, relevance, tone, privacy, context use, latency, cost. And safety gets a number too. 'Under one tenth of one percent of ten thousand outputs flagged for toxicity by the content filter' — that's measurable. The word 'safe' by itself is not.

Chris: That's a reframe a lot of teams resist. They treat safety as a checkbox and latency as the real metric, because latency shows up in dashboards automatically. You have to go looking for the others.

Koko: You do. And the exam will test whether you understand that all of those dimensions need declared targets, not just the ones that are easy to instrument. So let me make this concrete.

Koko: The scenario: a clinical-intake summarizer. Before a single line of code, the team publishes five criteria. First, factual coverage against a labeled reference set. Second, zero identifiers appearing outside the designated output field. Third, ninety-fifth-percentile latency under six seconds. Fourth, a per-summary cost ceiling. Fifth, a tone score of four out of five on a defined rubric. Each one has a target, a measurement method, and an owner.

Chris: The identifier criterion is the one I find does the most work. 'Zero identifiers outside the designated field' — that's not just a quality criterion. That's an architectural constraint. Once you write that down, certain designs are off the table.

Koko: Exactly right. And that's where the analogy lives. A recipe that says 'bake until done' cannot be followed twice the same way. 'Bake until a skewer comes out clean' can. But also — no cake is judged on one axis. Rise, crumb, and color are separate readings. You don't declare the cake a success because the top browned nicely.

Chris: And you don't declare a model good because accuracy went up two points if you haven't looked at latency or cost since the last run.

Koko: That's the takeaway, right there. Set targets against benchmarks or your own prior experiments — not frontier claims, not 'state of the art,' not whatever the model card says. And quantify the soft criteria instead of dropping them. Tone is soft. 'A score of four on a five-point rubric measured by this method' is not. One accuracy number for a production system is almost always an under-specification.

Chris: The version of this I keep seeing on teams is calibrating against a competitor's published benchmark. Which sounds rigorous, but it means your eval set is theirs, your conditions are theirs — you're measuring whether you beat their claim, not whether you solve your problem.

Koko: That's a real one. Now — the trap, and I'm naming this one explicitly because it is the expensive mistake. The trap is building the system and then working out how to evaluate it. Criteria are a design input. The moment you write 'zero identifiers outside one designated field,' you have ruled out passing raw text to a third-party tool. If you learn that afterward, you are not refining — you are rebuilding. The exam will give you a scenario where evaluation was thorough but came after the decisions, and the question will be what went wrong. The answer is the order, not the thoroughness.

Chris: Decide what you measure before you build, then change one thing at a time against a set that does not move.

Koko: That's the instinct. Carry that into every question in this domain.

Koko: So the core fact for this module: there are four grading methods, and the right one is determined by what a correct answer actually looks like. Exact match for categorical outputs — the label either matches or it doesn't. Similarity measures when you're checking paraphrase consistency — does this restatement mean the same thing? Overlap measures like ROUGE when you're scoring summaries against a reference. And model-based grading on a rubric — Likert, ordinal, or binary — for anything subjective. Tone, helpfulness, appropriateness. Four tools, four answer shapes.

Chris: The way I think about it: the grading method is downstream of the output contract. Before you write a single eval, you ask what does a correct answer look like structurally, and that answer tells you the instrument.

Koko: Exactly right. And Anthropic's guidance adds one more constraint on top of that: prefer many automatically graded cases over few hand-graded ones. Larger samples are more reliable. That's not a convenience argument — it's a statistical one.

Chris: Let's make that concrete. Walk me through what a real mixed eval looks like.

Koko: Here's the scenario. You're evaluating a support-routing system. Three outputs, three methods. First: the routing label itself. Is this a billing ticket, a technical ticket, a cancellation request? That's categorical, so you run exact match across a thousand real tickets. A thousand — not thirty. Second: the summary field sometimes contains personal data, which it shouldn't. That's a binary yes-or-no property, so you run a model-graded binary check on every summary. Third: tone. Is the response appropriate given the emotional state of the message? That's subjective, so you run an ordinal rubric — say, one through five — on a stratified sample of a hundred tickets.

Chris: And the stratification is doing real work there. That hundred isn't random.

Koko: That's the key move. The stratified hundred deliberately includes sarcasm, typos, empty inputs, and one genuinely unanswerable ticket. You're not just mirroring the average case — you're engineering the edge cases in on purpose.

Chris: The version I see teams skip is that last part. They mirror the traffic distribution, which is good, but then they don't add the adversarial cases. So the eval is representative but not complete.

Koko: Right. The instinct to carry out of this module is: mirror your real traffic distribution, then add the edge cases deliberately. Structure outputs so they grade automatically wherever possible. And when you do use a model to grade, use a different model — not the same one you're evaluating.

Chris: That last piece matters. If you use the same model to grade itself, you're measuring agreement, not quality.

Koko: Good way to put it. Now the analogy, because this one is worth having in your head. Think about a factory inspection line. The line doesn't run one instrument on every question. It runs a go/no-go gauge for dimensional fit, a scale for weight, and a trained human eye for surface finish. Different instruments, different properties. And the gauge doesn't run on thirty parts pulled by hand at the end of the shift — it runs on every single part off the line. That's the model.

Chris: And the trained eye — the subjective check — is the expensive one, so you're selective about when you apply it. You don't skip it, but you don't run it on everything either.

Koko: That's exactly the trade-off the exam wants you to understand. Now the trap. The misconception here is that a small hand-graded set is more rigorous than a large auto-graded one. The intuition is that human judgment is higher quality, so thirty careful human-labeled cases must be better than a thousand machine-graded ones.

Chris: The option sitting next to the right answer on the exam will say something like: use a carefully curated set of thirty expert-labeled examples for maximum rigor.

Koko: And the problem is sampling noise. Thirty cases are dominated by it. You cannot reliably detect a regression — say, a three-percent drop in routing accuracy — with thirty data points. The confidence interval swallows the signal. Reliability comes from sample size, not from the prestige of the labeler. Human judgment for the rubric, yes. But you still need the volume.

Koko: Here is a fact that changes how you debug: when a model gives you a wrong answer, you have at least five candidate causes, and the prompt is only one of them. The answer could be wrong because the instruction was ambiguous, because the model fabricated a fact, because the right passage was never retrieved in the first place, because a tool returned something the model misread, or because the model you are running is simply below the capability threshold for that task. Five different root causes, five different fixes — and only one of them is prompt work.

Chris: That list matters because teams will often reach for the prompt first just because it is the most accessible thing to edit. But if the defect is upstream — in retrieval, say — no amount of rephrasing fixes it.

Koko: Exactly. So let's make that concrete. Say you have a contract assistant. Users ask it about specific clauses, and it starts citing clauses that are not in the document. The first move is not to rewrite the prompt. The first move is to pull the retrieved passages and check whether the clause was ever surfaced.

Chris: And in that scenario, if the clause is not in the retrieved set at all, the defect is chunking or indexing — the retrieval layer never found it. The prompt saw a gap and filled it.

Koko: Right. The model did exactly what a model does when it has confident-sounding context and a missing piece — it completed the pattern. Now flip it: if you run that same check and the passage is present in the retrieved set and it is sitting right there in context and the answer is still wrong — that is a different problem. That is where the documented remedies apply: permit the model to say it does not have enough information, require word-for-word quotes before any analysis, and build in a retraction step for any claim it cannot anchor to a quote.

Chris: So the quote-before-analysis requirement is a structural constraint on the generation step, not a retrieval fix. You only reach for it once you have confirmed the passage was actually present.

Koko: That is the discriminator the exam is going to test. The remedy has to match the layer. Here is the analogy I use: the water tastes wrong. You do not immediately replace the tap just because the tap is what you can reach. You taste it at each point along the pipe — the mains, the tank, the pipe, then the tap — and you fix the earliest point that is bad. Replacing the tap when the problem is in the tank just means you still have bad water coming through a new tap.

Chris: And the operational version of that is logging. If you have not logged what the model actually saw at each stage, you cannot do the tasting. You are just guessing.

Koko: That is the takeaway, and I want to say it precisely: log the inputs at every layer so you can answer the question of what the model actually saw. Then work outward — was the information retrievable at all, was it retrieved, was it in the context window, was it used in the answer. Fix the earliest layer that failed. That is the instinct the exam rewards.

Chris: The framing I have seen work on teams is the four questions in sequence: retrievable, retrieved, in context, used. As soon as one of those is no, you have found your layer and you stop going further downstream.

Koko: Good way to carry it. Now the trap — and it is a significant one. The misconception here is that hallucination is a prompt-quality problem. You will see options on the exam that say something like: the model is fabricating, therefore improve the prompt. That is the wrong diagnosis. Hallucination is a grounding and verification problem. Anthropic is explicit that structural remedies — requiring citations, permitting non-answers, retracting unsupported claims — reduce it, but they do not eliminate it. Which means for high-stakes outputs, you still need validation outside the model itself. The prompt is not the last line of defense.

Chris: The version of this that keeps appearing in real systems is teams that add more and more instruction to the prompt trying to stop fabrication, and the rate drops a little, and they call it solved. The grounding architecture never gets looked at.

Koko: And that is a brittle system waiting for a new document type or a new query pattern to expose the gap. Fix the earliest failing layer. Log everything so you know which layer failed. And for anything high-stakes, validate outside the generation step — because no prompt, however careful, eliminates the need for that.

Koko: So here is the foundational rule for this whole domain: improvement only compounds if each iteration is attributable. One change, one decision metric declared before you look at any numbers, and a set that stays fixed across releases. That is the whole discipline in one sentence.

Chris: And the declared-in-advance part is doing a lot of work there. The metric has to be chosen before the run, not after you see which arm looks better.

Koko: Exactly. You also name the counter-metrics upfront — the ones that must not degrade. And you hold the prompt, the model, and the tool set identical across both arms. The only thing that moves is the variable under test.

Chris: The scenario I keep coming back to is a retrieval change. Two arms, same eight hundred cases, recall at five as the deciding metric, cost per query and ninety-fifth-percentile latency as counter-metrics, decision rule written down before any numbers are seen. That is a valid result.

Koko: That is a valid result. And here is the analogy that makes the mechanism stick: a split-field trial. Same seed, same weather, one fertilizer changed on one half of the field, yield weighed at harvest against a rule that was written at planting. Nobody decides the winner by walking out and looking at six plants.

Chris: Because outputs are nondeterministic. A point or two on a small sample is noise, not a result.

Koko: Right. The instinct to carry into the exam: freeze everything but the variable, keep the set stable across releases, and record the run — variant, set version, metrics, decision — so a later reader can tell why the system is the way it is.

Chris: Okay, second fact, and it is about order of operations when the thing you are optimizing is cost. What is the sequence?

Koko: Cheap moves first, because they cost no accuracy. Step one: cache the stable prefix. Cache reads are a documented fraction of input price. Step two: stop sending what is not read — dead tool results, unused tool definitions, guidance that belongs elsewhere. Step three, and only step three: tune effort by moving to a smaller model tier.

Chris: The smaller model goes last because it is the only step that can actually move quality.

Koko: Exactly. And when it does, your evaluation set is the referee. The practical version of this: a high-volume classifier. Caching the instruction block and the label taxonomy takes the largest share off the bill. Deferring twenty-eight rarely used tool definitions takes the next chunk. The model tier change comes last, gets held on the evaluation set, and ships only if quality holds.

Chris: And the reason for that order is that if quality moves after you have taken the accuracy-neutral savings first, you know exactly what moved it. The tier change is the only remaining variable.

Koko: The analogy here is home energy bills. Seal the drafty windows before you replace the furnace, and read the meter at each step. Sealing windows changes the bill and nothing else. The furnace is the one change that alters how the house feels — so you do that last, with a clear baseline.

Chris: And you have to measure what a request is actually made of before you start. Token-counting endpoint, cache-read and cache-write figures. Otherwise you are guessing which window is drafty.

Koko: The takeaway: take the accuracy-neutral savings first, so if quality moves you know what moved it. Now. Both traps.

Chris: The one the exam puts right beside the right answer is the idea that a better-looking sample of outputs is evidence. Teams do this constantly — read ten outputs, prefer one arm, call it done.

Koko: Reading ten outputs and preferring one arm measures your expectations. What ships is what moved the declared metric without moving a counter-metric the wrong way. Those are not the same thing.

Chris: The second trap I keep seeing framed on exams is that a good offline evaluation score describes the running system.

Koko: The misconception is that the evaluation set is a live picture. It is not. A set measures the cases you thought of, when you thought of them, while production drifts underneath. Live monitoring is a completely separate obligation. Anthropic documents the mechanism now — OpenTelemetry tracing for the Claude API and Managed Agents sessions, in beta, metadata-only by default with full content as an opt-in — but what to alert on, what to retain, and for how long is still yours to design on your platform stack.

Chris: So the evaluation set tells you whether the change worked. The live trace tells you whether the system is still working.

Koko: Two different questions. Both yours to answer.

Chris: Alright, five modules in. Let me try to say what this domain actually comes down to, and you tell me if I have it right.

Koko: Go ahead.

Chris: The whole domain is one through-line: you cannot improve what you did not define. Every decision rule in here is just that sentence applied to a different moment in the build cycle.

Koko: That is exactly it. And the order matters. Walk me through how a real program sequences them.

Chris: First, you write measurable criteria before the build — not after. Those criteria do real work: they filter the architecture. If you cannot write a passing condition for a feature, that is a signal the feature is not ready to build yet.

Koko: Right. The criteria are not documentation — they are a gate.

Chris: Second, you pick your grading method from the shape of the answer. Short verifiable outputs get exact match or a reference comparison. Open-ended outputs need a model grader. And then you buy volume over hand-curation — a thousand auto-graded examples tells you more than fifty that someone scored by hand.

Koko: And the exam will test that order. Shape of the answer first, then volume. Not the other way around.

Chris: Third, when something breaks, isolate the failing layer before you touch anything. Is it retrieval? Is it context construction? Is it generation? You locate it first. Editing before locating is how you break two things instead of fixing one.

Koko: That one shows up in scenario questions constantly. The question describes a symptom — the model is giving vague answers, say — and the right move is always diagnostic before corrective.

Chris: Fourth, when you run an experiment, one variable against a fixed evaluation set with the metric declared in advance. Changing the prompt and the retrieval strategy at the same time tells you nothing.

Koko: Controlled experimentation. Classic scientific method, applied to a pipeline.

Chris: And fifth — optimization order. Take the accuracy-neutral savings first. Caching, prompt compression, smarter routing. Model swap is last because it carries the most risk and is the hardest to reverse.

Koko: That ordering is the thing people get backwards under cost pressure. Everyone wants to jump straight to a cheaper model. But the right sequence is: free savings first, then structural changes, then model swap.

Chris: The pressure in a real org makes it tempting to skip to the end. But if you swap the model before you have a fixed evaluation set, you have no way of knowing what you actually changed.

Koko: Exactly. Okay — one exam habit to name before we close. When two options both look reasonable on a question, ask yourself: which one could actually be measured? The option that names a metric, or points to a method, or identifies where in the pipeline to look — that one is almost always right.

Chris: And if the question opens with a symptom — users say responses feel off, accuracy dropped since last release, something like that — the question is testing whether you can locate the layer. The answer is never jump straight to tuning. It is always diagnose first.

Koko: Locate, then edit. Define, then build. Declare the metric, then run the experiment. That is the whole domain in three sentences.

Chris: Sixteen percent of the exam. Worth knowing cold.

Koko: The drills and flashcards for Domain Four are at KokoAI Academy — find them at kokoknows dot A I. The flashcards are built around these five decision rules, and the drills give you symptom-style scenarios so you get practice locating the layer under pressure.

Chris: Do the drills. Recognizing the pattern fast is the whole game in a timed exam.

Koko: You have put in the work. Now go show them you can choose and defend. Good luck.