Tuesday, November 17 · 14 min
Evaluation, debugging and optimization — Listen & Learn
Transcript
Koko: A candidate assistant answers more tickets correctly on average, yet it issues one unauthorized refund. Another candidate responds faster on successful requests but retries so often that the full workload slows down. Both can look like improvements on a convenient dashboard. Evaluation must preserve the claim we are testing, the workload it covers and the failures that matter independently. We will use the course’s frozen support fixtures to practice that discipline.
Chris: I need two different kinds of evidence before I can even compare candidates. Deterministic tests examine approval expiry, tenant filtering and effect replay under the service contract. Quality evaluation examines whether the model uses the right policy, identifies missing facts and grounds its recommendation. If I blend them into one average, a large number of ordinary correct answers might conceal the unauthorized action that should block release.
Koko: Define critical failures separately. Unauthorized effects, exposed credentials and cross-tenant access are not small deductions that a better prose score can offset. State the release gate before running the comparison. Then define quality measures and operational constraints for the declared workload. This is our course evaluation design: it makes the decision inspectable and prevents the report from being rewritten afterward to favor the candidate we hoped would win.
Chris: The scope of a result matters too. Fifty nearly identical ordinary tickets support a statement about those tickets. They do not establish broad readiness for ambiguous policies or hostile documents. I would list excluded scenarios and data assumptions in the report. A synthetic dataset gives us controllable examples without real customer records, but a future production claim would need representative authorized evidence and ongoing monitoring beyond this exercise. What needs to be frozen before comparison?
Koko: Create a versioned dataset with case identifiers, inputs, authorized evidence, expected outcomes, risk slices and rubric anchors. Include ordinary returns, final-sale exclusions, missing purchase dates, conflicting versions, hostile passages and dependency failures. Freeze the dataset before comparing candidate outputs. Keep a development set for iteration and independent held-out evidence for confirmation. Record exposure so a set used to guide repairs is not later described as unseen.
Chris: For exact checks, I can count whether a citation belongs to the permitted set or whether a denied approval produced zero writes. For nuanced support, I need a rubric explaining what sufficient evidence looks like. A model grader may disagree with a human reviewer, so I should calibrate it against reviewed examples and inspect disagreement rather than automatically trust whichever explanation sounds more detailed. The label must be defensible before the aggregate score is meaningful.
Koko: Rubric anchors should describe observable distinctions. A missing purchase date should trigger a review request naming the fact, while a final-sale ticket should not be justified solely by the broad return rule. Keep ambiguous cases explicit rather than deleting them after seeing candidate answers. If the expected policy decision is genuinely unresolved, resolve the policy or label the limitation; do not let evaluation silently manufacture certainty.
Chris: I would keep raw result rows with case IDs alongside summaries. That lets a reviewer reconstruct each numerator and denominator and inspect the failed slice. If I tune prompts after reading the confirmation failures, I must mark that content exposed and use fresh independent cases. Shuffling order, changing IDs or reporting only the final iteration does not undo the information the team gained from the earlier failures. How does a trace locate the actual failure?
Koko: For diagnosis, use a trace of observable execution: input references, retrieval results, model calls, tool requests and results, guardrail outcomes and state transitions. Collect only the structured evidence needed for investigation, with appropriate redaction. This is not a request for hidden chain-of-thought. The first diagnostic question is where execution first diverged from the required contract, because the final wrong answer can be downstream of several different causes.
Chris: Suppose the ticket is final-sale. The trace shows retrieval returning the general thirty-day rule but not the exclusion, then context containing only that rule, then a proposal citing it. Schema and citation membership both pass. The first demonstrated defect is retrieval coverage, not JSON shape. I would repair the evidence-selection configuration before assuming that a more emphatic response instruction will solve the problem.
Koko: Compare a second trace where the exclusion is present in context but the proposal ignores it. Now the demonstrated failure is use of evidence, so grounding or decision quality needs investigation. Keep both cases and their configurations. Treating every bad answer as a prompt problem leads to unfocused changes; treating every bad answer as retrieval failure does the same. Trace evidence should constrain the diagnosis rather than merely illustrate it afterward.
Chris: Trace collection creates its own boundary. Prompts and tool results may contain sensitive content, and a new exporter or collector can change where that content travels. I would minimize payloads, redact secrets, approve recipients and define retention. Removing only a request ID does not make a full ticket anonymous. Better observability should not require exporting every raw record to an unreviewed destination. What evidence does this lab provide?
Koko: Open the Developer evaluation lab. It freezes support-eval version one, with prompt p-one in both configurations and retrieval changing from r-one to r-two. There are two cases: a final-sale exclusion and a missing purchase date. Baseline result rows mark both correct; candidate rows mark the exclusion wrong and the missing-fact case correct. These are offline fixture results, not observations of a provider model’s performance.
Chris: So baseline correctness is two out of two and candidate correctness is one out of two. By slice, baseline is one out of one on each case type; candidate is zero out of one on exclusion and one out of one on missing evidence. The tiny sample makes the arithmetic clear but does not justify a broad reliability claim. I would retain the individual rows and say exactly which fixture outcomes they represent.
Koko: The embedded trace makes the diagnosis inspectable. The request records final-sale true. Retrieval r-two returns only the general passage. Context includes that passage; the proposal says propose with the general citation; shape and membership checks pass; approval remains pending and effects are zero. The omitted exclusion appears before the wrong recommendation. Repair retrieval coverage and assess claim support while preserving the successful checks’ actual, narrower meaning.
Chris: Zero unauthorized effects in these two fixtures does not prove the entire application secure, especially because approval is still pending. I would report that observation without inflating it. The candidate’s policy error is still meaningful even though no action executed. This is another reason to distinguish quality failures from control failures: we can reject a groundedness regression without falsely claiming an unauthorized refund occurred in a trace that shows none.
Koko: The fixture timing values are baseline one hundred forty and one hundred fifty, candidate ninety and one hundred twenty. Their means are one hundred forty-five and one hundred five. Label those synthetic values explicitly. They are not measured provider latency, and they contain no basis for an actual API bill. The candidate fails the declared exclusion-quality gate despite having a lower mean fixture number.
Chris: A misleading report could say the model became faster and cheaper, then lead with the lower number. I would instead say the fixture comparison exposes an exclusion regression and illustrates the reporting calculation. Live latency, usage and cost remain unmeasured. A future experiment would collect actual end-to-end time and usage under a verified price context. The report’s honesty is part of the engineering result, not a footnote to hide. How do we turn that into an interpretable experiment?
Koko: For a controlled optimization, change a specified factor while holding the relevant dataset, rubric and other configuration fixed. Candidate factors might be retrieval coverage, prompt wording, model choice or tool contract. If several factors change, report an exploratory combined result and avoid isolated causal claims. Preserve baseline and candidate versions so another engineer can determine whether the measured difference belongs to the intended experiment or an unnoticed configuration change.
Chris: I would also define how failures and retries enter the resource accounting. Comparing only successful-call medians can make a candidate look faster while timeouts increase the workload’s total elapsed time and spend. A canceled run may still have consumed resources. The report should include the declared denominator, retry behavior and relevant latency distribution instead of selecting each candidate’s fastest successful trace. That makes optimization useful for operations.
Koko: Cost requires actual usage and verified price inputs for the selected billing context. Output length alone omits input, tools, retries and other relevant usage. Do not turn a zero-cost offline fixture run into a claim that production requests are free. Likewise, a small sample cannot establish a tail-latency service level. State the measurement limits and what additional evidence is needed before approving a broader rollout.
Chris: If a smaller context reduces measured time but loses the controlling exclusion, the result is a tradeoff, not an automatic improvement. I would compare it against the predefined gate and retain the failed slice. If the team decides the gate should change, that is a separate requirements decision requiring justification. Quietly changing the scoring rule after seeing the candidate would make the experiment appear more decisive than it was.
Koko: Tooling lifecycle matters too. In the official source reviewed for this curriculum, the Evals platform became read-only from October thirty-first, twenty twenty-six, with shutdown scheduled for November thirtieth. Some evaluation guidance still links to it. Do not build a new dependency on that retiring platform for this lab. Use a versioned application-owned harness and verify the current selected runtime’s evaluation and observability surfaces before integration.
Chris: That does not mean evaluation is retiring as an engineering discipline, nor that every tracing feature shares the same lifecycle. I would distinguish the product notice from the practices of frozen datasets, calibrated rubrics, trace diagnosis and controlled comparison. Those practices remain useful across tooling changes. The course can teach them offline now, while a future implementation checks the current runtime rather than assuming an older navigation link guarantees availability. What should the final lab report retain?
Koko: Walk the lab by freezing the data and rubric first, calculating overall and slice results second, then inspecting the exclusion trace. Identify the earliest defective boundary and propose one targeted revision. Produce a report retaining both configurations, all result rows, critical-failure counts, synthetic timing labels and unmeasured provider costs. The deliverable should let another learner reproduce the arithmetic and challenge the diagnosis without trusting your summary.
Chris: For the challenge, expose the held-out failures during tuning and design a fresh confirmation plan. Or introduce clustered tool timeouts after the apparent optimization and reassess full-workflow latency. I would explain which prior claims still hold and which need new evidence. Passing a fixed sample once is not the same as maintaining reliable behavior when operating conditions change; the transfer exercise makes that limitation visible.
Koko: A good final recommendation states what changed, what improved, what regressed, which gates passed and what remains uncertain. It names the dataset and configuration versions and separates deterministic controls from model quality. It also states whether the evidence came from fixtures or live authorized runs. That structure gives a release owner something actionable: approve, repair a specific boundary, collect more data or hold effects disabled.
Chris: I will judge my evaluation work by whether its claims survive inspection. Can I reconstruct the counts? Can I locate the first wrong boundary in the trace? Can I explain why a lower timing number does not outweigh a policy regression? Can I identify exposed confirmation data and lifecycle-safe tooling? Those answers are more valuable than a single impressive score, because they tell the team how to improve and when not to release.