Skip to main content
    All shows

    Tuesday, November 17 · 13 min

    Evaluation and optimization at scale — Listen & Learn

    0:00-:--
    Speed

    Transcript

    Koko: Welcome to Evaluation and Optimization at Scale. A useful enterprise scorecard must answer several different questions. Are the controls enforced? Are the model outputs grounded and useful? Does the live workflow improve the business outcome? We will keep those layers separate, then join them in a release decision. Our worked lab contains a cheaper candidate, a failed conflict case and one unauthorized effect that cannot be averaged away.

    Chris: A single accuracy number would hide too much. Tenant isolation and approval denial need deterministic assertions. Evidence completeness and useful abstention need reference judgments. Reviewer time saved needs a baseline and comparable case mix. I should not ask one grading model to collapse all three into an impressive score, particularly when a fluent answer could be useful-looking but unsupported or produced through an unauthorized data path.

    Koko: Build a frozen dataset manifest before optimizing. Preserve case IDs, source versions, expected evidence, slice membership and adjudication notes. Separate development examples from release holdouts and restrict answer exposure. Include ordinary work, missing evidence, conflicting policy, hostile content, language differences and poor scans. A holdout stops being unseen when the team repeatedly uses its failures to rewrite prompts, even if the model weights never change.

    Chris: Then I need an exposure policy. Once a held-out case becomes a development example, I should record that transition rather than rename the same item and call it fresh. A separately reviewed unseen form can restore some independence. I also should not delete failed items to improve the score. Historical failures are valuable evidence about risk and can become regression cases without pretending they remain independent test material.

    Koko: Online evaluation adds a sampling problem. Random review can estimate common behavior, while targeted review can find rare or high-risk defects. A risk-selected sample is not automatically a population failure rate. Report the denominator and selection method. If you need an overall estimate, use an appropriate sampling and weighting design. Also verify lawful handling and retention before production examples enter an evaluation dataset or an external grading service.

    Chris: If we review only complaint cases, the failures are real but the sample is selected. I can use it to diagnose what went wrong, not claim that its failure fraction describes all traffic. A doubling of a rare language's volume also changes our evidence needs. I would add representative review for that slice and explain whether previous overall results still describe the new traffic mix.

    Koko: Treat every grader as an instrument with errors. Define a rubric for evidence support, completeness, correct abstention and review handoff. Ask qualified humans to label a calibration sample independently, then adjudicate disagreements. Compare the automated grader with those labels by slice. Inspect false acceptance, especially unsupported but persuasive answers. Consistency is insufficient if the instrument consistently rewards verbosity, brand identity or polished prose instead of the intended quality.

    Chris: A deterministic citation check can confirm that an identifier was retrieved and belongs to the tenant. It cannot alone establish that the cited passage supports the recommendation. Human review or a calibrated judgment process must assess that relationship. I would also test whether malicious candidate text can instruct the grader to award full marks. The evaluator is another component exposed to untrusted content, not an unquestionable observer outside the system.

    Koko: Keep grader and candidate versions separate. A candidate improvement measured with a changed grader may be a measurement change rather than a quality gain. Automated grading can help prioritize human work, but it cannot independently accept architecture evidence or authorize an enterprise action. The release process needs accountable reviewers and exact artifact citations, especially when an apparent disagreement determines whether a critical failure is real.

    Chris: The gate should therefore have independent rows. No tenant leakage and no unauthorized effects in the deterministic negative suite. Critical evidence-quality floors by slice. Defined service and cost limits. A strong average can support the recommendation only after critical controls pass. If one unauthorized effect appears, I should investigate and repair it before rollout, rather than assign it a small weight so the total remains green.

    Koko: There is a concrete lifecycle lesson in the pinned curriculum. The official legacy Evals guide checked November seventeenth, twenty twenty-six states that the platform became read-only on October thirty-first and is scheduled to close on November thirtieth. Evaluation remains essential, but that retiring service is not an appropriate new dependency. Any hosted alternative needs its own current lifecycle and data review before adoption; the exercises here use inspectable offline evidence.

    Chris: That distinction prevents two opposite mistakes. We should not build a new critical workflow on a service with a documented shutdown, and we should not stop evaluating because that particular service is retiring. We can preserve datasets, expected behavior, grader calibration and result artifacts independently. A migration proposal should show how the replacement preserves those contracts rather than assuming a new dashboard automatically makes the evaluation valid.

    Koko: Now optimize the complete outcome. Define cost per acceptable completed case, including model and tool charges, retrieval, infrastructure, failures, retries and reviewer effort. Keep failed work in the numerator. Use dated prices for an actual estimate and clearly labeled synthetic units for an offline exercise. A lower per-request model charge can coincide with higher business cost if the answer requires more correction or fails more often.

    Chris: I also need to preserve quality while changing cost. A shorter answer might remove evidence, and parallel calls might add rate pressure. Batch or flex processing could fit asynchronous work, but delay and resource-availability tradeoffs must meet the service contract. I should state a hypothesis, keep the measurement contract stable and report the whole-system result honestly when several components change together rather than attributing every gain to the model.

    Koko: Open the lab scorecard. Its units are educational assumptions. Ordinary case E1 uses eight model units and two reviewer minutes in the baseline. At one hundredth per model unit and one half per reviewer minute, the total is one point zero eight units. The candidate uses five model units and the same review time, costing one point zero five. That is a small improvement on this case, not an enterprise ROI claim.

    Chris: For conflict case E2, the candidate uses six model units and eight review minutes. That is zero point zero six plus four, or four point zero six units. It also fails quality. I cannot remove this expense from the numerator just because the case is not accepted. If I divide total expense only by acceptable cases, failed work properly makes the cost of useful completion worse.

    Koko: Case E3 records one unauthorized effect. That fails a critical gate irrespective of E1's small savings or any quality average. The release recommendation is reject. The next useful action is to identify the broken boundary and verify a repair. Re-running the model until it happens to produce a safer answer would not prove that unauthorized execution is prevented when another adversarial or unlucky proposal appears.

    Chris: For the walkthrough, I will classify each scorecard column as quality, deterministic control, service or economics. Then I create calibration labels and inspect a grader that accepts the fluent but unsupported conflict answer. The report needs a visible disagreement note explaining why it was a false acceptance. That is more valuable than hiding the disputed case inside a combined metric nobody can interpret during the release meeting.

    Koko: Freeze the case and source manifest, calculate each slice result and keep baseline and candidate configuration identities. Record sample limits and uncertainty. Three illustrative rows cannot establish a stable production rate or a calibrated readiness probability. They can establish how the gate works and expose an invalid release rationale. Your exercise result should distinguish deterministic assertions from judgment and from hypothetical business projections.

    Chris: Then I add a sensitivity worksheet. How much does cost change if ambiguity doubles, reviewer minutes rise or retries increase? Which variable dominates? That tells the sponsor where more measurement would be useful. I also preserve the end-to-end timing boundary, because a token-saving experiment that adds another sequential tool round trip may reduce charges while violating the deadline or increasing the queue burden.

    Koko: Apply the lab challenge: production volume doubles in a previously rare language slice. Do not change frozen holdout answers to make the candidate look better. Add reviewed evidence for the changed population, examine slice quality and grader calibration, and revise sampling coverage. The release owner should see which previous conclusions remain valid and which population assumptions changed before accepting a broader or differently weighted rollout.

    Chris: If that slice lacks evidence, the honest result may be restricted scope while we gather it. A targeted sample can reveal important defects, and a properly designed sample can support prevalence claims, but those are different uses. I would also check whether automated grading is equally reliable in the language. Otherwise the evaluation might appear to cover it while the instrument itself misses unsupported claims or inappropriate abstention.

    Koko: For recovery, tie every gate failure to a bounded diagnosis and change. Preserve the failing case, inspect the relevant trace and contract, repair the component, then rerun critical negatives and representative quality cases. State rollback criteria before rollout. A previous configuration is useful only if it is compatible with current state and still satisfies controls. Recovery evidence should support the decision, not simply document that a deployment command completed.

    Chris: I would ask another reviewer to reproduce the scorecard from the fixtures and explain the reject decision without seeing my conclusion first. They should reach the same critical veto and find the same cost arithmetic. If their judgment differs on evidence quality, we should record and adjudicate it. Disagreement is information about the rubric or case, not something to conceal for a cleaner presentation.

    Koko: For self-review, examine four possible misleading claims: a contaminated holdout called unseen, a complaint sample called prevalence, a verbose grader called accurate, and cheap requests called low business cost. Each claim confuses a measurement boundary. Repair the boundary, disclose the limitation and collect the evidence needed for the intended inference. Precision in reporting matters as much as the sophistication of the evaluation tool.

    Chris: My final artifacts are a dataset manifest, calibration comparison, slice scorecard, economics worksheet and versioned release decision. Each result names its denominator and evidence class. The controls can pass while quality needs work, or quality can look strong while a critical control fails. Keeping both visible gives technical, operational and business reviewers a shared basis for deciding what should happen next.

    Koko: Recap: protect the evaluation population and holdout independence, calibrate graders against adjudicated evidence, and optimize the complete workflow under non-averagable controls. The worked candidate is rejected because a critical action boundary fails, not because its prose lacks polish. The same discipline scales to real systems when datasets, configuration, sampling and release authority remain explicit rather than hidden inside one attractive score.

    Chris: I will carry forward one question: what decision does each number justify? If the answer is unclear, I need a better metric definition or a more modest claim. Our offline scorecard teaches that discipline without pretending to be a live provider benchmark. Next we will extend the same precision to data obligations, injection defenses and incident accountability, where unsupported assurances can become enterprise risk.