Skip to main content
    All shows

    Tuesday, November 17 · 11 min

    Reliability, evaluation and resource tradeoffs — Listen & Learn

    0:00-:--
    Speed

    Transcript

    Koko: Our final Foundations scenario is an evidence review that pauses for a human approval and then asks a publication stub to record an action. The stub may accept the action and time out before returning its response. The workflow also faces missing evidence, conflicting policy, a tenant mismatch, an expired approval, and duplicate delivery. We will use those synthetic cases to design recovery and evaluation, not to claim results from a live provider.

    Chris: The timeout is the part that makes me nervous. If I see no response, I might assume the action failed and send it again. But the first action could already be recorded. And if the process restarts while the human is reviewing the packet, a model conversation alone may not contain the authoritative approval state. We need to know what the application stores and who is responsible for resolving uncertainty.

    Koko: Start with a failure model. List each important step, the failure that can occur, who detects it, which state survives, and the safe next transition. Keep a durable request identifier and a stable identifier for a consequential action. Store the evidence version, proposed parameters, approval status, deadline, and execution outcome in application-owned workflow state when reliable resumption requires it. A temporary model message is not a substitute for that record.

    Chris: Suppose the process stops just before the reviewer responds. On restart, the application loads the pending case and its exact evidence version. It checks the reviewer’s current authority and the approval record rather than relying on a browser flag or reconstructing the decision from a model summary. If the approval expired while the workflow was paused, the workflow requests a fresh decision instead of treating a once-valid approval as permanent.

    Koko: Recovery must also account for changed inputs. A reviewer may have approved packet version three, but version four arrives during the pause. The system should identify the mismatch and request a new review of the changed material. The same principle applies to a revoked permission or altered action parameters. Resuming a workflow is not merely restarting computation; it is rechecking the assumptions that made the next step authorized.

    Chris: Now the ambiguous write. The application sends the approved publication request with a stable action key. The stub accepts it but the response times out. The next step is to query the action service using the same key and compare the stored action parameters. If it reports a completed action, return the existing outcome. If the outcome cannot be determined, keep that uncertainty visible and use the documented reconciliation or escalation process.

    Koko: A fresh key on every retry is the wrong design because the downstream service may see each attempt as a separate publication. A bounded retry can be appropriate for a transient read when its contract permits it, but a write needs an idempotency guarantee and reconciliation. Cancellation also has limits: prevent new side effects, yet still reconcile an action already sent. Do not claim that canceling a request automatically reversed a committed external effect.

    Chris: I would turn those ideas into a state table: received, collecting evidence, awaiting approval, approved, execution pending, completed, denied, expired, and uncertain. For each transition, I would record the actor, conditions, stored identifiers, and failure response. I would test process restart, duplicate delivery, stale approval, a changed evidence version, and downstream unavailability. The exact states can vary, but the ownership and safe recovery cannot remain implicit.

    Koko: Evaluation then asks whether the system behaves as intended across representative cases. Separate deterministic control tests from judgments about answer or evidence quality. A tenant mismatch must not expose another tenant’s record. A denied or expired approval must not create an action. Replaying an action key must not create an additional effect. Those properties need direct checks; a high average on recommendation quality cannot compensate for an unauthorized publication.

    Chris: OpenAI’s evaluation guidance describes evaluations as structured tests for variable model behavior. Build a versioned dataset and define the decision each test supports. The legacy Evals platform documentation says it became read-only for existing users on October thirty-first, twenty twenty-six, and is scheduled to shut down on November thirtieth, twenty twenty-six. Evaluation as a discipline remains essential. For this lab, use an offline harness and check current runtime guidance before choosing any live integration.

    Koko: What should the dataset contain? For this workflow, I would include ordinary supported recommendations, no-evidence cases, conflicting versions, tenant mismatches, expired approvals, duplicate requests, and the accepted-but-timeout case. I would record the dataset version, the expected control outcomes, and who supplied the labels. The difficult cases should not disappear into a single mean score, because each failure slice tells us something different about the architecture.

    Chris: For nuanced evidence quality, use expert review or a grader whose judgments have been checked against independent human examples. Measure disagreement and examine where the grader is systematically too lenient or too strict. A grader’s agreement with itself is not calibration. Keep an appropriately protected holdout for later testing, and identify who can change labels or add cases. A model-generated grade does not independently authorize a business action or issue an award.

    Koko: Imagine a report with ninety-nine good answers and one cross-tenant disclosure. The average may look strong, but the disclosure is a critical failure of confidentiality. The release owner should investigate the boundary and require the relevant control tests to pass before proceeding. We should preserve the failed case, trace its cause, and add a regression test. Removing it from the dataset or averaging it away would make the report look better while leaving the risk unresolved.

    Chris: Also be precise about what a passing suite shows. If zero unauthorized actions occur in a defined collection of tests, that is useful evidence about those cases. It is not proof that all possible inputs are safe. Report the dataset, environment, sample, observed results, limitations, and unresolved risks. Monitoring and periodic re-evaluation are still needed as the workload, tool contracts, data, and models change.

    Koko: The next decision is resource tradeoffs. A team may say that several specialist agents are cheaper because each has a narrow role. That could be true for a particular workload, but it is not a measured fact until we compare it with a simpler baseline. We need to include the total number of calls, retrieved context, handoff overhead, retries, human intervention, and the cost of handling failures.

    Chris: Measure quality, latency, and cost together on the same versioned cases. Look beyond a mean: slow tail cases and high-risk slices can matter more than an attractive average. If request volume or current pricing is unknown, label the assumption and show how the estimate changes across plausible values. Do not present a hypothetical number as an observed production metric. A decision record should name the owner, tradeoffs, and the evidence that could change the recommendation.

    Koko: For the lab, first draw the workflow and populate a durable record with case ID, evidence version, action key, approval status, and execution status. Next trace a normal approved publication. Then change one condition at a time: the approval is denied, it expires, the evidence changes, a request arrives twice, or the stub accepts a write and times out. State the expected action count and the recovery owner for every case.

    Chris: A defensible worked answer has zero effects for denied, expired, or materially changed approvals. Repeated delivery with the same action key returns the recorded result and produces no additional effect. After an accepted-but-timeout response, reconciliation finds the existing outcome before a retry is considered. If a service cannot answer the reconciliation query, the workflow remains uncertain and escalates according to policy. It must not guess success or manufacture a failure receipt.

    Koko: Next, write an evaluation plan for the same fixture set. Run the deterministic control suite separately from a human review of whether recommendations are supported by the cited evidence. Report each failed slice, not just an overall score. The synthetic fixtures can demonstrate that the local control logic behaves as designed, but they do not measure a live model’s quality, real provider latency, or production cost. Those require separate observed runs and appropriate authorization.

    Chris: Now change a requirement: the policy corpus is updated while a case is waiting for approval. The design should invalidate stale evidence, identify the affected pending cases, and require renewed review when the change is material to the proposed action. Re-run the relevant retrieval, approval, and regression cases. A model change, tool update, or authorization policy change can also invalidate earlier assumptions, so the decision record needs explicit reevaluation and rollback triggers.

    Koko: One last reviewer question: who can pause or roll back the workflow when a critical control fails? The answer should be a named operational owner and a documented mechanism, not the model itself. The handoff should identify the signal, the decision threshold, the retained audit evidence, and how the team reconciles outstanding actions. If the system cannot establish an outcome, it should preserve that uncertainty instead of generating a reassuring but unsupported status.

    Chris: The recap is to design state and recovery before the first timeout, use durable identifiers and action reconciliation, and revalidate approvals after a pause. Build evaluation around representative cases, calibrated judgment, and separate critical control gates. Compare architectures using measured quality, latency, and total cost, with assumptions and change triggers visible. The result is a decision that another engineer and an accountable owner can inspect, challenge, and revisit.