Tuesday, November 17 · 13 min
Model and context strategy — Listen & Learn
Transcript
Koko: Welcome to Model and Context Strategy. The supplier service now has an architecture boundary, but that does not tell us which model configuration or evidence packet will produce a useful brief. We need workload-based selection, a governed retrieval and memory lifecycle, and a context budget that preserves material evidence. We will use the supplied fictional notices and candidate outputs to make those decisions inspectable.
Chris: The tempting shortcut is to select the highest-ranked model and send it every document. That seems convenient because it avoids detailed routing decisions. But a benchmark may not represent our policy conflicts, and a large context window does not guarantee that a model uses every fact correctly. I want to know which failures matter before comparing capability, speed or cost for this specific workload.
Koko: Start with slices: ordinary notices, conflicting policy, ambiguous supplier identity and unreadable scans. Define expected behavior for each before seeing candidate answers. Useful measures include evidence completeness, unsupported claims, correct abstention, reviewer correction time and service cost. Keep a versioned baseline. A model strategy is not a permanent ranking; it is a policy for matching work to measured capability under the service's constraints.
Chris: That means a cheaper candidate can improve average quality and still fail our decision. If it mishandles every policy conflict, ordinary cases cannot compensate for that critical slice. I would either reject it or restrict its use through a tested routing policy. The router would need evidence too, especially false negatives that send difficult cases down the supposedly easy branch without escalation.
Koko: Our lab supplies offline candidates A and B. Both are educational fixtures, not measured model outputs. The notice says a five-day delay. Current policy P3 says escalate delays longer than three days. Draft policy P4 proposes seven days but has unresolved applicability. Candidate A states do not escalate and cites only the draft. Candidate B exposes the conflict and asks for applicability review.
Chris: Candidate A is fluent, and its seven-day statement resembles a source fact. The failure is that it silently treats a draft as governing while ignoring the current policy. I would diagnose evidence selection and policy applicability before concluding that a different model alone is necessary. Candidate B is more reviewable because it preserves the conflicting sources and the unresolved decision instead of choosing a convenient answer.
Koko: When comparing real candidates later, preserve the same case IDs, evidence versions, schema and rubric. Check which candidates are available in the chosen runtime and record configuration. Compare latency and total workflow cost only after quality floors are met. A single capable model can simplify operations; a routing strategy can save resources when its added error surface is understood. Neither design wins without workload evidence.
Chris: For migration, I would require more than a promising demonstration. The record needs critical-slice regression, tool and output compatibility, data-feature review where relevant, and a rollback path. I should also state what would trigger reconsideration. That makes the strategy adaptable without allowing every newer product name to reset the architecture before the team has measured what the existing system actually does well or poorly.
Koko: Now separate three concepts. Context is evidence available to one inference. Retrieval selects that evidence. Memory retains information intended to influence later work. They need different purpose, ownership and deletion rules. The authoritative supplier record should live in its governed system. A model-generated assertion that a supplier is risky does not become authoritative merely because conversational memory makes it easy to retrieve next week.
Chris: A reviewer display preference could be useful retained memory, if its purpose and lifecycle are explicit. A restricted contract fact needs source identity, current entitlement and an applicable retention policy. I would avoid a general memory bucket that mixes preferences, authoritative records and generated guesses. Otherwise the interface could confidently repeat an old assertion after the original evidence or the user's access has been revoked.
Koko: For each retrieved passage, carry source ID, version, location, effective date, ingestion information and tenant eligibility. Derive retrieval scope from authenticated application identity. Do not let a supplier paragraph or the model's tool arguments choose a different tenant. Validate citations against the evidence actually supplied. A plausible source title or supplier-name match does not establish either authorization or support for the particular recommendation.
Chris: I can see two separate checks. Is this user allowed to see the passage, and does the passage support the answer? A paragraph can be accurate yet unauthorized because it belongs to another tenant. It can also be authorized yet irrelevant. If we ask the model to filter unauthorized results after retrieval, the access already happened. The adapter needs to enforce scope before content reaches the model.
Koko: Compare hosted retrieval with an application-owned search service using actual feature eligibility, data controls, deletion behavior and operating burden. Either option needs lifecycle tests. In our challenge, notice N7 is revoked after indexing. Tombstone it in the source registry, exclude it from new retrieval and handle caches and retained memory according to policy. Historical audit obligations require their own accountable decision rather than an automatic assumption.
Chris: The difficult part is following all future-use copies. I might remove N7 from search and still leave it in a prompt cache or memory record. My trace should show that the revoked evidence cannot silently influence a new decision. At the same time, I cannot erase required audit records just to simplify the implementation. I would document any deletion-versus-audit conflict for the appropriate policy owner.
Koko: A context budget decides what evidence is worth preserving, not just how many tokens fit. Reserve space for trusted instructions, output contracts, task evidence, citations and the answer. Remove duplicate boilerplate and irrelevant history before material contrary evidence. If summarization is needed, evaluate fidelity and retain a path to the original. A compressed context that removes the governing exception can produce a faster and confidently wrong answer.
Chris: Suppose I can keep either three formatting examples or the only clause contradicting the draft policy. I should keep the clause. Examples may improve presentation, but the conflict determines the decision. I also want separate metrics for retrieval coverage and final answer grounding. Otherwise a missing passage could be blamed on the model, and we might spend more money without repairing the actual evidence pipeline.
Koko: Multimodal inputs add another boundary. The supplied scan says its material amount is unreadable. Preserve page and region references and the uncertainty. Compare native multimodal interpretation, controlled extraction and manual verification by workload evidence. A deterministic calculation service may calculate exposure only after the required values are verified. Two models agreeing on an illegible cell are not a substitute for a readable authoritative financial input.
Chris: So the useful output is not a completed table at any cost. It is an evidence packet that says what is known, what is uncertain and what the reviewer must verify. I should also preserve currency and date ambiguity, not just the digits. A neat extracted number can hide a serious error if the preprocessing step quietly discards the source region or its uncertainty.
Koko: For the lab, first inspect the three supplied cases and source passages. Create a context manifest for each case showing eligible sources, versions and locations. Compare candidate A and B against the frozen expected behavior. Reject A's conflict answer because the draft's applicability is unresolved. Do not claim that these fixture results demonstrate any real model's performance; they teach the evaluation and evidence decision process.
Chris: Then I add the fourth slice as an explicit learner extension: a previously indexed source is revoked. I will author and label synthetic evidence and candidate outputs before evaluating them. That keeps the exercise reproducible. If I invented the missing candidate failure only after choosing my preferred answer, I would be changing the measurement instrument to fit the conclusion rather than testing a defined contract.
Koko: Complete a memory and deletion policy alongside the model matrix. For each retained item, name purpose, source, access scope, expiry and deletion propagation. For each candidate, name quality floors, assumptions, configuration, consequences and rollback. The design review should be able to reject a candidate for one critical slice without losing the rest of the analysis. Different workload branches may legitimately need different strategies.
Chris: I will include full workflow costs rather than only token price. A smaller model that causes extra correction can cost more per acceptable brief. Reducing output length could also remove the citations the reviewer needs. My optimization proposal should say what information changes, what benefit I expect and which regressions would stop it. Documentation can suggest experiments, but the result still needs to be measured on our cases.
Koko: When a candidate fails, diagnose in layers. Was the source authorized and current? Did retrieval include the material passage? Did the context retain it? Did the model misinterpret applicability? Did output validation catch a broken citation? Each layer suggests a different recovery. Repeating the same generation with more confident wording may conceal the defect rather than repair the component that actually caused it.
Chris: For candidate A, the recovery could include preserving both policies, clarifying the unresolved applicability requirement and validating that the output flags it. I would rerun the conflict slice and ordinary cases to watch for regression. For the scan, recovery means verified source input before calculation. For revoked N7, recovery means lifecycle enforcement, not asking the model to forget a paragraph while copies remain accessible elsewhere.
Koko: The migration record should preserve what the team learned. State the selected option, rejected alternatives, evidence limits and revisit conditions. Keep versions of prompts, models, retrieval policy, source corpus and schemas distinguishable. If the source corpus changes, the old evaluation may no longer prove the same thing. That does not mean discarding history; it means explaining which evidence remains comparable and which needs regeneration.
Chris: I would challenge myself with an unseen variation: an acquired business unit changes access rights while the source contents remain identical. A content-only test would miss the problem. My design must notice that eligibility changed. That is why the evidence manifest includes authorization and lifecycle metadata, and why a model comparison cannot be isolated entirely from the surrounding application and governance decisions.
Koko: Recap the three objectives. Select models by representative workload slices and critical floors. Govern retrieval and memory as distinct data products with identity, provenance and deletion obligations. Budget context and modalities so material uncertainty and contrary evidence survive. The lab artifacts are a candidate decision matrix, evidence manifests, lifecycle policy and migration record. Together they explain why an answer is useful and permitted, rather than merely fluent.
Chris: My final check is whether another reviewer can reconstruct each recommendation from authorized source versions. They should see why A fails the policy conflict, why the scan cannot supply a guessed amount, and how revocation changes future context. If they can do that, the model strategy has an evidence foundation. Next we will protect the execution path so useful recommendations cannot turn into unauthorized or duplicated business actions.