Tuesday, November 17 · 11 min
Context, retrieval and output contracts — Listen & Learn
Transcript
Koko: Picture a policy reviewer answering a question for tenant A. The synthetic corpus contains an old version of policy P one, a current version of the same policy, and a separate document belonging to tenant B. The reviewer needs a supported answer, not simply a fluent paragraph. Our task is to decide what enters the model context, how evidence is retrieved and cited, and how the application checks the output before anyone relies on it.
Chris: The obvious temptation is to put the entire history and every document into a large context window and hope the model chooses well. But a larger window does not establish which source is current or which user is entitled to see it. It may also crowd out the precise passage and task instruction needed for the answer. How should we turn that into a design decision rather than a vague request to be concise?
Koko: Start with a context budget. Reserve space for the system’s applicable instructions, the current request, authorized evidence, the expected response, and operational margin. Estimate and then measure token use with the tooling for the selected model. The budget is an allocation, not a guarantee of quality. If history competes with current evidence, decide what can be summarized, what must remain explicit, and how the application will disclose a missing piece.
Chris: For example, the case identifier, tenant scope, policy version, and approval state are not merely conversation details. They affect what the system may retrieve and whether a later action can proceed. I would store those facts in an application-owned record with appropriate access controls. A generated summary may help a model understand prior discussion, but it should not become the authoritative record of a human approval or the user’s identity.
Koko: Exactly. A summary can omit a qualifier, blend two versions, or retain a fact after the underlying source changes. Give it a freshness rule and invalidate or regenerate it when the source version or access policy changes. If the available evidence exceeds the budget, select relevant authorized material and mark any unresolved coverage gap. Silently dropping a decisive document while claiming a complete review is an architectural failure, not just a summarization problem.
Chris: Suppose the summary says policy P one version one is current, but the source has moved to version two. Adding more old conversation to the context does not make the summary fresh. The application should detect the source change, invalidate the stale summary, and retrieve the applicable version. The reviewer should see which version supported the recommendation and whether any required evidence remains unavailable.
Koko: There are competing approaches worth testing. Full conversation history may preserve useful detail but consume budget and carry irrelevant or stale content. A short summary can reduce that burden but may lose a critical qualification. Selective retrieval can focus the context on the present question, but it depends on the corpus and query. Compare those approaches on the same representative cases and inspect evidence quality as well as latency and cost.
Chris: And if a policy has a legally important exception, a summary that leaves out the exception could make a response appear more certain than the source permits. The application should preserve a path back to the original passage. It should also define when to abstain or escalate if the available authorized evidence does not support an answer. A confident model response is not a repair for a missing source.
Koko: OpenAI’s retrieval documentation describes semantic search over data using vector stores. That capability helps locate potentially relevant material, including passages that do not share the exact query terms. It does not decide the organization’s authorization policy or guarantee that a retrieved passage is current and sufficient. The architect still defines ingestion, source metadata, tenant scope, versioning, deletion or revocation handling, and how results will be evaluated for the intended workload.
Chris: For the policy case, I would attach tenant, document identifier, version, status, and effective date to each source. The trusted adapter derives tenant A from the authenticated request and applies the relevant access checks before any passage reaches the model. It then selects candidate evidence for the actual question. Policy P one version two may be applicable; the old version and tenant B’s P two require different treatment for different reasons.
Koko: That distinction matters. The old policy might be visible for historical comparison but should not silently be treated as the current rule. Tenant B’s document should not be exposed to tenant A merely because its embedding is similar. The retrieval record can capture the query, authorized scope, source identifier, source version, selected passage, and retrieval time where policy permits. That record gives the later output validator something concrete to check.
Chris: Consider a citation that says “Policy P two” even though P two was never retrieved for this user. It might look like a real title, but it does not establish provenance. The application should flag or reject the unsupported reference and, if appropriate, perform a new authorized search. It should not use a global administrator credential to fetch the document simply to make the citation appear valid.
Koko: We also need to evaluate retrieval independently from the answer. Build labeled cases that include synonyms, ambiguous questions, an obsolete version, no relevant evidence, and a document the user cannot access. Inspect both missed relevant passages and unauthorized exposure. If the answer is wrong, the trace can reveal whether the source was absent from the index, filtered incorrectly, retrieved but ignored, or cited without actually supporting the claim.
Chris: Now turn to the output contract. OpenAI’s structured outputs guidance describes constraining completed, non-refusal model output to a supported JSON schema. The application still needs to handle refusals, incomplete responses, and API errors, and it must validate business rules the schema does not express. Confirm support and schema limits for the chosen API and model. A valid object is easier to parse, but validity of its shape does not prove the contents are true or authorized.
Koko: A compact schema for our case might include a recommendation, a list of evidence references, an uncertainty field, and a proposed next step. The application parses the result and checks that each reference is present in the authorized retrieval record. It also checks whether the cited passage supports the recommendation and whether required evidence is missing. If the model returns an incomplete response, the system needs an explicit incomplete state rather than filling in a decision.
Chris: A field named approved is especially dangerous if downstream code treats it as an actual human approval. The model can propose an action, but the application must verify a separate approval bound to the exact action and current evidence version. Similarly, a syntactically valid document identifier does not show that the principal can access that document. Shape validation, semantic validation, evidence lineage, and authorization are distinct checks.
Koko: In the lab, begin by writing the budget categories rather than inventing a universal token limit. Identify what the application must retain outside the model context, then define which evidence goes into the current request. Mark P one version one as superseded, P one version two as current for the synthetic question, and P two as belonging to tenant B. The authorized retrieval path should not expose P two to tenant A.
Chris: Next create the retrieval record and output contract. Record the applicable source identifier, version, and selected passage for P one version two. Return a recommendation with references and uncertainty. Validate those references against the actual permitted result set. If the model cites a nonexistent or unauthorized source, reject or flag the answer. If P one version two cannot be found, report insufficient evidence and route the question for review rather than pretending the old version is current.
Koko: Now test the stale-summary case. The cached summary still describes P one version one, while the source registry says version two is current. The system should invalidate the summary and retrieve current authorized evidence. Then test an invented citation: provide a schema-valid answer with an identifier absent from the retrieval record. The parser may accept the object, but the semantic validator should not treat the reference as verified.
Chris: The changed-condition challenge is a permission revocation after indexing. The document’s presence in a vector store is not permanent authorization to expose it. The system needs a defined way to propagate the revocation or enforce current access before returning content. Test the timing and failure behavior: if the access service is unavailable, the application should follow a documented safe path rather than assume that an old index entry grants access.
Koko: How would we explain an error to the reviewer without hiding the useful distinction? We can say that the requested answer lacks sufficient authorized evidence, identify which source or version could not be verified where appropriate, and offer a review path. We should avoid presenting a fabricated citation or a generic success message. The reviewer needs to know whether the gap comes from missing content, access denial, stale metadata, or an incomplete model response.
Chris: The recap is to manage context as a deliberate budget with durable application-owned facts, retrieve only evidence the principal may access, and preserve source lineage and freshness. Then treat structured output as an integration contract, not proof of truth or consent. The synthetic policy example shows how to inspect each boundary. Actual answer quality, performance, and the effectiveness of a retrieval index still require measured evaluation on the intended workload.