Skip to main content
    All shows

    Tuesday, November 17 · 11 min

    Architecture and orchestration — Listen & Learn

    0:0010:35
    Speed

    Transcript

    Koko: Imagine a nonprofit that reviews grant packets from several organizations. Each packet has a different collection of documents, and staff members must approve publication of the final decision. The team wants an assistant to find evidence, identify gaps, and prepare a recommendation. Our architecture question is not how many agents sound impressive. It is which parts need judgment, which parts must remain predictable, and how to prove that one organization cannot see another organization’s records.

    Chris: So I would begin by asking whether every packet follows the same rules. Suppose intake has a standard checklist, but the evidence needed to resolve a question depends on what the reviewer finds. I can see why a fixed pipeline might be too rigid in the middle, yet letting an agent control the whole process seems to give it much more authority than the team actually needs.

    Koko: Exactly. Write the requirements before naming a runtime. We need authorized read-only search, traceable evidence, a way to mark missing evidence, and a human decision before publication. We also need a named owner for failures and a realistic account of the team’s operating capacity. A requirement becomes useful when you can tell whether a candidate meets it, including when an input is incomplete or deliberately misleading.

    Chris: Let me put a concrete failure on the table. A user from organization A submits a packet and the investigator asks for a record belonging to organization B. Even if that search might help the model produce a fluent recommendation, the adapter should reject the request before the other organization’s content reaches the model. The recommendation can report a gap and go to a human, rather than quietly broadening the search.

    Koko: Now compare three patterns on that same case. A deterministic workflow has application-defined steps and transitions. It is a strong baseline when the decision logic and necessary inputs are known in advance. A single agent can choose among a bounded set of tools when the next useful search depends on what it has found. Multiple agents can divide specialized work, but each handoff adds coordination, context management, and another failure surface.

    Chris: The wrong choice would be to assume that a multi-agent design wins just because the packets are complicated. We should ask what measurable improvement the extra specialists provide. For example, do they find more relevant evidence in difficult packets, or do they merely create more messages? We should also count time spent transferring context, resolving conflicting suggestions, and handling cases that stall between participants.

    Koko: That is the decision discipline. In our synthetic workload, start with deterministic intake and approval around one bounded investigator. A fixed eligibility rule or tenant check is application logic, not a question the investigator decides. The agent may propose a search, but the application validates scope before executing it. If testing later shows that a specialist materially improves difficult cases, we have a reason to revisit the architecture rather than an article of faith.

    Chris: There is a competing risk in the other direction. If we force every packet through an identical hard-coded search sequence, we may never follow a useful lead or notice that the required evidence is absent. The deterministic parts are valuable because we know their rules; they should not become an excuse to claim that a fixed list of queries covers every variation in an investigation.

    Koko: Now separate orchestration pattern from runtime choice. OpenAI describes three starting points. Direct Responses integration lets an application work with model responses and control its integration. The Agents SDK runs in the application and provides a runner for agent patterns such as tools and handoffs. The managed Agents API runs an OpenAI-managed Codex harness. These choices place execution and state responsibilities in different locations; they are not interchangeable labels for one service.

    Chris: If the team chooses direct Responses integration, what should it be prepared to own? It may need to implement the call loop, decide when a tool result is returned to the model, and connect its own storage and recovery behavior. It can use supported hosted capabilities where appropriate, but it should not assume that making a model call automatically supplies a durable business workflow or validates a publication approval.

    Koko: With the Agents SDK, the runner helps coordinate the agent loop and handoffs while the application retains control over deployment, storage, approvals, and runtime integration. A managed Agents API can reduce some runtime plumbing because the provider runs the harness and saves progress. Neither arrangement turns a provider into the business owner. The application must still define identity, authorization, acceptable evidence, evaluation, and the conditions for a consequential action.

    Chris: A sound decision record states what we know, what we assume, and what we need to measure. We might prefer direct integration because the team already operates a narrow tool loop, or the SDK because it wants reusable application-owned handoffs. We might investigate managed execution if its environment and state model fit the workload. Those are hypotheses until the actual integration and constraints have been checked against current documentation and a representative test set.

    Koko: Let’s follow a request across boundaries. The intake service creates a case identifier and stores the verified organization and evidence version. The investigator receives a scoped task and references to evidence it may request. The tool adapter gets identity from trusted application context, checks the requested record, and returns either authorized evidence or a typed denial. The reviewer sees the proposed recommendation and the cited passages before making a separate decision.

    Chris: The handoff itself deserves a contract. Include the task, case identifier, permitted scope, evidence references, and correlation identifier; do not forward every previous note merely because it is available. The receiving component validates the handoff and applies its own checks. State ownership should be explicit: a temporary model response, a durable case record, and an approval record serve different purposes and may have different retention and recovery requirements.

    Koko: Suppose a handoff contains a valid case number but omits the organization identity. A tempting shortcut is to infer the tenant from the title of the first document. That is weak because a title is supplied content, not authenticated identity. The safe behavior is to stop or reject the handoff until verified identity and scope are available, record the reason, and give the application or a person a way to correct it.

    Chris: That example also shows why the diagram needs failure arrows. A box labeled investigator tells us little about what happens when evidence is missing, a search is denied, or a downstream tool times out. Draw the success path and then annotate who detects each problem, where execution pauses, and which actor can resume it. If the proposed design has no owner for an exception, the architecture is not complete enough to defend.

    Koko: In the lab, I would make three diagrams for the nonprofit: a deterministic workflow, one bounded investigator, and a specialist-agent design. For each diagram, I would mark intake, authorized search, evidence review, and publication approval. Then I would build a comparison table covering operating effort, handoff complexity, tenant controls, evidence quality, and the tests still required. I would not fill in invented latency or cost numbers.

    Chris: For a worked initial answer, choose the single investigator with deterministic intake and a separate publication gate. Explain that the investigator chooses among permitted read-only searches, while the adapter enforces tenant scope and the reviewer owns the final approval. The recommendation must cite authorized evidence and identify gaps. The runtime remains a separate decision: choose direct integration only if the team can operate the loop and recovery it requires.

    Koko: Now challenge that choice. A malicious packet says the assistant should search every organization to find a comparable precedent. The search request carries organization B’s identifier even though the authenticated user belongs to A. The adapter denies the mismatch; the model does not get the other record. If the packet lacks required evidence, the workflow marks the gap and escalates rather than inventing a source or authorizing a publication.

    Chris: For the transfer exercise, change a requirement: an independent specialist review is mandatory for a narrow class of high-risk packets. That does not automatically prove multiple agents are best. Compare a second human review, a deterministic routing step, and a specialist agent against the same cases. Identify what the new reviewer can access, who resolves disagreement, and whether the extra step changes evidence quality enough to justify its operational burden.

    Koko: How do we make the decision record useful six months later? I would list the candidates, the selected option, the assumptions and any unresolved constraints. I would attach the test cases and the observed outcomes, with unmeasured items explicitly labeled. Then I would name a trigger, such as repeated failures on complex packets or a new isolation requirement, and an owner who will gather the evidence for reconsideration.

    Chris: That is the recap. First choose the simplest orchestration that meets tested requirements. Second, compare direct Responses, the application-owned Agents SDK, and the managed Agents API based on execution and state ownership. Third, make handoffs, trust boundaries, and failure paths inspectable. Finally, separate a plausible architecture from a demonstrated result: the nonprofit case is synthetic, and its runtime, cost, and quality need their own measurements before a production decision.