Tuesday, November 17 · 14 min
Application requirements and OpenAI platform selection — Listen & Learn
Transcript
Koko: A customer asks for a refund, and the assistant produces an excellent explanation. Before we call that application successful, ask what actually happened. Did it find the applicable policy? Did it propose the right outcome? Did a supervisor approve that exact proposal? Did the action service write one refund or two? Those are different engineering questions. Today we will turn them into requirements before choosing an OpenAI runtime.
Chris: I can imagine a product review where everyone likes the answer and nobody inspects the refund ledger. The demonstration looks complete because the conversation ends naturally. But the user outcome includes an external effect, so I need evidence beyond the transcript. For our synthetic retailer, I would start with one ordinary return, one missing-policy case, and one customer whose approval expires while the workflow is paused.
Koko: Good. Write each requirement as a trigger, an expected state, and an observable effect. Given an expired approval, the workflow returns approval-required and creates zero action-service writes. Given missing eligibility evidence, it requests review rather than inventing policy. Given a valid approved proposal replayed twice, the synthetic action adapter returns the same receipt and records one total effect. None of these criteria depends on how confident the answer sounds.
Chris: That wording also tells me where to look for proof. I inspect the workflow state and the authoritative effect count instead of searching for reassuring phrases. A requirement such as reliable support is too broad to fail clearly. If a test fails, I want to know whether the model misunderstood policy, the application accepted invalid approval, or the action adapter duplicated an already committed resolution. How should we organize those different kinds of proof?
Koko: Keep three evidence categories separate. Quality asks whether the recommendation uses the right facts and policy. Controls ask whether scope, approval and operation uniqueness hold under failure. Operations asks whether the declared workload stays within its time and resource constraints. A high quality average cannot compensate for an unauthorized refund. Likewise, perfect authorization tests do not prove that the model gives useful advice. Each category needs its own acceptance evidence.
Chris: For the retailer, quality might include citing the final-sale exclusion and asking for a missing purchase date. Control tests would include another tenant’s ticket, expired approval and duplicate resumption. Operational tests would include a slow dependency and an overall deadline. I would avoid writing a latency target as though we measured it already. We can specify the intended constraint now and gather actual measurements in an authorized experiment later.
Koko: Exactly. Distinguish a desired threshold from an observed result. In this lesson our fixtures establish only the synthetic behavior we define and inspect. They do not establish provider accuracy, latency or cost. The requirements table should name the actor, input conditions, expected output or state, effect boundary, and evidence artifact. That structure makes review practical because a colleague can challenge a claim without having to interpret a polished demonstration.
Chris: There is a tempting shortcut here: put every requirement into a long system instruction and call the resulting prompt the application contract. That may influence behavior, but it leaves the action boundary unenforced. The model could still propose an unauthorized operation, and a careless adapter could execute it. I would use instructions to describe the task while keeping identity, approval and effect checks in application code. Which runtime ownership differences matter before choosing?
Koko: Now compare the runtime choices by ownership. Direct Responses integration gives the application direct control of its model integration and any loop it builds. The Agents SDK runs in the application and supplies a runner for reusable agents, tools and handoffs. The managed Agents API runs an OpenAI-managed Codex harness and saves progress across tasks. These are different execution choices, so begin with where the loop runs and who maintains it.
Chris: If our first version has one read-only policy lookup and one recommendation, a direct Responses integration could be enough. I should not add handoffs just because an SDK supports them. If the workflow genuinely needs application-hosted specialized tools and reusable handoffs, the SDK runner could reduce loop implementation work. For long-running work that benefits from the managed Codex harness, I would investigate the Agents API against our integration requirements.
Koko: State needs the same precision. An SDK session, an Agents API session, a Responses conversation and a sandbox are distinct resources. A sandbox is an execution environment; a conversation provides context; an application business ledger records approvals and effects. Do not draw one box called session and assume it solves every lifecycle. For each resource, identify creation, access, resumption, retention and cleanup ownership. Then identify the authoritative record for business decisions.
Chris: So a managed runtime may save progress without taking ownership of refund authorization. Similarly, an SDK can help manage an agent loop without becoming our supervisor approval database. I would keep a responsibility matrix with separate rows for orchestration, conversation history, execution environment, identity, approval, action deduplication and cleanup. A vague answer such as the platform handles state would be a warning that we have not located the real boundary. What do we persist when approval interrupts execution?
Koko: Walk through the pause. Our application creates proposal three for ticket seventeen, records the evidence versions, and computes a digest of the exact action being proposed. The supervisor’s approval binds to that digest, the permitted scope and an expiry. The workflow then waits. This is our educational application design, not a claim that selecting any particular runtime automatically creates the business approval record or enforces its validity.
Chris: Tomorrow the worker resumes, and a browser request includes the word approved. I must reload the authoritative record, not trust that word or the saved conversation. I check the current actor, tenant, proposal digest and approval expiry. If policy changed overnight, I also check our freshness rule. A remembered approval may have been valid for an earlier proposal, so conversational continuity does not imply permission to execute now.
Koko: Now make the failure experiment harder. The action service commits the resolution, but the connection closes before the worker records the receipt. A second resume arrives. We need a stable logical operation key and an authoritative way to reconcile its outcome. In the synthetic idempotent adapter, replaying that key returns the original receipt. Creating a new key for each attempt would turn a recovery attempt into a second business operation.
Chris: And if the real action service offers neither idempotency nor status lookup, I cannot honestly promise one effect merely because our application remembers submitting. The outcome is uncertain. I would block automatic replay and define a reconciliation process. That limit may affect the runtime decision less than it affects the downstream service contract, which is useful: our experiment is identifying the actual engineering dependency rather than blaming the model for everything.
Koko: That distinction is the point of a falsifiable decision. Write why the chosen runtime fits the workload, what the application still owns, and what evidence would reverse the choice. For example, select the SDK under an assumption that service adapters must run inside our application deployment. If that assumption changes to a long-running managed sandbox task, revisit the decision. Do not simply rename the session and claim the architecture has migrated.
Chris: My rejected alternatives should be workload-specific too. I would not reject Responses by saying it cannot use tools, because that is inaccurate. I might reject a custom loop because the team cannot safely maintain the required handoff and interruption logic. I would not reject managed execution as inherently unsuitable; I would identify an unproven integration or lifecycle requirement and describe the experiment needed to decide whether it can meet that requirement. How do we make that reasoning concrete in the lab?
Koko: For the lab, begin with the small offline starter. Fill in approval owner, state owner and execution owner before implementing any provider adapter. Create fixtures for normal completion, denied approval, expired approval and duplicate resume. Each fixture should name expected final state and total effect count. Keep the action counter in the synthetic service, because checking that a method returned success does not prove how many writes occurred.
Chris: I will write nine acceptance criteria across the three evidence categories and map them to artifacts. For the normal fixture, one valid approved operation should produce one receipt. Denied and expired fixtures should produce zero writes. Duplicate resume should still produce one total effect. I would record the fixture inputs and observations rather than merely asserting that the code looks idempotent. The repository should let another learner repeat those exact cases.
Koko: Next build the three-runtime responsibility matrix. Include resource names rather than generic checkmarks. Mark which component runs the loop, stores conversation context, executes tools and owns the business ledger. Then add an architecture decision record with the selected option, assumptions, rejected alternatives and reversal condition. The lab is successful when a reviewer can trace a requirement through an owner to an experiment, even if the chosen runtime differs from the worked example.
Chris: Suppose I choose the SDK because our scenario assumes in-process adapters and application-owned deployment. My decision is defensible if I explain that assumption and prove the interruption boundary. Another learner could choose Responses for a simpler workflow and pass with equally strong evidence. The rubric should evaluate the fit and ownership reasoning, rather than reward whichever product name happened to appear in the sample answer.
Koko: A common weak submission includes a detailed product comparison but no negative test. Repair it by choosing the boundary most likely to invalidate the design. Pause immediately before submission, expire approval, and resume. Then run a valid approval with duplicate resumption. Inspect the ledger and state transitions. If the model writes an excellent explanation while the service records two effects, the implementation has failed its control requirement regardless of presentation quality.
Chris: I would also test a proposal-version mismatch. Approving proposal two must not authorize proposal three merely because the ticket and refund amount match. The revised proposal could cite different evidence or perform a different action. This teaches transfer beyond the starter: the invariant is approval of an exact permitted proposal, not approval of a conversation topic. The same reasoning applies to booking appointments, reserving inventory or changing a subscription. What changes when the execution environment moves?
Koko: For the challenge, move execution to a managed sandbox and revisit every ownership row. What state remains in our application? Which resources persist between tasks? Who deletes the sandbox and provider session when the task is over? What happens to approval if the task resumes elsewhere? The changed constraint should produce revised contracts and tests. Reusing an unchanged diagram with a different logo does not demonstrate an architecture decision.
Chris: Before defending the choice, I would state the limits of our evidence. The offline fixtures can prove their own expected transitions and effect counts under the synthetic service contract. They do not compare real model quality across runtimes or establish production resource costs. A future live experiment would need explicit authorization, current documentation checks and actual measurements. That makes the decision provisional in an honest, useful way rather than pretending our fixture exercise settled every question.
Koko: Carry three questions into implementation. What observable contract are we trying to satisfy? Which runtime owns the execution mechanics we need? Which responsibilities remain application-owned even after that selection? Answer them with acceptance criteria, a resource lifecycle matrix and a failure experiment. Those artifacts are the bridge from an attractive assistant concept to a system whose behavior a colleague can inspect, test and challenge.
Chris: My takeaway is that runtime selection begins with requirements and ends with evidence. I can distinguish recommendation quality from authorized effects, compare Responses, SDK and managed API without merging their resources, and explain what could change my decision. For the lab review, I will bring the requirements table, responsibility matrix, decision record and interruption results. If any ownership answer remains vague, that is the next experiment to design.