Skip to main content
    All shows

    Tuesday, November 17 · 14 min

    API mechanics and application integration — Listen & Learn

    0:00-:--
    Speed

    Transcript

    Koko: Imagine a support page showing half a recommendation when the connection drops. The customer sees text, the server sees an interrupted stream, and the action service may already have received a request. Calling the whole interaction failed is too imprecise. We need separate contracts for the server boundary, the response lifecycle and downstream effects. We will build those contracts using offline event fixtures and a synthetic service that can time out after committing.

    Chris: The interface encourages me to think in messages: request in, answer out. But streaming introduces a sequence, and effects introduce another system with its own outcome. If I collapse all three into one success boolean, I cannot explain whether the text was complete, whether approval was valid, or whether an external action happened. I want typed states and correlation identifiers so each observation belongs to the right operation.

    Koko: Start at the application server. The browser should send permitted user input and an identifier such as a ticket ID. The server authenticates the caller, derives authorized scope, validates input limits and invokes a configured provider adapter. Provider credentials stay in the server’s secret environment. A client-visible error should not contain an internal key or raw configuration. This boundary is useful even before a live adapter exists, because fixtures can exercise its behavior.

    Chris: If the browser supplies tenant retail-b while the authenticated actor belongs to retail-a, the server should reject the access before retrieval. Valid JSON and a real ticket ID are not enough. I would also reject arbitrary SDK options forwarded from the browser. Letting callers select every provider parameter would bypass our request budget and configuration policy, even if the SDK accepts those parameters syntactically. What should our normalized server contract contain?

    Koko: Define a normalized request with an application request ID, permitted input, authenticated caller context and deadline. Keep model choice, adapter version and resource policy in server configuration. Define normalized outcomes such as completed candidate, incomplete, refused, dependency failure or canceled. These names are application design choices, but they should map clearly to observed provider events. A normalized boundary lets us test the application without claiming a fixture predicts how a live model will answer.

    Chris: For diagnostic evidence, I would keep a safe correlation ID in the user response and a redacted internal record for the operator. Invalid input needs a useful client error; a dependency timeout needs a different category. Hiding every error behind success makes the interface look calm while destroying recovery information. Returning every raw exception has the opposite problem: it can expose sensitive dependency details that users do not need.

    Koko: Now consider the adapter abstraction. Its contract should describe inputs, observable lifecycle events, completion output and cancellation limits. An offline adapter can emit a known sequence without a key or network connection. A later live implementation must validate current SDK event shapes and selected model capabilities. Keep those two modes explicit in test reports. Passing a reducer test establishes local transition behavior, not live provider connectivity, cost or generation quality.

    Chris: That separation gives me a practical first test. I can feed created, text delta and failed into the application without invoking a provider. Then I inspect whether the UI state is failed and whether the visible text remains marked as an incomplete draft. If the page instead labels the draft final, I have found an application bug that a more capable model would not solve. How do we encode that lifecycle explicitly?

    Koko: Responses HTTP streaming uses typed server-sent events. Dispatch on event type rather than treating every event as prose. A created event and text deltas describe different observations, and a successful completion is different again. Our reducer can use pending, streaming, completed, failed and canceled states. The exact application state names are ours, while the event categories come from the documented interface. Maintain identifiers so interleaved items do not become one undifferentiated text buffer.

    Chris: I would write the transition table before the reducer. From pending, creation starts the run. While streaming, text deltas update a draft. Completion ends successful generation, subject to later application validation. Failure ends the run as failed. Cancellation ends local consumption as canceled. A malformed event should not magically create a successful state. Unknown event categories also should not be printed into the answer as if they were generated text.

    Koko: Partial function arguments need particular care. An argument fragment may happen to parse as JSON while the tool call is still being assembled. Do not execute because one prefix looks valid. Wait for the appropriate complete invocation, validate the full arguments and authorize the requested capability. The tool-call identifier connects a result to its request; it does not grant permission. Streaming is an observation mechanism, not an alternative authorization channel.

    Chris: The same distinction applies to structured response output. A user may see a complete-looking sentence or object before the run reaches its final successful state. I would keep presentation updates separate from acceptance of a final domain proposal. That protects downstream logic from treating a draft recommendation as approval. It also helps the UI explain an interruption honestly: we can retain useful draft text without pretending the operation finished successfully. What happens when events arrive after cancellation?

    Koko: Test a late event. The sequence is created, delta, application cancellation, late completion. In our workflow, canceled is terminal. The reducer must not reopen the run when that late event arrives. It may record a diagnostic observation, and other application components may reconcile a receipt, but the visible generation state should remain consistent with the cancellation contract. A guard at the terminal-state boundary is easier to inspect than scattered conditional updates.

    Chris: What if an external action was submitted just before the user clicked cancel? I cannot promise that closing the browser reversed it. Local stream consumption, remote generation and a committed business effect are different processes. I would abort the local read where supported, preserve the action operation key and reconcile its outcome. The UI should avoid a misleading promise such as everything was undone when the server has not established that fact.

    Koko: Exactly. Describe cancellation capabilities precisely at each adapter. It may stop local waiting, request remote cancellation, or prevent future local transitions, but none of those observations automatically compensates an already committed action. This is why the business ledger remains separate from text state. Even after the chat is canceled, an authoritative receipt may be essential operational evidence. Discarding it because the UI stopped listening can leave the system unable to reconcile an effect.

    Chris: I can see another wrong choice: resetting the reducer whenever a late delta appears, so the latest event always wins. That is simple code, but it allows failed or canceled runs to become streaming again. I would make terminal-state precedence explicit and test several orderings, including late text, late completion and a tool result. The test should assert final state, retained draft and absence of any newly authorized action.

    Koko: Streaming also changes content review. A partial sentence may be hard to evaluate before its completion, and checks on a completed output do not automatically cover text already displayed. The documented streaming guidance calls out moderation difficulty with partial output. Depending on the application risk, buffer until the relevant validation boundary or clearly limit what early content can trigger. Never let displaying a delta automatically commit a high-impact action.

    Chris: For this support lab, early text can show that a draft is being prepared, while the final proposal still needs schema, evidence and approval checks. That is a product decision tied to the use case. I would document the tradeoff rather than assume every application must stream every token. A faster first visible character is not the same metric as faster completion of a safe, useful workflow. How do we choose the right retry boundary?

    Koko: Now choose retry scope. A transient read failure before any effect may permit a bounded retry with backoff, jitter and an overall deadline. An invalid request needs correction, not repeated identical attempts. A timeout after an action submission is different: the service may have committed before the connection failed. Blindly restarting the agent can duplicate the action. Inspect SDK defaults too, because application retries layered over SDK retries can multiply attempts.

    Chris: Suppose the total deadline is two seconds and only fifty milliseconds remain. Starting another normally half-second search would violate the intended bound even if our retry counter has room. I would return a bounded failure or review state. A retry policy needs both count and time limits. Otherwise every layer can claim it made only a few attempts while the combined workflow becomes much slower than anyone expected.

    Koko: For the synthetic action service, persist one logical operation key before submission. After a commit followed by timeout, query that key or replay it under the service’s idempotency contract. It returns the original receipt instead of creating another effect. Keep attempt identifiers separately if useful for diagnostics. The stable business key identifies the intended action across attempts; a fresh random key on every retry defeats this protection.

    Chris: If the service lacks a status query and cannot deduplicate, I should use an explicit uncertain outcome. Marking failure would invite another action; marking success would invent a receipt. Uncertain is operationally inconvenient but accurate. The runbook needs an owner and reconciliation path. Our unit tests can prove decisions under the synthetic contract, but they cannot establish exactly-once behavior for a production dependency whose contract we have not demonstrated.

    Koko: Record the evidence needed to review that behavior: attempt count, operation key, deadline exhaustion, terminal generation state and authoritative action receipt when available. Redact payloads and never log credentials to make troubleshooting easier. Distinguish a provider response identifier from an application request ID and a business operation key. They serve different correlation purposes, and conflating them can accidentally create a new effect when a transport request is retried.

    Chris: I would build a retry decision table with rows for invalid input, unauthenticated caller, transient read timeout, generation interruption and action timeout after commit. Each row names whether retry is allowed, its budget, its key behavior and the terminal outcome if recovery fails. That table makes review concrete. It prevents an optimistic generic retry wrapper from silently imposing the same behavior on reads and irreversible effects. Which fixtures would prove these distinctions?

    Koko: In the lab, implement the reducer first with offline sequences, then add request-boundary tests, then the synthetic commit-and-timeout service. Test a stream without any terminal event, cancellation followed by a late delta, malformed arguments and caller-scope rejection. The crucial effect test submits a valid approved operation, simulates losing the receipt, and retries with the same key. Inspect the service counter and require one total effect, not one effect per successful response.

    Chris: My deliverables would include the transition table, typed server module, reducer tests, retry table and receipt log. I would also include a deliberate limit: removing downstream idempotency changes the expected outcome to uncertain, rather than leaving the same one-effect claim in place. That challenge shows whether I understand the service guarantee or merely copied a successful fixture. No live request is necessary to review those deterministic contracts.

    Koko: Before finishing, trace one request across all three boundaries. The server authenticates and limits it. The reducer interprets typed events without promoting drafts into final acceptance. The action adapter uses authoritative approval and a stable operation key. A failure at one boundary must not erase evidence at another. This layered view is what lets an operator distinguish an incomplete answer from an uncertain external effect and choose a safe recovery.

    Chris: I will leave with a precise integration promise: secrets stay server-side, terminal states survive late events, and retries respect the operation’s semantics and remaining deadline. When a dependency cannot support authoritative reconciliation, the application says uncertain rather than guessing. The next code review should be able to find each promise in a transition rule, a request validator, an adapter contract and an executed fixture test.