Agentic Architecture & Orchestration

Koko: Here is the thing about agentic systems: a weak prompt just gives you a weak answer. Maybe a little embarrassing, easily fixed. But a weak architecture? That double-charges a customer, loops forever, or burns through your entire compute budget before anyone wakes up. That is why this domain exists, and that is why it is the biggest one on the exam — a little more than a quarter of the whole thing.

Sam: Okay so the stakes are higher because the system is actually doing things in the world, not just answering questions.

Koko: Exactly. When an agent can call an API, write to a database, send an email — now a mistake has real consequences. The exam is testing whether you can turn 'have the AI do a big job' into something that survives real traffic. And there is one instinct that runs through every single question in this domain.

Sam: Just one? What is it?

Koko: Make the system predictable and recoverable — not just clever. That is it. That is the whole domain in seven words.

Sam: Predictable and recoverable. So like, structured over spontaneous?

Koko: Right. Think of it like this. Imagine you are running a busy restaurant kitchen. You could hire one incredibly talented chef and just tell them 'make something great tonight' — full autonomy, trust the genius. Or you could have named stations, a printed menu, a prep list, checkpoints between courses. If the sauté station goes down, the rest of the kitchen keeps moving. That second kitchen is predictable and recoverable. The first one is impressive until it is not.

Sam: And the AI version of 'hire one genius chef and trust them' is just... giving the model full autonomy and hoping?

Koko: That is almost word for word the trap the exam sets. 'Let the model decide' and 'give it full autonomy' sound like the advanced, sophisticated answer. They sound like you really trust the AI. But for a stable production system, that is usually the wrong choice.

Sam: So I would be tempted to pick the answer that gives the model more freedom, because it seems more powerful.

Koko: And that temptation is exactly what the exam is testing. Predictable beats impressive. The exam consistently rewards the more controlled, observable design. Every time you see a question where one option hands the model more freedom and another option adds structure, checkpoints, or guardrails — your instinct should lean toward the structure.

Sam: Okay but why does structure win? Is it just about being cautious?

Koko: It is about survivability. Decompose the work into named steps — so you know where you are. Checkpoint state so a run can resume instead of starting over. Make every step safe to retry — what engineers call idempotent — so a failure does not leave you in a weird half-finished state. And grant autonomy only in proportion to the risk of what that step is doing. Low-stakes step, sure, give it room. Step that sends money or deletes records? You bound that tightly.

Sam: So it is not that autonomy is always bad — it is that autonomy has to be proportional.

Koko: That is exactly the nuance the exam is looking for. The wrong answer is 'no autonomy ever.' The right instinct is 'autonomy scaled to risk.' And that instinct — controlled, observable, recoverable — is the lens you put on every question in this domain.

Sam: Got it. Predictable and recoverable. Not just clever.

Koko: Write that on your hand. We are going to see it come up in every single topic from here on out.

Koko: So we just covered what an agent actually is. Now let's talk about how it moves — because the loop is where architectures either hold together or fall apart.

Sam: Right, the loop is the thing that keeps the agent running until the job is done. But I'm a little fuzzy on what actually defines a good loop versus a sloppy one.

Koko: Think of a recipe card. Not the instruction 'cook dinner' — that's useless. An actual recipe: chop, sauté, simmer, plate. Four named steps, fixed order, and you're done when you check off the last line. Not when you feel finished. The loop is that recipe card.

Sam: So the nightly research assistant from the notes — that's the concrete version of this?

Koko: Exactly. Instead of one giant call saying 'analyze this company,' you decompose it: gather financials, pull peer data, fetch transcripts, draft insights, synthesize. After each step the loop asks a single question: what's next? When that pointer returns nothing, the plan is exhausted. The loop ends because the list ran out, not because the model decided it felt done.

Sam: Okay but — isn't letting the model decide each step dynamically more powerful? Like, more genuinely agentic?

Koko: It sounds more agentic, and that's exactly the trap. For a stable pipeline, open-ended dynamic planning costs you three things: you can't checkpoint it, you can't resume it if it crashes, and you can't budget it. The exam is going to test whether you know when fixed beats flexible.

Sam: So save the open-ended reasoning for genuinely open-ended tasks, not for a nightly pipeline that runs the same shape every time.

Koko: That's the instinct. Fixed ordered steps with an explicit next-step pointer, so the loop terminates because the plan runs out. Write that on your mental flashcard.

Sam: Alright. What's inside each turn of the loop that the orchestrator actually reads?

Koko: Every model reply comes with a stop reason — a small label saying why the turn ended. It finished naturally, it wants to call a tool, it hit a length limit, it hit a stop sequence. Four flavors, and your loop branches on that label.

Sam: Instead of just reading the text and guessing whether it's done.

Koko: Right. Think of a printer. It doesn't make you squint at the half-printed page and guess. It reports 'out of paper' or 'job complete.' You react to the status. Same thing here — the stop reason is the control signal.

Sam: So tool-use stop reason means run the tool and keep going, length limit means continue or raise the budget, end-of-turn means you're actually done.

Koko: That's the branching logic. And the trap is treating every response as final text. If you do that, you silently drop tool calls — the model asked for a tool and you shipped the partial prose as a finished answer. Nothing errors. It just quietly does the wrong thing.

Sam: Silent failure is so much worse than a loud crash.

Koko: Always. Read the stop reason first, prose second.

Sam: Okay, so the loop has named steps and reads stop reasons. But if each step is a separate execution, doesn't the agent forget everything between steps?

Koko: Complete amnesia, yes. So you give it a durable notebook — one labeled slot per step, in external storage, keyed by a run ID. Every time a step finishes, you write to that notebook immediately. Interrupted run? Resume from the last checked-off slot.

Sam: Like painting a fence one plank per visit and ticking off which planks are done.

Koko: That's the one. You can leave and come back all day without losing work. And for jobs longer than one execution window — say a long audio render that would blow past the platform's time limit — you do a bounded slice per invocation, save it, and let an external caller re-drive the loop until every slice exists.

Sam: So I shouldn't just keep results in memory and write once at the end, right? Like one big flush?

Koko: That's the trap, and it has a companion: 'run it as a long-lived background task.' Both lose everything on the first crash. And platforms routinely reclaim background tasks seconds after the response is sent — so fire-and-forget isn't just risky, it's often already dead.

Sam: Durable and incremental beats in-memory until the end.

Koko: Every time. The instinct: externalize state to durable storage keyed by a run ID, update it after every step, and keep each execution slice bounded so an external caller can safely re-drive the job. That's what makes a loop resumable — and resumable is what the exam calls production-grade.

Koko: Okay, so we just covered how agents plan and decide. Now let's talk about what happens when something goes wrong mid-execution — because it will. And the first concept you need tattooed on your brain is idempotency.

Sam: Idempotency. Same result whether it runs once or five times?

Koko: Exactly. The elevator button rule. You can mash that button ten times and exactly one elevator shows up. Same idea: a step in your pipeline should produce the same end state on the second run as the first — no duplicate records, no double charges.

Sam: Okay, but the instinct I'd reach for is just... prevent the retry. Add a lock. Make it so the step can only fire once.

Koko: And that is the trap. Preventing retries is the wrong goal. Networks blip. Callers time out and reload. Retries are how you survive those blips. You can't stop them, so you make them harmless.

Sam: So instead of blocking the second run, you make the second run a no-op.

Koko: Right. Gate the expensive work — the charge, the external call — behind an already-done check. Make every write a keyed upsert so a retry refreshes the one record instead of inserting a new one. The phrase the exam loves: at-least-once delivery plus an idempotent handler equals effectively-once.

Sam: That's a clean equation. Make the repeat cheap rather than impossible.

Koko: Exactly. Instinct on the exam: skip-if-done plus keyed upsert. Any answer that says disable retries or add a lock to prevent reruns is the wrong answer.

Koko: Now let's scale up. One agent is fine. But most real systems are a fleet. And the moment you have multiple agents, you need a strict org chart.

Sam: Strict how?

Koko: Three distinct roles. An orchestrator that scopes the job and delegates. Doer sub-agents that execute specialized work. And adversarial validators that inspect every output before anything ships. Think general contractor, subcontractors, and an independent building inspector.

Sam: And the inspector never quietly fixes the problem — they just flag it.

Koko: That's the key word: read-only. Validators can only inspect. They can't fix. And they're told to default to fail, to actively try to refute the output. That's what makes them adversarial.

Sam: So for, say, a report-assembly system — research doer and data doer run in parallel, their drafts go to separate fact-checking and math-checking validators, and only if every validator passes does the final report get assembled?

Koko: Perfect. The gate is an AND, not a majority vote. Ship if most checks pass is a weaker gate. You need every adversarial validator to pass.

Sam: What about giving every agent the delegation tool? More flexibility, right?

Koko: Classic trap. The moment every agent can delegate to other agents, you've turned a governable tree into an ungovernable web. You lose your audit trail. Only the orchestrator holds the delegation tool. That's what keeps the structure legible.

Sam: And the author checking their own work doesn't count as adversarial.

Koko: Never. The author is the worst-placed party to catch their own errors. Separation is the whole point.

Koko: Last concept for this module, and it ties the others together. When something fails in a multi-agent system, what's the first move?

Sam: Retry it?

Koko: Nope. Name it. Classify the failure before you react. Think triage in an emergency room — you sort by type before you treat.

Sam: Okay, what are the categories?

Koko: Four. Transient — a timeout, a blip — retry with backoff. Validation — bad input — fix or reject, do not retry. Business — a legitimate no from the system — surface it to the caller. Permission — missing access — escalate, and never route around it.

Sam: So only transient errors are actually retryable.

Koko: Exactly. The trap is wrapping everything in a blanket retry. A validation error will never fix itself on the next attempt. A permission error definitely won't. You just burn calls.

Sam: And when a sub-agent fails, it should return structured information — not just throw a bare exception.

Koko: Right. What it was doing, why it failed, any partial results, and a suggested alternative. That gives the orchestrator something to work with — it can degrade gracefully instead of just crashing. Classify first, then react. That's the instinct.

Koko: So here is the first edge case that trips people up badly on this exam. Picture a fuse box. When something goes wrong, the fuse blows and cuts the power. It does not wait to see if maybe everything is fine. That is the behavior you want from any guardrail protecting a costly or irreversible action — when it is uncertain, when it is misconfigured, it denies. That is called failing closed.

Sam: Okay, failing closed means deny on doubt. So the opposite would be — if the config is missing, just let it through so setup is not blocked?

Koko: Exactly the trap. That is fail-open, and it exposes your endpoint to the whole internet. The instinct the exam rewards is: for costly or irreversible actions, the guard runs before execution and when in doubt, it denies. Never lets through.

Sam: Got it. But here is the part I was fuzzy on — what about a spend monitor? Can you just have the agent check its own spend and shut itself down?

Koko: That is the second trap. Think about the fuse box analogy again — you also want a circuit breaker, but it has to be mounted outside the appliance it protects. If the agent has a bug, that same bug can disable the self-check. The kill switch has to be out of band, meaning a separate monitor that the agent cannot touch.

Sam: So the independence is the whole point. The monitor watches the agent, but the agent cannot reach back and mess with the monitor.

Koko: Independence is the load-bearing word here. A scheduled cost monitor checks the day's spend, hits a threshold, alerts a human, pauses the expensive functions. It lives outside the thing it is watching. That separation is what makes it trustworthy.

Sam: Okay, so let's say I need a guarantee that every write is scanned for secrets before it goes out. Can I just put that in the system prompt? Make it really clear, maybe repeat it?

Koko: That is exactly the temptation and exactly the trap. A prompt is a polite sign on the wall that says please badge in. A hook is the turnstile that physically will not turn without the badge. No amount of bold text turns a suggestion into a guarantee.

Sam: So the model might just — skip the instruction?

Koko: Prompts are best-effort. The model usually follows them, but there is no enforcement mechanism. A hook is code that fires on a lifecycle event — the write event in this case — and blocks the action if the scan has not happened. The model's decision does not matter. The hook runs regardless.

Sam: So the rule is: anything that must always happen, or must never happen, goes in a hook — not in prose.

Koko: That is the instinct. Invariants — the always and never — belong in deterministic code on an event. Judgment calls and guidance belong in prompts. If a guarantee is load-bearing, make it a hook.

Sam: Alright, autonomy tiers. My instinct says — if I am not sure, route everything to the strongest model. Safest approach, right?

Koko: That is the exam trap in a sentence. It throws away cost-aware escalation, which is the whole point of the architecture. Think of a junior analyst. They handle routine screening fast and cheap and kick anything alarming to a senior. You do not send every document straight to the senior.

Sam: Okay, so how do you decide what escalates?

Koko: Tie autonomy to risk and materiality. Read and compute steps — autonomous, cheap, fast. Anything that writes to a real system is human-gated by default, with the gate tied to how material the action is. A classifier reads thousands of documents with a cheap model and escalates a single case to an expensive model only when two cheap signals disagree. An agent auto-approves small actions but requires human sign-off above a threshold.

Sam: That makes sense. And what about rolling out a new agent — do you just flip it to autonomous once testing looks good?

Koko: Never a flip. That is the other trap. You advance on a staged ladder — shadow mode first, then progressive rollout, then hybrid. Shadow means the agent runs but a human still decides. You are watching for surprises before you give it any real rope.

Sam: So high accuracy in testing is not enough to skip the ladder.

Koko: Not even close. Testing does not surface the edge cases production will find. The ladder does. Staged rollout is how you earn autonomy, not how you skip to it.

Koko: Alright, let's put everything we've covered to work on some exam-style scenarios. First one: a system that assembles a research report. Several specialist agents fan out to gather information, then something needs to make sure the final report is accurate before it ships. Picture it like a newsroom — reporters out in the field, an editor back at the desk. Now, what does the tempting design look like?

Sam: I'd probably give every agent the ability to spin up its own helpers if it needs them. And for checking the draft — why not let the writer review its own work? It knows the material best.

Koko: And that is exactly the trap. When every agent can delegate, you end up with an ungovernable web. Nobody owns the audit trail. Who spun up which helper? Who authorized that sub-task? You lose the ability to reason about what the system is doing.

Sam: So delegation leaking everywhere is the problem, not delegation itself.

Koko: Exactly. The exam rewards one clean instinct: only the orchestrator delegates. The doer agents do their jobs and report back. That's it.

Sam: Okay. And the self-review thing — why is that wrong? The writer really does know their own draft.

Koko: Because an author grading their own work is not adversarial. They'll rationalize their own mistakes. You need an independent validator whose entire job is to try to refute the draft — read-only access, no stake in the outcome, and it defaults to fail. The report ships only when every validator passes.

Sam: Fail by default. So silence equals rejection, not approval.

Koko: Right. Fail closed. Now tie it to parallelism — the specialist doers can run in parallel because their work is independent of each other. Only genuine data dependencies get sequenced. If agent three needs agent two's output, that's a real dependency. If they're just researching separate topics, run them simultaneously.

Sam: So the architecture question is always: what actually has to wait, and what's just waiting because nobody thought about it?

Koko: That's the instinct. Unnecessary sequencing is waste. Real dependencies get sequenced. Everything else runs in parallel.

Koko: Scenario two. A nightly job processes a large batch of records. The platform has an execution-time limit, so the job keeps getting killed partway through. When it restarts, it either loses all progress or double-processes records it already handled. What's your first instinct?

Sam: Just run it as one long background task. Remove the time limit if you can, or find a platform that doesn't have one.

Koko: Classic answer. And it doesn't work, because the platform limit is usually not your limit to remove. You have to design around the constraint, not wish it away.

Sam: So what does the right design look like?

Koko: Three instincts, all connected. First: externalize state. Keep a record of what's been processed, keyed by a run ID, in storage outside the job itself. When the job restarts, it reads that checkpoint and picks up where it left off. Nothing lives only in memory.

Sam: So the job doesn't hold its own progress. The outside world holds it.

Koko: Exactly. Second instinct: process a bounded slice per invocation. Do a chunk of work, stop safely before you hit the time wall, checkpoint, and let an external caller re-drive the loop. You're not one long job — you're many short, safe invocations.

Sam: Like laps on a track instead of one marathon sprint.

Koko: Good analogy. Third instinct: make every step idempotent. Before processing a record, check if it's already done. Skip it if it is. Use a keyed upsert so if something runs twice, the second time is a cheap no-op, not a duplicate.

Sam: So even if a record gets reprocessed because of a crash, nothing bad happens.

Koko: Nothing bad happens. And here's the model to name out loud on the exam: at-least-once delivery combined with idempotent processing equals effectively-once behavior. The platform might deliver a record more than once. Your code ensures it only matters once. That's the contract you're designing for.

Sam: At-least-once plus idempotent equals effectively-once. That's clean. I can hold onto that.

Koko: Write that one on the back of your hand. It shows up everywhere — batch jobs, event queues, retry loops. The trap is assuming you'll only ever get a record once. Design for the retry. Make the retry harmless.

Koko: Alright, we have covered a lot of ground in this domain. Before you walk into the exam, let me give you the seven instincts in one tight pass — because this is twenty-seven percent of the test, and you want these to feel automatic.

Sam: Yes, please. Hit me.

Koko: First: prefer the fixed, named-step pipeline over 'let the model decide' whenever the work is stable. Stable work deserves a stable shape — observable, resumable, budgetable.

Sam: So the exam is not rewarding clever flexibility, it is rewarding predictability.

Koko: Exactly. Second: always read the stop reason and branch on it. Never treat every reply as final text. The stop reason is the signal — ignoring it is like ignoring a traffic light because the car is still moving.

Sam: Right, end-turn versus tool-use versus max-tokens are completely different situations.

Koko: Third: state lives in durable storage, keyed by a run id, checkpointed every step. Never in memory until the end, never a fire-and-forget background task. If the process crashes, you want to resume from the last checkpoint, not start over.

Sam: I keep wanting to say 'just hold it in memory, it is simpler.' But that is the trap.

Koko: It is always the trap. Fourth: make retries harmless. Skip-if-done plus keyed upserts. Do not try to forbid retries — assume they will happen and make them safe.

Sam: Idempotency by design, not by hope.

Koko: Nicely put. Fifth: many agents means a strict org chart. Only the orchestrator delegates. Independent read-only validators fail the gate on any single miss — any single one.

Sam: So a validator that passes everything through to be polite is worse than no validator at all.

Koko: Much worse. Sixth: classify the error before you react. Only transient errors are retryable. A permanent error that you keep retrying is not persistence, it is a loop.

Sam: Classify first, then decide. Got it.

Koko: And seventh, at the risky edges: guards fail closed, the kill switch lives out of band, invariants go in hooks not prompts, and autonomy scales with risk on a staged ladder. More risk, smaller steps, more checkpoints.

Sam: That staged ladder keeps coming up. Low stakes, move fast. High stakes, slow down and verify.

Koko: That is the through-line for this entire domain. When two answers compete on the exam, pick the one that could survive a crash, a retry, and a genuinely bad day in production. The design that is predictable and recoverable beats the design that just gives the model more freedom.

Sam: That framing actually makes the choices feel obvious once you see it that way.

Koko: That is the goal. Keep sharpening these with the flashcards and the quiz over at KokoAI Academy on koko knows dot A I — this is the biggest slice of the exam, so it is the one most worth drilling. You have done the work. Now go show them.