Context Management & Reliability

Koko: Here is the instinct that unlocks this entire domain: assume every run will be interrupted, re-triggered, and long — and design so none of it hurts.

Sam: That is a pretty bleak way to open a lesson.

Koko: Good bleak, though. Domain five is context management and reliability — the unglamorous machinery that keeps a system remembering, recovering, and not repeating or crashing. It is about fifteen percent of the exam, which sounds modest until you realize its failures cascade into every other domain.

Sam: Cascade how? Like if context breaks, what actually goes wrong?

Koko: Think of it like a surgeon who steps out mid-operation and comes back with no memory of where they were. The scalpel is still sharp. The skill is still there. But without the context of what has already happened, the next move can undo everything that came before it.

Sam: Okay, so it is not that the model is bad — it is that the scaffolding forgot.

Koko: Exactly. And the exam knows that. Questions in this domain describe something going wrong — a pipeline re-triggers, a context window overflows, a retry fires twice — and they reward the design that bends under pressure and keeps its footing, not the one that assumed the happy path.

Sam: So what does that instinct actually tell you to do in practice?

Koko: Several things that all hang together. First, pin the critical context and bound the history so it cannot grow forever. If you let history accumulate without a ceiling, eventually you blow the context window or your latency tanks. You have to make deliberate choices about what stays.

Sam: And I assume the trap is just keeping everything, because more context feels safer?

Koko: That is the trap exactly. More is not safer if it crowds out the things that actually matter. The instinct is to be selective and bounded, not maximal.

Sam: What else does the instinct cover?

Koko: Give a recurring generator — say a nightly digest — a memory of its own output, so it does not repeat itself across runs. Tag every fact with where it came from, so the system can reason about provenance. And then on the reliability side: retry transient errors, but politely and with idempotency so the retry does not make things worse.

Sam: Idempotent meaning if it runs twice the result is the same as if it ran once?

Koko: Exactly right. At-least-once delivery is a real world fact, and your system has to be designed so that running an operation twice does not double-charge someone or duplicate a record.

Sam: And the last piece?

Koko: Degrade gracefully instead of crashing — and choose deliberately, per guard, which failures fail open and which fail closed. That last word, deliberately, is doing a lot of work. The exam will give you a scenario and ask which behavior is right. The trap is picking one rule for everything. Some guards should always fail closed. Others can let traffic through. You have to reason about each one.

Sam: So the whole domain is basically: do not assume the happy path, and have a plan for every way it can go sideways.

Koko: Pin that to the wall. Every question in domain five is testing some version of that sentence. The system that earns points is the one that already expected the interruption.

Koko: So let's talk about what actually goes in the context window and how you control it. The instinct most people start with is just send everything and let the model sort it out. That instinct will hurt you on this exam and in production.

Sam: Okay but if the window is big enough, why not?

Koko: Because bigger windows still fill up. And unbounded history is a slow-motion cost and truncation bug. Here's the move: you do two things at once. You pin the load-bearing context into the message stream so it's always there, and you independently bound the history so it cannot grow without limit.

Sam: What do you mean by pin it into the message stream?

Koko: Say you have a chat agent and it needs a long policy document. You inject that document directly into the first user message. It doesn't live somewhere external and get hoped for. It is in the stream, guaranteed in context for every follow-up. That's pinning.

Sam: And the bounding part is like a turn cap?

Koko: A turn cap and a per-message length cap. Both. The analogy I like is taping the key document to the meeting-room wall so everyone can see it, while agreeing to only carry the last half hour of conversation in your heads. The document is always visible. The chatter is bounded.

Sam: And there's something about where in the context things land?

Koko: Yes, this is the trap a lot of people miss. Material buried in the middle of a long context is easier to lose than material at the start or end. It's called the lost-in-the-middle effect. So placement is a design choice. Put must-not-miss instructions where attention is strongest.

Sam: So the exam instinct is: placement is deliberate, not accidental.

Koko: Exactly. Context management should be deterministic, not accidental. And the trap to avoid is thinking the answer is just a bigger window or sending the whole history every time.

Sam: Okay so what happens in a long-running agent? I feel like even with bounding, context fills up fast when tools are involved.

Koko: It absolutely does. A long-running agent's context fills with stale tool output. The full API response you needed one field from. Transcripts you've already summarized. You have to trim.

Sam: Trim meaning delete it?

Koko: Keep the distilled fact, drop the raw payload. The window should hold signal, not history. The analogy is taking one note from a phone call and recycling the transcript. Not stapling every transcript to your forehead.

Sam: That's a vivid image. But what if you need to refer back to it?

Koko: That's where the scratchpad comes in. You write intermediate work to a file or a durable note and reference it rather than carrying every detail inline. And you persist structured state in labeled slots so your conclusions survive even as the raw conversation is trimmed.

Sam: And for really heavy sub-tasks?

Koko: Delegate to a subagent with its own context and take back only the summary. You offload the weight, you get the answer.

Sam: So the trap is keeping every raw payload just in case.

Koko: Right. Keeping every raw payload and full transcript just in case it's useful later crowds the window and pushes the actually important thing off the edge. Trim to the fact, scratchpad for intermediate work, subagent for heavy lifting.

Sam: One thing I've seen come up in practice is a daily content generator that just keeps repeating itself. Same topics, same angles. My first instinct is to dial down the temperature.

Koko: That's the trap. Temperature governs word-level variety. It does not tell the model it already covered a story yesterday.

Sam: Oh. So it's not a creativity problem.

Koko: It's a missing-context problem. Repetition almost always is. The fix is to give the model explicit memory of its recent outputs.

Sam: How does that work in practice?

Koko: You compute the set of topics the generator covered in recent editions and hand that list to the model as do not repeat these, plus the last few outputs. And you persist a per-run ledger so that memory survives across days, not just within a session.

Sam: So it's like skimming last week's headlines before you write today's column.

Koko: Exactly that. You don't re-run a story you already covered because you actually know you covered it. The model doesn't know unless you tell it.

Sam: So the instinct is freshness is an input problem, not a parameter problem.

Koko: That's the one. Give the model a ledger of its recent outputs, instruct it to build on them. The exam will offer you the temperature dial as the obvious answer. Don't take it.

Koko: Alright, next idea is provenance, and it is one of those things that sounds like housekeeping until you realise it is doing three jobs at once.

Sam: Three jobs — okay, what are they?

Koko: Trust signal, cost signal, and honest abstention. Every fact your pipeline retrieves should carry a tag that says where it came from and when it was true. Think of it like a museum label on every exhibit — without the label you cannot tell the genuine article from a replica.

Sam: And the when matters as much as the where?

Koko: Exactly. A figure without a date silently goes stale and gets read as current. So a data layer stamps each result with its source and its timestamp. One glance tells you whether this came from a cheap deterministic API or an expensive model-driven web search — and that is also your cost audit right there.

Sam: So provenance is not just about correctness, it is literally money.

Koko: Right. And when the model cannot find a source at all, you instruct it to return an empty result with a note — not to invent one. Honest abstention instead of a hallucination.

Sam: What about when two sources disagree? Do you just average them?

Koko: That is the trap. Averaging two conflicting sources produces a number nobody actually reported, and now you have hidden the disagreement. Surface the conflict with attribution — tell the caller both figures and who said each one.

Sam: Okay, so the instinct is provenance travels with the claim, not in a droppable footnote.

Koko: Nail it. And the trap the exam loves: the answer is what matters, where it came from is metadata we can drop. Drop it and you cannot verify, you cannot cite, you cannot catch a hallucination, and your cost audit is blind.

Sam: Alright, next is error handling. My instinct is just stop the whole pipeline the moment something fails — keep it clean.

Koko: That is the tempting answer and the exam will punish it. Think about a navigation app that loses signal. Does it go black?

Sam: No, it says recalculating and keeps you on the last known road.

Koko: Exactly. Fail loud where it matters, fail soft where you can still be useful. When an optional document cannot be fetched, the pipeline injects a note — none found, proceed without it and ask the user to paste it — and delivers a reduced but real answer.

Sam: Reduced but real. I like that. What about when the model call itself throws?

Koko: Emit a machine-readable, typed error the caller can act on. Not a blank spinner. The blank spinner is the worst outcome — zero value delivered and zero signal about why.

Sam: So there are actually two traps here: stop everything on any failure, and swallow the error silently.

Koko: Two sides of the same bad coin. The instinct is: structured typed error the caller can act on, and prefer degrading to a partial or qualified result over hard-failing when the task can still deliver value.

Sam: Okay, last one for this module — retries. If something fails, just retry immediately until it works, right?

Koko: That is the trap in a single sentence. If the phone line is busy you wait a bit longer before each redial rather than hammering redial. And you stop dialing a number that has been disconnected.

Sam: So two different behaviors depending on the type of failure.

Koko: Exactly. Transient errors — rate limits, server errors, network blips — you retry with exponentially growing delays plus a little randomness, jitter, and you cap the number of attempts. Permanent errors — a not-found — you fail fast, because a not-found will never become a success.

Sam: What is jitter actually doing there?

Koko: If every client backs off on the exact same schedule they all slam the server again at the same moment. Jitter spreads them out. And you also throttle how many requests fly at once with something like a semaphore — so you do not cause the rate limit yourself.

Sam: You can be your own problem. That is a little embarrassing.

Koko: Very common. The instinct is: retry transient with exponential backoff and jitter, cap the attempts, throttle concurrency, distinguish transient from permanent. The trap is retrying immediately in a tight loop — you hammer an already-struggling service, stay rate-limited, and if the error is permanent you loop forever.

Sam: So provenance for trust and cost, structured graceful errors instead of silent failure or hard stops, and smart retries that know when to wait and when to quit. That is the whole trust and recovery picture.

Koko: That is it. Each one is the same underlying instinct: give the system — and the people downstream — enough signal to act, not just a result with no story behind it.

Koko: So when a guard's backing service hiccups — the rate-limit store goes down, the secret isn't loaded — the system still has to do something. And that something is a deliberate design choice, not a default.

Sam: Right, it either lets traffic through or it blocks everything.

Koko: Exactly. Fail open or fail closed. And the exam wants you to match that direction to the guard's actual purpose. Think of two different door mechanisms. A turnstile in a building unlocks automatically in a fire — it fails open — because the priority is getting people out. A bank vault, on the other hand, stays locked if its controller dies — it fails closed — because the priority is protecting the assets.

Sam: So a rate limiter is more like the turnstile.

Koko: Exactly right. A per-IP rate limiter is an availability and cost-smoothing guard. If its store errors, you let users through. A transient glitch should not become an outage for real users.

Sam: But an admin gate on a billed endpoint — that one stays locked.

Koko: Right. If the secret is unset, refuse everything. A broken deploy is much better than an open door burning credits. The cost-authorization gate fails closed.

Sam: So the tempting wrong answer on the exam is just picking one rule for everything — like, guards should always fail closed to be safe.

Koko: That's the trap. A blanket always-fail-closed rule turns your rate limiter into an outage button the moment your Redis instance wobbles. And always-fail-open turns your auth gate into a free-credits vending machine. The instinct is: ask what this guard is protecting. Availability and cost-smoothing — fail open. Security and cost-authorization — fail closed.

Sam: Okay, shifting gears. We've talked a lot about the system around the model, but what about the model itself knowing when it's out of its depth?

Koko: That's confidence calibration, and it's one of the quieter reliability levers. A well-designed agent has explicit escalation triggers — low confidence, genuine ambiguity, high stakes, something out of scope. When one fires, it doesn't guess. It hands off.

Sam: And the handoff is more than just saying I don't know.

Koko: Much more. Think of a good junior on a team — they don't just drop a case in your inbox. They hand you a clean file: here's what the user asked, here's what I tried, here's why I'm escalating. The human can act immediately without re-deriving everything from scratch.

Sam: So the structured handoff has three things — context, what it tried, and why it's escalating.

Koko: Exactly. And there's a prerequisite that often gets missed: the confidence itself has to be honest. If the model is calibrated to always sound sure, the trigger never fires. A confidently wrong answer on a high-stakes case is worse than a handoff.

Sam: So the trap is telling the model to always give its best confident answer.

Koko: Right. A model that's sure of everything can never escalate. The instinct is: define your triggers, require a structured handoff with full context, and make sure the confidence signal is honest enough to actually fire.

Sam: Alright, last one for this module. You've got a pipeline running at scale — tens of thousands of outputs a day. You can't review all of them. But you also can't just assume it's fine.

Koko: Right, and this is where teams get caught. The failure mode isn't a crash — it's quiet drift. Quality slowly degrades and nobody notices until a user complains, which means the damage is already done.

Sam: So sampling. But I'd probably just check the first ten outputs in the morning, make sure they look okay.

Koko: That's the trap. Checking the first ten isn't representative. Think about quality control in manufacturing — you don't pull samples only from the morning shift on one line. You pull across every shift and every line, because that's how you catch something that's only failing in one segment.

Sam: Stratified random sampling — covering the important segments of your data.

Koko: Exactly. It lets you measure accuracy at scale and quantify drift before it becomes a crisis. And the exam pairs this with graceful degradation — your sampling tells you something is wrong, and your architecture is built so a partial failure in one part doesn't collapse the whole system.

Sam: So the instinct is: sample across segments to measure, and design for partial failure so you don't need everything to be perfect to stay running.

Koko: That's it. And the trap to remember is both ends of the wrong answer — assuming it's fine until someone complains, and reviewing only a non-representative slice. Neither one actually tells you what the system is doing.

Koko: Alright, let's put everything we've been talking about into scenarios — the kind the exam actually throws at you. First one: you have a chat agent. It fetches a long reference document at the start of a session, answers the first question beautifully, and then three or four turns later it starts contradicting the document. Like it never read it.

Sam: Oh, that's the context window filling up. So I'd reach for a model with a bigger context window, right? More room, problem solved.

Koko: That is exactly the trap. And it's tempting because it feels like a capacity problem. But bigger windows still fill up — you've just bought yourself more time before the same failure.

Sam: So more room, same cliff, just further away.

Koko: Exactly. The real fix is deterministic, not hopeful. You pin the load-bearing document into the message stream so it is guaranteed present for every single follow-up turn. It doesn't drift. It doesn't get squeezed out by conversation.

Sam: Okay, but what stops the conversation itself from eventually pushing even the pinned document off the edge?

Koko: You bound the history independently. Cap the number of turns you carry. Cap per-message length. So runaway chatter can never grow large enough to crowd out the thing you pinned. You're controlling both sides of the equation.

Sam: So pinning holds the important stuff in place, and bounding keeps everything else from expanding to fill the space.

Koko: Right. And then you go one step further — trim verbose tool output down to just the actual fact you needed, and persist the key conclusions you've already extracted. That way even as raw turns get trimmed away, the meaning survives.

Sam: So the instinct the exam wants is: manage what's present deliberately, don't just hope the window is big enough.

Koko: That's the one. Pinning plus bounding is deterministic. Bigger window is just optimism.

Koko: Second scenario. You have one system protected by two guards. A per-user rate limiter — its job is smoothing load. And an admin gate protecting a billed, expensive endpoint. One day the backing store behind both of them hiccups. What do they do?

Sam: Security instinct says fail closed. Everything refuses until the store is back. Safe by default.

Koko: Good instinct, wrong application. That blanket rule breaks one of the two guards.

Sam: Which one?

Koko: The rate limiter. Its job is load smoothing — it's a convenience guard. If its backing store has a transient glitch and it fails closed, you've just locked out real users over a hiccup. The cost of that failure is worse than briefly skipping the rate check.

Sam: So the rate limiter should fail open. Let traffic through, deal with the glitch, no harm done.

Koko: Yes. Now the admin gate — that one guards a billed endpoint. If its secret is unset or unreadable, what happens if it fails open?

Sam: Anyone gets through. Credits burn. Potentially a lot of them.

Koko: So that gate must fail closed, hard, always. A broken deploy is far better than an open door burning spend. Same backing store hiccup, opposite correct behaviors, because the guards are protecting completely different things.

Sam: So the instinct isn't secure by default across the board — it's match the failure direction to what the guard is actually protecting.

Koko: That's the exam instinct. And there's a related one here: meter and persist spend incrementally. If a crash happens mid-run and you haven't been recording costs as you go, you've lost your cost control for that whole run. The crash doesn't just cost you the run — it costs you the accounting.

Sam: So you track spend in small steps continuously, not in one big write at the end.

Koko: Exactly. Because the end might not come cleanly. The trap is assuming your process completes — design assuming it won't.

Koko: Alright, last stop. Domain five is fifteen percent of the exam, and we've covered a lot of ground. So let's do what good architects do before they ship — run the checklist.

Sam: I'm ready. Hit me.

Koko: First instinct: pin the load-bearing context into the message stream and bound history independently. Never reach for a bigger window as your answer.

Sam: Right — the window is a budget, not a fix.

Koko: Exactly. Second: trim stale tool output down to the fact that matters. Use a scratchpad for working memory and persist the state you actually need to carry forward.

Sam: So the scratchpad is ephemeral, persisted state is intentional.

Koko: Good distinction. Third: a recurring generator needs a memory of its recent output to stay fresh. Not a colder temperature.

Sam: That one tripped me up early on. Temperature feels like the obvious dial to turn.

Koko: And that's exactly why it's a trap. Repetition is a memory problem, not a randomness problem. Fourth instinct: provenance travels with every claim — source and time — and when two sources conflict, you surface the conflict with attribution. You never silently average them.

Sam: Silent averaging. That one sounds so reasonable until you think about what you're actually hiding.

Koko: You're hiding the disagreement from the person who most needs to know about it. Fifth: propagate a structured, typed error and degrade to a partial result. Don't crash silently and don't halt entirely.

Sam: Partial over nothing, and never silent.

Koko: Sixth: retry transient errors with backoff and jitter, cap your attempts, throttle concurrency, and fail fast on permanent errors. Permanent errors don't get retried.

Sam: Because retrying a permanent error is just expensive stubbornness.

Koko: Beautifully put. Seventh: match a guard's failure direction to its job. Availability guards fail open. Authorization guards fail closed. And you meter spend incrementally — you don't commit the whole budget upfront.

Sam: Fail open, fail closed — the job tells you which direction.

Koko: Eighth and last: escalate on honest low confidence with a structured handoff, and sample across segments to catch quiet drift before it becomes a real problem.

Sam: Quiet drift is sneaky. The system keeps running, just slowly getting worse.

Koko: Which is sometimes harder to catch than a full outage. Now, here's the through-line that ties all eight together. Assume interruption. Assume re-triggering. Assume length. Design so that none of those things hurt you.

Sam: So when two answers compete on the exam, pick the one that holds up under partial failure.

Koko: Not the one that assumes everything goes right. That's the whole domain in one sentence. And yes, it's the smallest domain — but these habits are what keep the other four from cascading into an outage. They're the load-bearing walls.

Sam: Okay, I feel like I actually have a handle on this now.

Koko: Keep sharpening with the flashcards and the quiz at KokoAI Academy — that's koko knows dot A I. That's the full five-domain tour. Now go pass that exam.