Context Management & Reliability

Koko: Here is the one thing Domain Five is testing underneath every question: assume the run will be interrupted, re-triggered, and longer than you planned — and design so that none of it hurts. That is the whole domain in one sentence.

Chris: So the exam is not asking whether your system works on the happy path. It is asking whether it survives the second and third run after something went sideways.

Koko: Exactly. And this is the unglamorous domain. No one puts context management on a conference slide. But when it breaks, it breaks everything else — which is why it is worth fifteen percent of the exam even though it is the smallest domain by weight.

Chris: The cascade problem. A bad memory layer poisons the outputs that every other domain depends on.

Koko: Right. Think of it like the plumbing in a building. Nobody tours the pipes. But if the pressure drops, every faucet on every floor feels it. Domain Five is the pipes.

Chris: So what does the exam actually test inside that instinct? Because there are several moving parts — context pinning, retry logic, idempotency, graceful degradation.

Koko: All of those, yes, and we will go through each one. But they all follow from the same posture. Pin the critical context and put a ceiling on history so it cannot grow forever. Give a recurring generator a memory of its own prior output. Tag every fact with its source. Retry transient errors politely. Stay idempotent so retries are safe. Degrade gracefully instead of crashing. And choose deliberately — per guard, not globally — which failures fail open and which fail closed.

Chris: That last one is interesting. Fail open versus fail closed is a per-guard decision, not a system-wide setting. So a rate limiter might fail open to keep traffic moving, but a safety check fails closed because the cost of a false negative is too high.

Koko: That is exactly the discriminator the exam is testing. You just restated the rule perfectly. The trap is treating fail-open or fail-closed as a one-size-fits-all architecture choice — as if you pick a mode for the whole system and you are done.

Chris: And in a question stem, the signal that it is a Domain Five problem is that something is described going wrong — a network hiccup, a long conversation, a repeated job — and the answers are different designs for handling that failure.

Koko: Yes. When the question describes pressure — an interruption, a retry, a context that has grown too long — the exam rewards the design that bends and keeps its footing. Not the one that assumes everything arrives once, in order, on the first try. That assumption is the trap the whole domain is built around.

Chris: So the instinct to carry into every Domain Five question is: where does this design break under the non-happy path, and does it recover cleanly?

Koko: That is it. Hold that question in your head for every scenario we cover from here. Let's go build the machinery.

Koko: So here's the core tension with a growing conversation: the context window is finite, but the conversation wants to grow forever. You have to do two things simultaneously — keep the load-bearing context visible, and stop the history from expanding without limit.

Chris: Pin the critical stuff in, cap everything else.

Koko: Exactly. Think of it as taping the key document to the meeting-room wall — it's always visible — while agreeing you'll only carry the last half hour of conversation in your heads. The document doesn't age out. The chatter does.

Chris: So concretely: a chat agent that needs a long reference document injects it directly into the first user message, so it's guaranteed in-context for every follow-up, and then you separately put a turn cap and a per-message length cap on the conversation history.

Koko: That's it. And here's the subtlety that shows up on the exam — where in the context you place things matters. Material buried in the middle of a long context is easier to lose than material at the start or end. That's called the lost-in-the-middle effect.

Chris: So placement is a deliberate design choice, not an afterthought.

Koko: Right. The instinct the exam rewards: put load-bearing context into the message stream, independently bound history length, and place must-not-miss instructions where attention is strongest. Make it deterministic, not accidental. The trap here is assuming you can just use a bigger context window, or send the whole history every time. Bigger windows still fill up, and unbounded history is a slow-motion cost and truncation bug waiting to happen.

Chris: What about long-running agents? A multi-step tool-using agent accumulates a lot — full API responses, intermediate reasoning, transcripts it already summarized.

Koko: This is where context editing comes in. When a tool returns a full API dump and you only needed one field, trim it. Keep the distilled fact, drop the raw payload. The window should hold signal, not history.

Chris: Like taking one note from a phone call and recycling the transcript instead of stapling every transcript to your forehead.

Koko: I love that image. Two more moves sit alongside trimming. First, the scratchpad: write intermediate work to a file or a durable note and reference it, rather than carrying every detail inline. Second, persist structured state in labeled slots — so conclusions survive even as the raw conversation gets trimmed.

Chris: And for genuinely heavy sub-tasks you can delegate to a subagent with its own context and just take back the summary.

Koko: Exactly — you get the answer, not the working. The instinct: trim verbose tool results to the fact, use a scratchpad and persisted state, offload heavy sub-work to a subagent, and keep only its summary. The trap here is keeping every raw payload and full transcript in case it's useful later. That strategy crowds the window and pushes the actually important thing off the edge.

Chris: Okay, what about a recurring generator — say a daily content pipeline that starts repeating itself? The option sitting next to the right answer on the exam will say lower the temperature.

Koko: And that's the wrong dial entirely. Temperature governs word-level variety — it does not tell the model it already covered a story yesterday. Repetition is almost always a missing-context problem.

Chris: So the fix is giving the model explicit memory of its recent outputs — a ledger of topics it's covered — and instructing it to differentiate.

Koko: Precisely. Like skimming last week's headlines before writing today's column so you don't re-run a story. A daily generator computes the set of topics it covered in recent editions, hands that list to the model as 'do not repeat these,' plus the last few outputs, and persists a per-run ledger so that memory survives across days.

Chris: So freshness is an input problem. You engineer it in by feeding recent history forward.

Koko: That's the instinct exactly. Give the model explicit memory of its recent outputs, instruct it to build on them, and persist the ledger. Reaching for temperature when the model is repeating itself is the trap — it treats a context gap as a randomness problem, and those are not the same thing.

Koko: Alright, provenance. Think of it as the museum label on every exhibit. Without that label you cannot tell a genuine artifact from a replica.

Chris: So every fact the pipeline surfaces needs to carry where it came from and, critically, when it was true — not just the value itself.

Koko: Exactly. Source plus time. A figure without a date silently goes stale and gets read as current. That staleness is invisible — which is what makes it dangerous.

Chris: The example in my notes is a data layer that stamps whether a result came from a cheap deterministic API or an expensive model-driven web search. One glance and you know both how much to trust it and what it cost.

Koko: Right, so provenance is doing double duty — it's a trust signal and a cost signal at the same time. And it lets the model say 'no source found' instead of inventing one. That honest abstention is a feature, not a failure.

Chris: What about conflicting sources? If two results disagree, the instinct is to surface the conflict with attribution rather than silently picking one?

Koko: Yes. Never average two disagreeing numbers into a figure nobody actually reported. Surface the conflict, name both sources, and let the caller decide. The trap here is treating provenance as droppable metadata — 'the answer is what matters, we can drop where it came from.' The moment you do that you cannot verify the claim, you cannot catch a hallucination, and the cost audit is blind.

Koko: Next: what happens when a step in the pipeline breaks. The nav app analogy — when it loses signal it says 'recalculating, proceed on the last known road.' It does not go black.

Chris: So the pipeline emits a structured, typed error the caller can act on, and where the task can still deliver value it degrades to a partial result rather than hard-failing.

Koko: That's it. 'Fail loud where it matters, fail soft where you can still be useful.' The notes give a clean example: if no provider can supply an optional document, inject a note — 'none found, proceed without it and ask the user to paste it' — and produce a reduced but useful answer.

Chris: And if the model call itself throws, it emits a machine-readable typed error rather than leaving a blank spinner.

Koko: Blank spinner is the worst outcome. No value, no signal. The exam will set up two traps next to each other: 'stop the whole pipeline on any failure' throws away value you could still deliver, and 'swallow the error silently and return nothing' gives the caller nothing to act on. The instinct to reward is: structured signal plus graceful degradation.

Koko: Last one: retries. If a phone line is busy, you wait a bit longer before each redial. You do not hammer redial. And you do not keep dialing a number that has been disconnected.

Chris: So exponential backoff plus jitter for transient failures — rate-limit responses, server errors, network blips — bounded number of attempts, and a concurrency cap so the pipeline does not cause the rate limit itself.

Koko: Perfect restatement. The jitter is important — randomness prevents every client from retrying in lockstep and spiking the service again together.

Chris: And the discriminator between transient and permanent matters a lot. A not-found is permanent — retrying it is pure waste.

Koko: Exactly. Fail fast on permanent errors. The trap here is 'retry immediately in a tight loop until it succeeds' — that hammers an already-struggling service, keeps you rate-limited, and can loop forever if the error is permanent. The option sitting next to the right answer on the exam will say something like 'retry up to ten times with no delay.' That's the one to walk past.

Chris: So the instinct is: transient gets backoff plus jitter plus a cap, permanent gets an immediate structured failure, and throttle concurrency so you are not the cause.

Koko: That's the whole picture. Provenance travels with the claim. Errors propagate with structure and degrade gracefully. Retries are patient, bounded, and know when to stop.

Koko: So when a guard's backing service hiccups — rate-limit store goes down, secret store is unreachable — the system still has to do something. And that something is a deliberate design choice, not a default.

Chris: Right, fail-open or fail-closed, and the choice has to match what the guard is actually protecting.

Koko: Exactly. Think of it as two very different doors. A turnstile in a building unlocks automatically in a fire — it fails open because the priority is flow, getting people out. A bank vault stays locked if its controller dies — it fails closed because the priority is protecting assets. Same logic, different jobs.

Chris: So a per-IP rate limiter should fail open — if the rate-limit store errors, you let users through rather than lock out real traffic over a transient glitch.

Koko: That's it. A few extra calls is a far better outcome than a full outage caused by a hiccup in one dependency. But flip the scenario: an admin gate on a billed endpoint — what does it do if its secret is unset?

Chris: Refuses everything. Better a broken deploy than an open door burning credits.

Koko: Perfectly put. And the instinct the exam rewards is: match the failure direction to the guard's purpose. Availability or cost-smoothing limiters fail open. Security or cost-authorization gates fail closed. Also — meter spend incrementally and persist it, so cost control survives a crash.

Chris: The trap I'd expect on the exam is a blanket rule — something like 'all guards should fail closed for safety.'

Koko: That's exactly where the exam will bite you. 'Always fail closed' turns a rate limiter into an outage button. 'Always fail open' turns an auth gate into an open door. The answer is always: it depends on the job.

Koko: Next one: a reliable agent knows the edge of its own competence. That sounds obvious, but it has a very specific shape.

Chris: It's about escalation triggers — low confidence, ambiguity, high stakes, out-of-scope — and when any of those fire, you don't guess, you hand off.

Koko: And how you hand off matters as much as when. Think of a good junior who flags a case they're unsure about — they don't wing it. They hand over a clean file: here's the context, here's what I tried, here's why I'm escalating. The human can act immediately without re-deriving anything.

Chris: So the structured handoff is context plus attempts plus reason. Got it. What's the discriminating question — when does escalation actually fail?

Koko: Great question. It fails when confidence isn't honest. If the model is miscalibrated and reports high confidence on something it's actually shaky on, the trigger never fires. The agent just answers with a confident guess on a high-stakes case — which is the worst possible outcome.

Chris: So calibration is the prerequisite. The trigger only works if 'not sure' actually evaluates to 'not sure.'

Koko: Right. And the trap here is 'always give a confident best guess.' The exam will make that sound responsible — it's not. A confidently wrong answer on a high-stakes case is strictly worse than a handoff. A model that's sure of everything can never escalate.

Koko: Last concept in this section. High-volume pipeline, tens of thousands of outputs a day. You can't hand-check every one. But you also can't just assume it's working.

Chris: Stratified random sampling — you pull a representative slice across the important segments. That way you're measuring accuracy across the whole distribution, not just whatever's easiest to check.

Koko: Exactly. The analogy is factory quality control: you pull samples across every shift and every line, not just the first item off the belt on Monday morning. The first ten outputs could be fine while a whole segment is quietly drifting.

Chris: And 'stratified' is the key word — sampling randomly across segments means you'd catch drift that's confined to one slice rather than averaging it away.

Koko: Exactly right. Pair that with graceful degradation — partial failures shouldn't collapse the system — and you have a pipeline that both measures its own quality and degrades safely when something goes wrong.

Chris: The option sitting next to it on the exam will say 'monitor user complaints' — surface errors only, no proactive measurement.

Koko: And that's silent drift. The system can be failing a whole segment for weeks before anyone reports it. The instinct is: sample across segments to measure accuracy at scale. Don't wait for complaints — they lag the problem by design.

Koko: Let's put what we've covered into a couple of real scenarios — the kind the exam drops you into. First one: you build a chat agent that fetches a long reference document at the start of the session. First question, perfect answer. Third question, it starts contradicting the document. What broke?

Chris: The document fell out of context. The conversation grew long enough that the document got pushed off the edge, and now the model is just winging it from whatever's still in the window.

Koko: Exactly. And notice two problems at once — the answers are wrong, and the bill is climbing. So what has to be true about that document?

Chris: It has to be guaranteed present, not just hoped for. You can't rely on it surviving a long conversation by accident.

Koko: Right, guaranteed, not hoped for — that's the instinct the exam rewards here. Now the trap: the option sitting right next to the correct answer will say 'use a model with a bigger context window.' Sounds reasonable. Why is it wrong?

Chris: Because bigger windows still fill up. It delays the problem, it doesn't solve it. You'll hit the same failure eventually, just later and more expensively.

Koko: Exactly. The real design is deterministic, not hopeful. You pin the load-bearing document into the message stream — it's present for every follow-up, not just the first one. And independently, you bound the history: cap the number of turns and limit per-message length, so runaway chatter physically cannot push the pinned document off the edge.

Chris: So pinning and bounding work together. Pinning keeps the document in; bounding keeps the noise out.

Koko: That's it. And then you extend it one more step: trim verbose tool output down to just the extracted fact, and persist the key conclusions somewhere durable. That way even as raw turns get trimmed, the important stuff survives.

Chris: So the trim-and-persist pattern is really about keeping semantic density high inside the window, not just managing bytes.

Koko: Well put. Every token in the window should be earning its place.

Koko: Second scenario. One system, two guards. A per-user rate limiter smooths load. An admin gate protects an expensive billed endpoint — you need a secret set in the environment to get through. One day the backing store behind both of them hiccups. Walk me through each guard.

Chris: Rate limiter first. Its job is load-smoothing, not security. If the store is unreachable and you can't check the counter, failing closed means real users get locked out over a transient glitch — that's the wrong tradeoff. It should fail open and let traffic through.

Koko: Good. Same hiccup, now the admin gate.

Chris: Opposite call. The admin gate is protecting a billed endpoint. If the secret is unset or unreadable, the correct move is to refuse everything. A broken deploy is vastly better than an open door burning credits.

Koko: Exactly right. And the trap the exam loves here is a tidy blanket rule: 'secure by default — everything fails closed.' It sounds principled. Why is it wrong?

Chris: Because it ignores what each guard is actually protecting. Applying it to the rate limiter would lock out legitimate users for no security gain at all.

Koko: The instinct the exam rewards is: match the failure direction to the guard's job. Same infrastructure failure, two opposite correct behaviors, decided entirely by purpose. And there's a broader principle hiding in this scenario — what does it connect to in cost control?

Chris: Meter and persist spend incrementally. If a crash happens mid-run, you don't want to lose all your cost tracking. You write it as you go, so a failure doesn't erase what you've already accounted for.

Koko: Right. A crash mid-run shouldn't reset the meter to zero. Persist incrementally, so the failure is bounded — in both the reliability sense and the financial one.

Koko: Alright, let's nail the lid on Domain Five. Eight instincts — let's run them fast. First: pin load-bearing context into the message stream and bound your history independently. Never rely on a bigger window to save you.

Chris: Right — the window is a physical ceiling, not a strategy. If the critical stuff isn't explicitly anchored, it gets pushed out as the conversation grows.

Koko: Exactly. Second: trim stale tool output down to the bare fact, use a scratchpad for intermediate reasoning, and persist state across turns. The trap there is keeping the full raw response in the stream every time — it bloats fast.

Chris: Third one is the one I keep wanting to double-check: the recurring generator needs a memory of its recent output to stay fresh, not a lower temperature. The option next to it on the exam will say 'reduce temperature to increase variety.'

Koko: And that's exactly the wrong lever. Temperature changes the randomness of each individual generation — it doesn't know what you already generated last week. Memory does. Good. Fourth: provenance travels with every claim — source and timestamp — and when sources conflict, you surface the conflict with attribution. You do not silently average them.

Chris: Silent averaging is what makes the output look confident when it shouldn't be. The exam will frame that as 'smoother for the user' — and it is, right up until someone acts on a number that was quietly split between two contradictory sources.

Koko: Perfect framing. Fifth: propagate a structured, typed error and degrade to a partial result. You are not crashing silently, and you are not halting entirely — you return what you can and you say what failed.

Chris: Sixth is the retry stack: backoff with jitter, cap your attempts, throttle concurrency, and fail fast on permanent errors. The discriminator I want to make sure I have — what exactly separates a transient from a permanent error in terms of handling?

Koko: Great precision. Transient errors — rate limits, timeouts, brief unavailability — can resolve on their own, so you retry with backoff and jitter. Permanent errors — a bad credential, a malformed request, a resource that simply doesn't exist — will not get better with waiting. Retrying them wastes cycles and can make things worse. Fail fast, surface it.

Chris: So retrying a four-oh-one is just burning time. Got it. Seventh: match the guard's failure direction to its job. Availability fails open so users aren't locked out. Authorization fails closed so access isn't accidentally granted.

Koko: And meter spend incrementally — don't commit everything upfront. The trap is treating all guards the same, defaulting everything either open or closed without asking which job that guard is actually doing.

Chris: Eighth: escalate on honest low confidence with a structured handoff, and sample across segments to catch quiet drift before it becomes a visible problem.

Koko: The through-line tying all eight together: assume interruption, re-triggering, and length — and design so none of it hurts. When two answers compete on the exam, choose the one that bends gracefully under partial failure over the one that assumes everything goes right. This is the smallest domain by weight, but these are the habits that keep the other four from cascading into an outage. Keep sharpening with the flashcards and the quiz at KokoAI Academy on koko-knows dot A-I. That's the full five-domain tour. Go pass that exam.

Koko: Alright, last stretch. Let's lock in the specifics the exam actually names — the ones where one wrong word costs you the point.

Chris: The one I keep coming back to is lost-in-the-middle. The model reliably uses what's at the start and end of a long input, but material buried in the middle gets dropped.

Koko: Exactly. Think of it like a sandwich — the bread gets remembered, the filling disappears. So the fix is: lead with a key-findings summary, and keep a persistent case facts block — amounts, dates, order numbers — anchored outside the summarized history in every single prompt.

Chris: And that's specifically because progressive summarization quietly converts precise numbers into vague prose. So the case facts block stays verbatim, not summarized.

Koko: That's it. The trap is assuming summarization is lossless. It isn't. Numbers are the first casualty. Now — escalation triggers. The exam names three. What are they?

Chris: The customer explicitly asks for a human, the policy has a gap or is silent on the situation, or the agent genuinely can't make progress.

Koko: Perfect. And here's the one that catches people: the exam will offer sentiment or the agent's self-reported confidence as a fourth option. The trap is treating those as reliable proxies for complexity. They're not. Don't escalate on vibes.

Chris: Right — a customer can sound calm and still have a genuinely hard edge case. Sentiment tells you nothing about actual complexity.

Koko: Exactly. Same instinct applies to ambiguous lookups: when a search returns multiple matches, ask for another identifier. Never guess.

Chris: And on failures — propagate structured error context. Failure type, the attempted query, partial results, alternatives. The two things you never do: silently mark a failure as success, or terminate the whole run because one subagent hit an error.

Koko: Those two are the exam's favorite wrong answers in that neighborhood. Now, context degradation in big explorations — four tools the exam names.

Chris: Scratchpad files, subagent delegation, structured state manifests for crash recovery, and slash-compact.

Koko: Good. Manifests are specifically for crash recovery — that detail shows up in questions. And on quality measurement — what's the instinct?

Chris: Stratified random sampling plus field-level confidence scores, because a ninety-seven percent aggregate can hide failures concentrated in one document type.

Koko: The trap is trusting the aggregate. A great overall number can mask a document type that's failing completely. Disaggregate, always. Last one — synthesis. What has to survive through the whole pipeline?

Chris: Claim-source mappings with dates. And when sources conflict, annotate them — don't average them. Averaging destroys the disagreement, which is usually the most important information.

Koko: That's the whole domain in one sentence, honestly. Alright — the handful of instincts to carry into the exam: anchor critical facts outside summarization, escalate on explicit triggers not on sentiment, propagate structured errors and never fail silently, disaggregate your quality metrics, and preserve the source mapping all the way through synthesis. That's Domain Five.

Chris: Clean list. I feel like the through-line is: be explicit. Explicit triggers, explicit error context, explicit source attribution.

Koko: That's a sharp way to remember it. Keep practicing with the flashcards and the full domain quiz over at KokoAI Academy — koko knows dot A I. You have put in the work. Go get that certification.