Prompt Engineering & Structured Output

Koko: Here is the single thing that unlocks Domain Four: the right answer is almost always the more constrained one. Not the bigger lever. The tighter specification. That one reflex will carry you through about a fifth of the whole exam.

Chris: A fifth of the exam is a serious chunk. And when you say constrained — you mean things like giving the model a specific role, spelling out exactly what the output has to look like, forbidding it from making things up?

Koko: Exactly that. Role, measurable criteria, output schema, grounding in source material, verification in code. Those are your tools. The domain is really about two things in combination: how you talk to the model, and how you make its output usable by a program downstream. Both problems have the same solution — sharper specification.

Chris: So the analogy I keep coming back to is a contractor. A vague brief gets you a vague building. You don't fix that by hiring a fancier contractor. You fix it by writing a better spec.

Koko: That's it exactly. And the exam is specifically designed to offer you the fancier contractor every single time. That is the trap: the wrong answers sound like good engineering. They feel decisive and technical. Raise the temperature. Use the largest available model. Fine-tune on your dataset. Run it three times and take the majority vote. Each of those sounds like something a serious engineer would say.

Chris: And the option sitting right next to those will be something like — cast the model in a role and add a schema constraint. Which can sound almost too simple by comparison.

Koko: Right, and that contrast is deliberate. The exam is testing whether you reach for the dial or for the spec. Most candidates feel the pull of the dial — it feels like you're doing something. The spec feels like paperwork.

Chris: So the instinct to drill in is: when the output is wrong or unpredictable, the fix is a sharper prompt, not a bigger model parameter.

Koko: Constrain, don't coax. Write that somewhere. Because every major topic in this domain — structuring prompts, getting clean output, running batch jobs reliably, extracting data from documents — they all come back to that same move. You're going to see 'fine-tune it' as an option repeatedly, and almost every time it's a distractor.

Chris: What's the tell that makes fine-tuning actually the right answer? Because it's presumably correct sometimes.

Koko: Good question, and we'll get into it properly later — but the short version is: if you haven't exhausted prompt-level constraints first, fine-tuning is premature. The exam will give you scenarios where prompting hasn't been tried seriously yet. In those, fine-tuning is always the trap. The moment you see a question that's really about 'the model isn't following my instructions,' your first move is sharper instructions, not a training run.

Chris: So the burden of proof is on escaping prompt engineering, not on using it.

Koko: Perfectly put. Constrain first. Verify in code. Only escalate when you've genuinely hit the ceiling of specification. That's the instinct this domain rewards, and it's the lens for everything that follows.

Koko: Alright, system prompts. Think of it as briefing a temp worker before they touch a single task. You're not saying 'please write some stuff.' You're saying 'today you are the editor of a financial brief — concise, neutral, no invented figures.' That framing changes how they show up for every task that follows.

Chris: Right, so the system prompt is doing the heavy lifting for consistency — role, voice, standing rules — all set once, and every single user turn inherits that framing automatically.

Koko: Exactly. So a daily-summary function, for instance, opens with a role, pins the voice — direct, neutral, numbers over adjectives — and drops in a non-negotiable output rule. Thousands of runs later, that behavior is still locked in because the instructions live in the right place.

Chris: And that's the exam instinct — durable behavior belongs in the system prompt, not repeated in every user message.

Koko: Nailed it. The trap here is thinking you can get reliability through encouragement — 'add more warmth,' 'raise the temperature for a nicer tone.' That's vague coaching. A creativity dial doesn't replace a clear role and explicit constraints.

Koko: Next one is closely related. Never tell the model 'pick the best.' Define what best means — ranked, measurable criteria, plus an explicit avoid list.

Chris: The rubric analogy. The difference between telling an intern 'use your judgment' and handing them a scored checklist. 'Use your judgment' gives you a different answer every time.

Koko: Exactly. A content selector instead of 'choose the best ten' gets five numbered, prioritized criteria — a market-moving signal outranks coverage breadth — plus an avoid list. A summarizer's fields are specified as measurable acceptance criteria: exactly three items, sixty to one hundred words.

Chris: So the instinct is: when a selector or classifier gives inconsistent results run to run, the problem isn't the model, it's the prompt. Encode the decision rubric — ordered, specific — don't rely on the model's implicit sense of what matters.

Koko: That's it. And the trap is treating the symptom instead of the cause. The options sitting next to the right answer on the exam will say 'raise the temperature,' 'run it three times and take the majority vote,' or 'swap in a bigger model.' All of those avoid the real fix.

Chris: Which is a sharper rubric. Those other moves are expensive workarounds for an under-specified prompt.

Koko: Last piece for this module — few-shot and reliable JSON. Instead of describing the output shape in prose, show the model a filled-in sample form. Showing beats telling.

Chris: And the examples you pick matter a lot. You want the tricky cases covered — the nullable field, the none-of-the-above scenario, the ambiguous input and how you want it resolved — because the examples are the specification.

Koko: Exactly. And then for paths that still read free text — you layer on a tolerant parser. It tries increasingly forgiving recovery strategies rather than throwing away a nearly-perfect answer over one stray character. If the response arrives with a coffee stain, rescue the legible content, don't bin the page.

Chris: So the instinct is: inconsistent format or ambiguity, reach for few-shot examples first, then a recovery parser. Before you do anything slower or more expensive.

Koko: Right. The trap here is two-headed. Most candidates reach for fine-tuning as the first move — it's expensive, it's slow, and few-shot is usually enough. The second trap is retrying the whole call when the format is slightly off. You already paid for that answer. Parse it tolerantly instead of re-paying for it.

Chris: So prevention through examples, recovery through a tolerant parser, and fine-tuning only if you've genuinely exhausted the cheaper options.

Koko: That's the stack. And it mirrors the system prompt logic — solve it in the prompt before you reach for infrastructure.

Koko: So let's talk about output contracts, because this is where a lot of architects draw a false line of safety. If your prompt says 'reply in JSON,' that is a polite request. The model will usually comply. Usually.

Chris: Right, and the exam distinguishes that from actually enforcing a shape at the platform level — structured outputs with a JSON schema, or strict mode on tool definitions.

Koko: Exactly. Think of it like the difference between 'write your answer on a napkin' and 'fill in this official form with these exact boxes.' The form doesn't let you skip the required fields.

Chris: And the two routes serve different consumers, right? Structured outputs constrain the model's direct answer, while strict tool use constrains the arguments of a tool call.

Koko: That's the discriminator. Pick the contract the consumer actually needs. A PDF extractor classifying findings into a fixed set of categories — that's an enum field in a structured output schema. Required fields get the required constraint.

Chris: And before trusting any of it, you inspect the completion state — confirm it actually completed, not a refusal, not truncated.

Koko: Yes. Validate completion first, then validate the business facts. Now, the trap here is two-part. First: adding 'return valid JSON' to the prompt feels equivalent, but it's strictly weaker — a prompt instruction is advisory, an enforced schema is validated by the platform. Second: a tool definition without strict mode is not a universal schema guarantee.

Chris: And there's the assistant prefill trick — starting the reply with a brace to nudge JSON output. That's rejected outright on Claude four point six and later.

Koko: Right, it's not a fallback anymore, it's an error. The exam will offer it as an option. Don't take it.

Koko: Next: grounding. The core idea is answer only from the source I gave you, and if it's not there, don't make it up.

Chris: And it's two layers — you instruct it in the prompt, 'use only this context, return empty rather than pad,' and then you verify in code. Check that any ids or numbers the model returned actually exist in your ground truth before saving anything.

Koko: Perfect. Open-book test where you also check the student's citations against the actual book. Now here's where the exam gets sharp. What's the instinct when someone says they want repeatable output?

Chris: A precise specification — a schema the platform enforces, plus code-side validation. Not a sampling dial.

Koko: Because temperature is gone. Removed in the API version dated October twenty-first, twenty twenty-six, already rejected on Claude four point seven and later when set to a non-default value.

Chris: So any exam option that says 'set temperature to zero to reduce hallucination' is wrong on two counts — it's not available, and low temperature never stopped the model from confidently inventing things anyway.

Koko: That's the trap. Variability and truthfulness are different problems. A deterministic model still hallucinates. And the adjacent trap: 'increase the length cap to make the answer better.' The length cap is a cost, latency, and truncation guardrail. Set it comfortably above expected output and leave it alone — it's not a quality dial.

Koko: Last piece: when a task is genuinely hard, split it. Extract, then classify, then draft — each step hands a clean result to the next.

Chris: Assembly line with a quality gate at each station. The alternative is one worker building the whole thing from memory, which is hard to debug when something goes wrong.

Koko: And when something does go wrong, you don't just re-run. You validate the output, find the specific failure, and feed that failure back for a targeted retry — capped at a fixed number of attempts.

Chris: The cap matters. If you retry unbounded, you're just spinning. And if you retry without feeding back what failed, you're giving the model no new information.

Koko: The trap the exam sets here is 'one giant prompt handles everything' — it's tempting because it feels simpler. But you can't isolate which step broke. One job per step, validate at each gate, retry with the specific error, and bound your attempts. That's the instinct.

Koko: Alright, domain four gives about twenty percent of the exam to prompt engineering and structured output, and a chunk of that is pure economics — how do you not blow your compute budget while still getting good answers. Three tools: model selection, message batches, and extended thinking. Let's take them one at a time.

Chris: Model selection first — and I think the core move is tiering. You don't send everything to your most expensive model.

Koko: Exactly. Think of it like a hospital triage. A cheap nurse takes everyone's temperature before the expensive specialist sees only the flagged patients. In a pipeline that's: cheap, fast model scores or classifies every incoming item, small output budget because a score is just a few numbers, and the expensive model runs only on the survivors.

Chris: And there's a second axis beyond volume — whether the model is grounded. If you've retrieved the facts and handed them in context, a mid-tier model can synthesize the answer. If there's no retrieved context, the model is drawing on its own knowledge, so you escalate to the strongest one.

Koko: That's the open-book versus closed-book distinction. Open book — you handed the student all the relevant pages — a competent student passes. Closed book — you need your single smartest test-taker. The exam rewards that instinct: cheap model for high-volume classification and filtering, expensive model reserved for low-volume synthesis on survivors or for ungrounded answers.

Chris: What's the trap the exam is trying to catch here?

Koko: Two traps, actually. The first is 'use the biggest model everywhere for the best quality' — that's the cost blow-up. The second is 'one mid-tier model for everything' — that forfeits both the cheap-filter savings on the front end and the escalation safety net on the back end. Neither extreme is right.

Chris: Got it. Now message batches — the way I'd frame it is you're deliberately trading latency for money.

Koko: Perfect framing. Dropping off a mountain of film to be developed overnight for a fraction of the one-hour-photo price. Batches run asynchronously — results come back within a processing window measured in hours — at roughly half the cost. Each request carries a correlation ID so you can match every result back to its input, and items succeed or fail independently.

Chris: So the discriminator is whether anyone is waiting. Classifying two hundred thousand archived documents overnight is a great batch job. A user sitting in a chat waiting for a reply is not.

Koko: That's exactly the instinct: human waiting means synchronous, nobody waiting plus high volume means batch. The trap the exam plants is batching an interactive request to save money. Batches have an hours-long window — completely useless when someone is staring at a loading spinner.

Chris: Extended thinking is the third one. On current models it's configured as adaptive thinking with an effort level rather than a fixed token budget — the model reasons at length before it answers.

Koko: Right, and the analogy there is showing your work. Worth every minute on a proof, completely wasteful on two plus two. A tricky multi-constraint planning problem justifies the extra tokens and latency. A straightforward classification does not.

Chris: The option sitting next to it on the exam will say 'enable maximum reasoning everywhere to be safe' — because more reasoning sounds like it can only help.

Koko: And that's the trap. You pay tokens and latency for problems that never needed it. The instinct is: match reasoning depth to task difficulty. And honestly that's the thread running through all three of these. Model size, batching strategy, thinking depth — every choice is about matching the compute to the actual difficulty in front of you. That's the single instinct the exam is testing across this whole cost-and-compute domain.

Koko: Let's run some exam scenarios, because this domain loves to test whether you can diagnose the real problem before you reach for a fix. First one: a team builds a feature that asks the model to choose the best ten headlines. Every run it returns a different ten. The team can't trust it. What's the exam going to offer as fixes?

Chris: The options sitting next to the right answer will be: raise the temperature, run it three times and take a majority vote, or upgrade to a bigger model.

Koko: And none of those touch the actual problem. Why not?

Chris: Because the model isn't broken — the prompt is. 'Best' is just an adjective. It means something different every run because nothing pins it down.

Koko: Exactly. Think of it like asking a panel of judges to score films, but you never gave them the rubric. Every judge invents their own criteria, so you get different winners every night. The fix isn't a better judge — it's writing the rubric down. In prompt terms that means explicit, ordered criteria: clarity first, then novelty, then engagement, whatever the team actually cares about. Plus a 'do not' list — no clickbait phrasing, no duplicate angles. Now 'best' is defined and applied the same way every run.

Chris: And if the output also has to be machine-readable, you layer a JSON schema on top so you're getting typed fields, not just hoping the model formats it correctly.

Koko: Right. The instinct the exam rewards: under-specified prompt, under-specified output. Fix both. Criteria tighten the selection; a schema tightens the shape.

Koko: Second scenario. A PDF extractor mostly works, but occasionally it invents a value for a field that wasn't in the document, and once in a while its JSON won't even parse. Two problems. Where do you start?

Chris: Separate them. Malformed JSON is a different failure mode from invented values. For the malformed JSON, you use structured outputs — a JSON schema the platform enforces — and you check that the response actually completed before you try to parse it. A prompt that just says 'return JSON' is much weaker than a schema the platform holds the model to.

Koko: That's it. Completion check matters too — a truncated or refused response will break your parser just as badly as bad formatting. What about the hallucinated values?

Chris: Ground it. The instruction should say: use only the provided text, and if a field is absent, leave it null. And then you verify in code — any value that doesn't trace back to the source document gets dropped.

Koko: That's the two-layer fix: grounding in the prompt, validation on your side. And I'll name the trap here before we move on — the exam will offer 'set temperature to zero so it stops making things up.' That sounds intuitive.

Chris: Low temperature makes the output repeatable, not truthful. The model will confidently invent the same wrong value every single run. And on current models you can't even set temperature anymore, so it's a double dead end.

Koko: The instinct the exam rewards: hallucination is a grounding problem, not a randomness problem. You fix it by constraining what the model is allowed to draw from, then verifying what it drew. Temperature is a red herring.

Chris: So across both scenarios the pattern is the same — diagnose first, then match the tool to the actual failure mode.

Koko: Exactly. Under-specified prompt, wrong model of the problem, wrong lever. The exam is testing whether you can tell the difference.

Koko: Alright, let's lock this in before you walk into the room. Seven instincts. We're going fast. First and most important: constrain, don't coax. When the tempting answer is a dial — temperature, a bigger model, fine-tuning — suspect it. The real fix is almost always a sharper specification.

Chris: That one keeps coming up. The exam is basically designed around that instinct — the plausible-sounding knob versus the specific constraint.

Koko: Exactly. And the knob always sounds reasonable, which is what makes it dangerous. Second instinct: durable role, voice, and rules live in the system prompt. Not the user turn, not scattered across the conversation — the system prompt.

Chris: Because it's the one place that persists and stays out of reach of the user's input.

Koko: Right. Third: replace adjectives with explicit, ordered, measurable criteria plus a do-not list. 'Be concise' is an adjective. 'Answer in three sentences or fewer, do not include caveats' is a constraint.

Chris: The do-not list is the part I used to underweight. It's doing real work.

Koko: It closes the gap that a positive instruction leaves open. Fourth instinct: enforce structure with a schema the platform validates — structured outputs, strict tool inputs, enums, required fields. And always check that the response completed before you trust it.

Chris: So the trap there is parsing hopeful JSON — just eyeballing the output and assuming it's well-formed.

Koko: The trap is exactly that. The exam will offer 'parse the output downstream' as a perfectly reasonable option, and the right answer is schema validation at the source. Fifth: ground the model in its source material, tell it to abstain rather than pad when it doesn't know, and verify factual claims against ground truth in code, not by asking the model to check itself.

Chris: Abstain rather than pad — that's the retrieval-augmented setup instinct. If the answer isn't in the context, say so.

Koko: Exactly. Sixth: show examples before you reach for fine-tuning, and chain single-purpose steps with a bounded validation-retry loop — not one giant prompt trying to do everything.

Chris: Most candidates reach for fine-tuning the moment few-shot examples feel like a lot of work. But the exam almost always rewards the example-first move.

Koko: Because fine-tuning is a knob. Seventh: match compute to difficulty. Cheap model for volume, strong model for synthesis, batch for non-urgent bulk, extended thinking only where it actually pays off.

Chris: And the option next to the right answer will usually say 'use extended thinking for all requests' — as if more compute is always safer.

Koko: More compute is a dial. Matching compute to difficulty is a constraint. That's the through-line for the whole domain. Almost every wrong answer is a plausible-sounding knob. Almost every right answer pins the model down. When two options are competing, pick the one that specifies, not the one that nudges. Carry 'constrain, don't coax' into the exam and you'll recognize the traps on sight.

Chris: Seven instincts, one sentence. I'll take it.

Koko: Keep sharpening with the flashcards and the quiz at KokoAI Academy — koko knows dot A I. You're ready.

Koko: Alright, final module — let's lock in the things that actually show up on exam day and bite people. We're going to move fast because you know this material. Think of this as the last walkthrough before you close the book.

Chris: Good. The one I want to nail down first is the reliability piece around criteria. The notes are pretty emphatic that vague hedges don't work.

Koko: Right. 'Be conservative' or 'only flag high-confidence cases' — those sound reasonable but they're not instructions, they're vibes. Explicit criteria define exactly what to flag versus skip. The exam instinct is: if you can't draw a line a junior engineer could follow, you haven't written a criterion.

Chris: And the trap is that the answer choice using 'high-confidence' sounds professional, so people pick it over the one that specifies the actual threshold.

Koko: Exactly. Now, few-shot examples — what's the key word the exam is testing there?

Chris: Generalize. Two to four examples let the model generalize to new cases, not match fixed ones. So few-shot is the right tool for ambiguous cases and format consistency.

Koko: That's it. Now the big structured-output distinction. Structured outputs and strict tool schemas eliminate JSON syntax errors — you never have to parse free text. But they do not prevent semantic errors.

Chris: Right, so line items that don't sum to the total slip right through. The pattern is to extract a stated total and a calculated total and flag mismatches yourself.

Koko: And before you even get to semantic validation, you check for refusal or truncation. The trap here is assuming that because the schema is enforced, the content is correct. Structure is syntax, not meaning.

Chris: tool_choice is the other one I want to be precise on. Auto lets the model return plain text. Any requires some tool call. Named selection requires a specific tool — but some current models reject any and named selection outright.

Koko: So on those models you use auto with a strict tool definition and then verify the tool request actually arrived. Good. Schema design — two things.

Chris: Nullable or optional fields so absence is expressible instead of fabricated. And an enum with an 'other' plus a detail field for categories that might grow.

Koko: Perfect. One more wrinkle on validation-retry: appending the specific error works for self-correction, but retries cannot help when the information simply isn't in the source. That's the discriminator — missing from the prompt versus malformed output. Two totally different problems.

Chris: And the Message Batches API — about fifty percent cheaper, up to a twenty-four-hour window, no latency guarantee. Great for overnight batch work, wrong for anything blocking. And each batched item is a single Messages call, so a multi-step agentic loop stays synchronous.

Koko: The trap there is assuming you can run a tool loop inside one batched request. You can include prior turns and tool definitions, but the loop itself is yours to manage. And custom_id is how you correlate results so you resubmit only the failures.

Chris: Okay — the instincts I'm carrying out of this whole domain: explicit over vague, validate meaning not just structure, match the API feature to the latency requirement, and design schemas that express absence so the model doesn't fabricate.

Koko: That's the whole domain in four lines. You're ready. Keep drilling with the flashcards and the practice quiz at KokoAI Academy — koko knows dot A I — and walk into that exam knowing you did the work.