Prompt Engineering & Structured Output

Koko: Here is the trap that breaks most Domain Three questions, and I want to name it before we do anything else. You are reading a scenario — a pipeline is returning inconsistent JSON, or a summarizer keeps hallucinating details that were not in the source — and the answer choices look like this: raise the temperature, use a bigger model, fine-tune it, or run it three times and take the majority vote. Every one of those feels like good engineering. They sound decisive. They have a technical ring to them.

Sam: I mean, if output is inconsistent, more model capacity seems like the obvious lever to pull.

Koko: Exactly what the exam is counting on you to think. And that is almost never the right answer. Domain Three is about two things: how you talk to the model, and how you make its output usable by a program. Twenty percent of the exam lives here. The instinct the exam is actually rewarding is the opposite of reaching for a bigger dial.

Sam: So what is the instinct?

Koko: Constrain, don't coax. Think of it like a job posting. If you write a vague job posting — 'looking for a good communicator who works hard' — you get a pile of applications you cannot evaluate. But if you write 'must produce a two-paragraph summary, cite only sentences from the provided document, return a JSON object with exactly these three keys' — suddenly you can screen automatically. The model is the applicant. A sharper specification beats a bigger budget every time.

Sam: Okay, so when you say constrain, what does that actually look like in practice?

Koko: Four moves. First, cast the model in a role — give it a persona with a clear scope, not just 'you are a helpful assistant.' Second, replace adjectives with measurable criteria. Not 'be concise,' but 'respond in at most three sentences.' Third, force the output shape with a tool schema or an explicit format definition instead of hoping the model produces clean JSON. And fourth, ground the answer in source material and explicitly forbid invention.

Sam: That last one — forbid invention — that is just telling it not to make things up?

Koko: Right, and you have to say it out loud in the prompt. Something like 'only use information present in the document below, do not add details from outside it.' Vague instructions leave gaps, and the model fills gaps. Explicit constraints close them.

Sam: So the trap the exam sets is making the technical-sounding options feel like the responsible engineering choice.

Koko: Perfectly put. And there is a fifth move I should add — verify the result in code. Do not rely on the model to self-report whether it followed the format. Parse the output programmatically and fail loudly if it does not match the schema. That is the architectural close on the loop.

Sam: So the whole domain is really one reflex: when something is going wrong, the fix is a sharper specification, not a bigger dial.

Koko: That is the one instinct. Write it on your hand if you have to. Every module we cover builds out a different corner of it — roles, criteria, output schemas, grounding, verification — but the reflex underneath is always the same. Constrain, don't coax.

Koko: Alright, so before the model sees a single user message, you get one powerful lever: the system prompt. Think of it as the standing job description — you're briefing a temp worker before the day starts. Not 'please write some stuff.' More like: 'today you are the editor of a financial brief — concise, neutral, never invent.' That job title changes how someone shows up.

Sam: So the system prompt is separate from the actual task instructions I'm sending each time?

Koko: Exactly. Per-request data goes in the user turn — the article to summarize, the ticket to classify. The system prompt holds the durable stuff: the role, the voice, the standing rules. A daily-summary function might open its system prompt with the role, pin the voice — direct, neutral, numbers over adjectives — and lock in a non-negotiable output rule. Every later instruction inherits that framing automatically.

Sam: Okay, and if my outputs are inconsistent run to run, the instinct is to go fix the system prompt first?

Koko: That's the exam instinct, yes. Durable behavior lives in the system prompt. And here's the trap: if results feel off, someone will say 'add more adjectives' or 'raise the temperature for a nicer tone.' That's vague coaching. A creativity dial does not replace a clear role plus constraints.

Sam: Right — encouragement isn't a specification.

Koko: Exactly. Role plus rules in the system prompt. That's the lever.

Koko: Next idea builds on that. Never tell the model 'pick the best.' Define what best means — a ranked, measurable checklist, plus an explicit do-not list. Think of the difference between telling an intern 'use your judgment' versus handing them a scored rubric.

Sam: So instead of 'choose the best ten articles,' I'd give it five numbered criteria and say criterion one outranks criterion two?

Koko: Precisely. A market-moving signal outranks mere coverage breadth — that priority is written down, not implied. And a summarizer doesn't get 'be concise,' it gets 'exactly three items, sixty to one hundred words.' Measurable acceptance criteria.

Sam: Okay so if my classifier keeps giving different answers on the same input, the instinct is encode the rubric, not fiddle with the model.

Koko: Right. And the trap here is tempting: 'raise the temperature,' 'run it three times and take the majority vote,' 'use a bigger model.' All of those treat the symptom. The cause is an under-specified prompt. The fix is a sharper rubric.

Sam: So I'd just reach for the biggest model, right? That'd fix the inconsistency?

Koko: That's the classic trap answer. A bigger model given a vague question gives you a more confident vague answer. Specify first, scale later only if you still need it.

Koko: Last piece of this module: showing beats telling. Instead of describing your output format in prose, put a filled-in example right in the prompt. A sample form someone can mimic.

Sam: And the examples have to be chosen carefully — not just a happy path?

Koko: That's the key move. Your examples are the specification. Cover the tricky cases — the nullable field, the 'none of the above,' the ambiguous input and how you want it resolved. If you only show clean examples, the model guesses on edge cases.

Sam: What about when the output comes back almost right — like valid JSON except one stray character?

Koko: Don't bin the page over a coffee stain. Layer a tolerant parser that tries increasingly forgiving recovery strategies before giving up. You already paid for that answer — rescue the legible content.

Sam: So the trap is retrying the whole call when the answer was basically there?

Koko: That's one trap. The other is jumping straight to fine-tuning when format is inconsistent. Fine-tuning is expensive and slow. Few-shot examples are cheaper, immediate, and usually enough. Reach for the rubric and the examples first. Fine-tune only when you've genuinely exhausted prompt-level tools.

Sam: So the pattern across this whole module is: specify before you scale, show before you train, and recover before you retry.

Koko: That's a good way to carry it into the exam. System prompt for durable rules, explicit rubrics for consistent decisions, examples plus a tolerant parser for reliable format. Those are your three setup levers.

Koko: So we just covered how you shape a prompt — now let's talk about what happens when the output has to be machine-readable. And the first thing I want to kill is a habit that shows up everywhere: writing 'please return valid JSON' at the bottom of a prompt and calling it done.

Sam: That's basically what I'd do. It usually works, right?

Koko: Sometimes. But 'usually' is not an architecture. A prompt instruction is a polite request. The platform has no obligation to enforce it. What you want instead is a tool schema — you define a tool, you lock down its input schema with exact types, an enum for every categorical field, required for every must-have field, and then you force the model to answer by calling that tool and nothing else.

Sam: So the model can't just hand back a blob of text.

Koko: Exactly. Think of it this way: 'reply in JSON' is telling someone to write their answer on a napkin. A forced tool schema is handing them an official form — specific boxes, no blank space for improvisation — and saying you may give me nothing else. The platform validates that the inputs match the schema before the call even lands in your code.

Sam: Okay, so in practice — say I'm building a PDF extractor that categorizes findings.

Koko: Perfect example. You define a submit tool. Its schema says the category field is an enum — three or four allowed values, not a free string. The required fields are marked required. You force that tool. Your downstream code just reads the structured input directly — no parsing, no try-except around a JSON decode.

Sam: What about when there's also a web-search tool enabled? Can I still do this?

Koko: Good catch — that's a real detail. When a server-side tool like web search is active, prefilling an opening brace gets rejected. So the pattern there is a dedicated 'submit answer' custom tool. That's your escape hatch. The instinct the exam rewards: for any output with a fixed set of allowed values and zero tolerance for malformed structure, define the schema, use the enum, mark your required fields, force the tool. The tool call is the answer.

Sam: Alright, forced schemas handle structure. But what about hallucination — the model making up content that sounds plausible?

Koko: That's grounding. And it works in two layers. First, you instruct it in the prompt: use only this context, do not invent percentages, return empty rather than pad. Second, you verify it in code: check any IDs, numbers, or enums the model returned against your actual ground truth and drop anything invented before it ever gets saved.

Sam: So it's like an open-book test — but you also check the citations against the actual book.

Koko: That's exactly the analogy. Both layers matter. The prompt layer is the instruction; the code layer is the enforcement.

Sam: Okay, and this is where I want to say — just set temperature to zero, and it stops hallucinating, right?

Koko: That's the trap. Temperature controls variability, not truthfulness. A low-temperature model will confidently invent the same wrong record ID every single time. Very consistently wrong. Lower temperature toward zero for deterministic extraction — for repeatability — but do not confuse that with accuracy.

Sam: And the output-length cap — bigger is better?

Koko: No. The length cap is a cost, latency, and truncation guardrail. Set it comfortably above your expected output so you don't truncate, but it is not a quality dial. Making it huge doesn't make the answer better.

Sam: So you've got good structure, you've got grounding — but what about tasks that are just genuinely complex? Is there a way to make the whole pipeline more reliable?

Koko: Yes, and the answer is: stop asking one prompt to do everything. Chain it. Split the task into single-purpose steps — extract, then classify, then draft — each one handing a clean result to the next.

Sam: My instinct is just to write one giant prompt that covers all of it.

Koko: And when it fails, where do you look? The whole thing is one black box. A chain gives you a quality gate at each station. Think of an assembly line — a car built by one person from memory versus a line where every station checks the previous station's work.

Sam: And when a step fails, you retry it?

Koko: You retry it with the specific failure fed back — not a blind re-run. If your schema check on the draft catches an error, you tell the model exactly what went wrong. And you cap the retries. A bounded max-attempts limit is non-negotiable, or one bad input spins the whole pipeline.

Sam: So 'retry on failure' by itself isn't enough — it needs to carry the error forward.

Koko: Right. Blindly rerunning adds no new information. The instinct: one job per step, validate at each gate, feed the specific failure back, and cap your attempts. That's what makes a complex pipeline actually debuggable.

Koko: So let's talk about money, because the exam cares a lot about cost and compute — not just correctness. And the first instinct to burn in is this: cheap model filters, expensive model synthesizes.

Sam: Okay, but isn't the safest move just to use the biggest model everywhere? Best quality, fewest regrets.

Koko: That's the cost blow-up trap, and the exam will absolutely bait you with it. Think about it like a hospital triage system. A cheap nurse takes everyone's temperature. The expensive specialist only sees the flagged patients. You don't pay a specialist's rate to check vitals.

Sam: So you're running a cheap model first to throw out the noise, then escalating the interesting stuff.

Koko: Exactly. Picture a pipeline that scores every incoming article. The cheap model does that scoring — tiny output budget, just a few numbers. Only the articles that pass the threshold get polished summaries, and that's where the stronger model runs. High volume goes cheap. Low-volume survivors get the good stuff.

Sam: What about when the model already has the facts handed to it — like retrieved context?

Koko: Great distinction. Think open-book versus closed-book exam. If the model has the facts in front of it, a competent mid-tier model can handle that — it just has to read and organize. But if you're asking the model to answer from its own knowledge with nothing retrieved, you want your single smartest test-taker. Grounded answers can go cheaper. Ungrounded ones justify the expensive model.

Sam: And the other trap — one mid-tier model for everything?

Koko: That one forfeits both the cheap-filter savings on the front end and the escalation safety net on the back end. You're paying too much for triage and not enough for synthesis. Match the compute to the actual job.

Sam: Okay, what about when you just have an enormous pile of work to get through — like hundreds of thousands of documents?

Koko: That's where batching comes in. The key trade is explicit: you give up latency and get roughly half the cost. Requests run asynchronously, results come back within a window measured in hours — not seconds.

Sam: Hours. So this is a deliberate choice, not a performance trick.

Koko: Deliberate is exactly the word. It's like dropping off a mountain of film to be developed overnight. You get a fraction of the one-hour-photo price, but you're not getting anything back before morning. Classifying two hundred thousand archived documents overnight is a perfect batch job. A user sitting in a chat interface waiting for an answer is not.

Sam: What holds the results together when they come back? Like how do you know which answer goes with which input?

Koko: Each request carries a correlation id. That's what you use to match every result back to its input. And items succeed or fail independently — one bad request doesn't tank the whole batch.

Sam: So the trap here is batching something interactive to save money?

Koko: Right. Someone is waiting on the answer — you cannot batch that. An hours-long window is the whole deal. Human waiting means synchronous. Nobody waiting plus high volume means batch. That's the instinct.

Sam: And then there's extended thinking — I've seen that come up. What's the architectural angle on it?

Koko: Extended thinking lets the model reason at length before it answers. Extra tokens, extra latency, better results on genuinely hard multi-step problems. The key word is genuinely.

Sam: So showing your work on a proof is worth it, but not on two plus two.

Koko: That's exactly the analogy. A tricky multi-constraint planning question — turn it on. A straightforward classification — don't. You pay for it whether or not the problem needed it.

Sam: And I'm guessing the trap is turning it on everywhere just to be safe.

Koko: Maximum reasoning everywhere is paying tokens and latency for problems that never needed the help. The instinct is match reasoning depth to task difficulty. And actually, step back — that instinct runs through everything we've covered in this section. Model size, batching, thinking depth — all of it is the same move: match the compute to the actual difficulty in front of you. The exam rewards that instinct every time.

Sam: Match the compute to the difficulty. Got it. That's a clean through-line.

Koko: Alright, let's put everything we've covered to work. Real scenarios, exam-style. First one: a team builds a feature that asks the model to choose the best ten headlines from a batch. Every run they get a different ten. They can't trust it. What's your first instinct?

Sam: Honestly? Run it three times, take a vote. Or maybe upgrade to a bigger model — more capable, more consistent?

Koko: Both classic distractors. Here's the diagnosis: the results are inconsistent because the prompt is under-specified. The word best is an adjective, not a rubric.

Sam: So the model is just... inventing its own definition of best each time.

Koko: Every single run. Think of it like asking ten different judges to score a gymnastics routine without giving them the scoring sheet. You'll get ten different winners, and they're all technically doing their jobs.

Sam: So the fix is write down the scoring sheet.

Koko: Exactly. Explicit, ordered criteria — clarity first, then relevance, then click-worthiness, whatever the product actually needs — plus a do-not list so the model can't wander. Once best is defined on paper, it applies the same way every run.

Sam: And the voting approach just papers over the real problem.

Koko: Right, you'd just be averaging inconsistency. Now extend the scenario: the output also has to feed a downstream system. Machine-readable.

Sam: Then I'd ask for JSON in the prompt.

Koko: And that's the next trap. Asking for JSON is weaker than enforcing a schema. Force a tool schema so you get typed, validated fields instead of hoping the model formats it correctly. The instinct the exam rewards here: under-specified criteria plus missing schema enforcement — those are two separate problems, and they need two separate fixes.

Sam: Okay, second scenario?

Koko: A PDF extractor. Mostly works great. But occasionally it invents a value for a field that wasn't in the document. And once in a while the JSON comes out malformed and breaks the pipeline.

Sam: Two separate bugs.

Koko: Two separate bugs, two separate instincts. Let's take the malformed JSON first.

Sam: Force a tool schema.

Koko: Fast learner. The platform validates the shape before anything downstream ever sees it. A prompt asking for JSON is a polite request. A schema is enforcement.

Sam: Good. Now the invented values — I'm going to say it before you do: set temperature to zero so it stops making things up.

Koko: And that is the trap. Low temperature makes the model repeatable. It does not make it truthful. You can get the exact same hallucination every single run at temperature zero.

Sam: So what actually fixes it?

Koko: Two moves. First, ground it — instruct the model explicitly: use only the provided text, and if a field is absent, leave it null. Second, verify in code. Any value that can't be traced back to the source document gets dropped before it ever touches your database.

Sam: So grounding handles the model side, and code-side validation is the safety net.

Koko: Defense in depth. Neither one alone is enough. The exam instinct here is to see hallucination and schema failure as distinct failure modes that each need their own countermeasure — grounding plus validation for truth, schema enforcement for shape. Temperature is a distractor.

Sam: I feel like both scenarios are really testing whether you know to fix the prompt and the architecture, not just reach for a bigger lever.

Koko: That's exactly the pattern. The exam is going to offer you a model upgrade or a temperature knob, and the right answer is almost always: tighten the spec, enforce the schema, validate in code. Build a system that can't lie to itself.

Koko: Alright, we've covered the whole domain. Before you walk into that exam room, let's compress everything down to the instincts that actually matter.

Sam: Yes, please. Give me the short list.

Koko: Number one, and this is the through-line for the entire domain: constrain, don't coax. Whenever a question dangles a dial in front of you — turn up the temperature, grab a bigger model, jump to fine-tuning — treat that as a red flag. The right answer is almost always a sharper specification.

Sam: So the tempting option is usually the knob, and the right option is the constraint.

Koko: Exactly. Pin it down rather than nudge it along. That one sentence will eliminate wrong answers faster than anything else in this domain.

Sam: Okay. What's next?

Koko: Second: durable identity lives in the system prompt. Role, voice, rules — set them once, set them there, and they hold across the whole conversation.

Sam: Right, not scattered through the user turns.

Koko: Third: swap adjectives for criteria. Replace words like thorough or concise with explicit, ordered, measurable requirements, and pair them with a do-not list. Vague adjectives are basically coaxing. Specific criteria are constraining.

Sam: Adjectives are coaxing in disguise. I like that framing.

Koko: Fourth: when you need structured output, force it with a tool schema — enums, required fields — rather than asking nicely for JSON and then parsing whatever you get back. The tool call is the answer, not the text.

Sam: So hoping the model formats correctly is the trap, and the schema is the constraint.

Koko: You've got it. Fifth: ground the model in its source. Tell it to abstain rather than pad when the answer isn't there, and verify against ground truth in code, not by re-asking the model.

Sam: Don't let the model grade its own homework.

Koko: Never. Sixth: show examples before you reach for fine-tuning, and when you chain steps, keep each step single-purpose with a bounded validation-retry loop so errors don't just snowball downstream.

Sam: Examples first, fine-tuning last resort. Chain steps tightly.

Koko: And seventh: match compute to difficulty. Cheap model for volume, strong model for synthesis, batch for non-urgent bulk, extended thinking only where it genuinely pays off. Throwing your heaviest model at everything is not an architecture, it's a budget leak.

Sam: That one will save someone's bill as well as their exam score.

Koko: Both things can be true. Now here's the meta-instinct that ties all seven together. When two answer choices compete, ask yourself: which one pins the model down, and which one just gives it a nudge? The one that constrains is almost always right.

Sam: Constrain, don't coax. That really is the whole domain in four words.

Koko: Carry those four words in and you will recognize the traps on sight. Keep sharpening with the flashcards and the quiz over at KokoAI Academy on koko knows dot A I — they're built around exactly these instincts. You have done the work. Now go show them.