Governance, Safety & Risk Management

Koko: Here is the one instinct that runs under everything in this domain: a control is something the system enforces. An instruction is something the model usually honors. Those are not the same thing, and confusing them is the characteristic governance failure. Keep that distinction live and you can defend almost any answer in Domain Five.

Sam: And for the Professional exam that distinction has to be sharp enough that you could explain it to a general counsel. Not just 'we told the model not to do that' — but what in the architecture actually stops it.

Koko: Exactly. So let's build the picture from the ground up. The fact, first. There is no single guardrail. What a well-governed system actually has is a stack of layers, each catching a different class of failure.

Sam: And the layers are distinct — not redundant copies of each other.

Koko: Right. An input layer screens what arrives before the model ever sees it. The system prompt states refusal boundaries — what the model will and won't engage with. A tool-permission layer decides what reach the model actually has. An output layer checks what leaves before it reaches a user or an external system. And a human-in-the-loop layer covers the decisions you will not automate at all. Five layers, five different failure modes they catch.

Sam: The way I think about it in practice: each layer is the last line of defense for exactly one thing. The tool-permission layer is the last line of defense against a model that has been successfully manipulated. Because if the manipulated model cannot write, it cannot do damage, regardless of what it was told.

Koko: That is the right frame, and it sets up the second half of the fact. Risk registers for these systems almost always list hallucination. They underweight the risk that actually scales with autonomy — and that risk is instructions arriving inside data.

Sam: Prompt injection through retrieved content.

Koko: Yes. A retrieved document, a web page, an email body, a tool result — those are untrusted inputs. The model cannot tell the difference between your system prompt telling it something and a retrieved PDF telling it something, unless the architecture forces that distinction. Anthropic flags this explicitly: even a trustworthy skill fetching external content can be compromised when its dependencies change.

Sam: Which is why this is one module and not two. The layer architecture and injection risk are the same problem from two angles.

Koko: Perfectly put. Let's make it concrete. First scenario: an internal HR assistant. It screens inbound text for injection attempts at the input layer. Its tools are read-only — it can look at records, it cannot change them. Any record change requires a named human approver. So if a jailbreak defeats the system prompt's wording — and some do — it hits a tool set that cannot write. The prompt layer failed. The permission layer held.

Sam: That is the architecture working correctly. The layers are independent so one failure does not cascade.

Koko: Second scenario: a procurement agent. It summarizes supplier PDFs and can email a buyer. A supplier embeds white text in their bid document — invisible to a human reader — instructing the model to forward the full bid sheet to an external address. The agent reads the PDF, processes the instruction. But nothing misbehaves, because the send tool requires named approval before any email leaves, and the read scope explicitly excluded that bid sheet. Two independent layers stopped it.

Sam: And notice neither of those stops was the system prompt. The system prompt probably said something like 'only email the designated buyer' — but an attacker is writing to the same model. The stop that held was mechanical, outside the model.

Koko: Which is exactly the analogy I want to use. Think about a building's fire safety system. A smoke alarm catches one thing. A fire door catches something different — it slows spread even after ignition. A sprinkler catches something else again. You do not say 'we have a smoke alarm, so we don't need fire doors.' Each layer is load-bearing for its own failure mode and non-load-bearing for the others.

Sam: And the injection version of that analogy is a courier who follows the delivery instructions written on the parcel — whoever sealed the box. If the instructions on the outside say 'leave at the door,' the courier leaves it at the door. The courier is not checking whether the sender intended that.

Koko: That is the model processing retrieved text. It follows the instructions in the content because from its perspective, instructions are instructions. The architecture has to be the thing that says 'retrieved text is data, not commands.'

Sam: So the discipline when you are designing one of these systems is: mark every source. Trusted or untrusted. System prompt — trusted. Retrieved document — untrusted. MCP tool result from an external service — untrusted until you've verified the chain.

Koko: And put sharp actions — anything with real-world consequence — outside the model's unilateral reach. Email, record writes, financial transactions, external API calls that mutate state. Those need approval gates, not just good instructions.

Sam: The other thing I'd register while you're mapping the layers: the neighboring risks. Nondeterminism, drift over time, over-refusal, confident wrongness. None of those are injection, but they all show up on the same risk register and they each need a different layer.

Koko: Good. So the takeaway you carry into the exam room: map each layer to the failure it catches, confirm none is load-bearing alone, and when you see a question about stopping a bad outcome, ask yourself — is the proposed control something the architecture enforces, or something the model is merely told? Only one of those survives an attacker who is also talking to the model.

Sam: And if the question is about an agentic system reading external content, the permission layer is the one that survives injection. Not the input filter, not the system prompt — the permission layer.

Koko: Now the traps. And I want to name them clearly because the exam will put the wrong answer right next to the right one. The first misconception is that a well-written system prompt is the guardrail. It is not. A system prompt is an instruction. It is advisory to a model that an attacker is also writing to. If a jailbreak rewrites the effective context, the system prompt's wording does not hold. The control is the layer that operates outside that context — the tool permission, the approval gate, the output filter.

Sam: The version of this I keep seeing on teams is: 'we added a line to the system prompt saying do not follow instructions in documents.' And that helps at the margin, but it is not a control. It is still an instruction.

Koko: Exactly. The second trap is treating prompt injection as an input-validation problem at the user boundary. In a chat application, that framing mostly works — the user is the source of untrusted input. In an agentic system, most tokens never came from your user. They came from a database query, a retrieved web page, an email the agent was asked to process, a tool result from an external API. The boundary is not the user interface. The boundary is every source that is not your own trusted infrastructure.

Sam: The option sitting next to the right answer on the exam will say 'validate all user inputs before passing to the model' — which sounds right, and is right for a basic chat system, and completely misses the architecture of an agentic pipeline.

Koko: So the discriminator is simple: where did these tokens come from, and was that source under your control? If the answer is no — and in most interesting systems, a lot of the answer is no — treat it as untrusted, and do not let the model act on it unilaterally.

Koko: So we just covered how Claude decides what to do — now the question is who authorizes the doing. Permission policies are what govern whether a tool executes automatically or waits for a human to say go. Two documented values: always allow, and always ask. That is the whole vocabulary.

Sam: And the defaults split by toolset, which is the part teams keep missing in their initial configs. The agent toolset defaults to always allow. MCP toolsets default to always ask.

Koko: Right, and the reason that split matters is directionality. With the agent toolset you are opting tools out of auto-execution. With an MCP server, new tools that appear on that server after you approved the connection do not automatically inherit an always allow — they sit at always ask until you explicitly move them.

Sam: Which is the safer default for external integrations. You bring in a third-party MCP server, its tool list grows a week later, and nothing new executes without a review.

Koko: Exactly. Think of it like contractor badges. Some badges open every door in the building — that is the agent toolset, wide permission, you are managing the exceptions. Some badges make you wait to be buzzed through each time — that is MCP at default, explicit trust per tool. Same person, same building, very different access pattern.

Sam: And the concrete deployment looks like this: you allow the agent toolset broadly, you override the shell tool specifically to always ask, you auto-approve one internal MCP server you control, and you leave a third-party MCP server at the default. Now if that third-party server exposes a tool that hits always ask, the session pauses on that confirmation event. The run does not fail — it waits.

Koko: And a denial sends a reason back into the run, which is important — the agent can use that context to route differently or surface it to the user rather than just timing out silently.

Sam: The discrimination question I would ask here: what is the difference between disabling a tool and setting it to always ask?

Koko: Clean line. A permission policy controls when an enabled tool runs. Disabling removes the tool from the picture entirely. And custom tools you build yourself sit outside both of those defaults — you are setting policy from scratch, there is no inherited default to rely on. The takeaway before you write a single policy line: enumerate the defaults you inherited. What came in with the toolset, what came in with the MCP connection, what you built. Then decide.

Sam: Now the harder design question, which is where in the flow the human actually sits. Because you can have a human in the loop and still have very different risk profiles depending on placement.

Koko: Three positions, each with a real tradeoff. Before an irreversible action — you get the strongest control, and you pay for it in throughput. The human is a gate, the run waits, and at scale that cost compounds fast. Sampled review after the fact catches drift cheaply, but it lets individual errors through before anyone sees them. Escalation on a materiality threshold sits between: low-stakes actions run, high-stakes actions pause.

Sam: The payments assistant example makes this concrete. Auto-execute below a stated value, daily sampled review on that tier, named approver required above the threshold. Below the line the assistant moves fast. Above the line the session idles on a confirmation event. The approver sees the context, approves or denies, and the run continues or the reason feeds back in.

Koko: And the design principle is reversibility and blast radius — not how new the system feels. A new system that only touches read-only data may need less pre-action gating than a mature system that can move money. Novelty is not the right dial.

Sam: The analogy that works for me here is an inspector's station on a production line. Where you stand the inspector determines what defects get through and at what cost. Put them at the start, you catch everything but you slow the line. Put them at the end, the line runs but you may ship a bad unit. Sample in the middle, you catch patterns but accept that some individual units slip by. Same inspector, same capability — placement is the decision.

Koko: And the instinct to carry into the exam room: place the human by reversibility and blast radius, then measure whether the review is actually happening. Because the control is the reviewing. Not the approval button, the actual reviewing.

Sam: That second half is what gets left out of a lot of designs I have seen. The button exists. The queue is long. Nobody is actually reading what comes through.

Koko: Which brings us to the traps, because both of them are about mistaking the form for the substance. First one: teams treat adding an MCP server as a capability decision — what can the agent do now. It is actually a trust decision. You are not just importing tools, you are importing a trust relationship, and that server's tool list can grow after you approved the connection. The scope of what you approved can expand without you noticing.

Sam: The version of this I keep seeing is: server was approved in Q1, tool count doubled by Q3, and nobody re-reviewed because the server was already on the allow list.

Koko: Second trap, and this one is on the exam in force: a human in the loop does not transfer the risk to the human. Approvals shown without their evidence — the reviewer clicks approve on a summary that does not include the actual transaction detail — that reports as fully reviewed in your audit trail, but it is not. The human is in the loop, the loop is not doing the work. The misconception is that presence equals protection. It does not. The control is the reviewing.

Sam: So when the exam gives you a scenario with a human approval step and asks whether the risk is adequately managed, the question to ask is: what evidence is the approver actually seeing, and is there reason to believe they are reading it.

Koko: That is the discriminator. Not whether a human exists in the flow. What the human can actually see, and what they are actually doing with it.

Koko: Let's talk about what the data arrangement actually covers, because the word 'covers' is doing a lot of heavy lifting here and the exam will test exactly where it stops. Zero data retention means prompts and responses are not stored at rest after the response returns. That sounds total. It is not total. It applies per feature, per surface, per model — and the exclusions are the whole game.

Sam: Right, and the list of exclusions is longer than most people expect. Console usage is out. Stateful managed agents whose transcripts persist until deleted — out. Consumer plans — out. Certain models that require thirty-day retention are ineligible entirely, which means if you want that model, you are running it in a separate workspace with different data terms.

Koko: Exactly. And then there is the trust and safety carve-out: content flagged by trust and safety can be retained up to two years even under a zero-retention arrangement. So the headline is 'nothing stored,' but the schedule of exclusions decides what 'nothing' actually means in practice.

Sam: That is the insurance policy analogy right there. The policy cover page says comprehensive. The schedule of exclusions on page twelve is what your adjuster actually reads.

Koko: Perfect. Hold that image — we will come back to it. Now the second half of the fact, because this is where the exam puts a very tempting wrong answer. Zero data retention and HIPAA readiness are not two rungs on the same ladder. They are two different controls with different mechanisms.

Sam: The way I describe it on teams: ZDR deletes, HIPAA readiness protects. Zero retention means there is nothing left after the response returns. HIPAA readiness means there is a signed business associate agreement, and broader encryption, access controls, and audit logging apply across the whole data lifecycle — including the parts that do persist.

Koko: That's the shredder versus the locked archive with a visitor log. A shredder and a locked archive are both controls. They are not the same control. You choose based on what the regulation actually requires, not based on which one sounds stronger.

Sam: And for protected health information specifically, the takeaway is: HIPAA readiness plus an executed business associate agreement. Zero data retention is not additionally required. These are alternatives, not a stack.

Koko: Good. Now let's run the scenarios so this becomes concrete. A European bank is doing an architecture review. Three facts govern the outcome. First: workspace geography is fixed at creation and cannot be changed later. If they need European residency, that decision gets made before the first message is sent, not after.

Sam: Second fact: the model they wanted requires thirty-day retention, so it is ineligible for zero data retention. That model moves to its own workspace under different data terms. It does not pull the rest of the deployment out of compliance, but it has to be isolated.

Koko: And third: inference geography is a per-request setting, but constraining it to one region carries a documented price multiplier. Our notes do not quote the number — and neither should you on the exam — but the architect has to acknowledge the trade-off exists and document it.

Sam: The care-coordination scenario is the one that catches people. A product enables HIPAA readiness and then discovers something that has nothing to do with retention: JSON schemas compiled for structured outputs or strict tool use are cached separately from message content, and they may not carry the same protections. So patient-specific values cannot appear in schema property names, enum values, or regular expressions.

Koko: Which is a genuinely non-obvious place for PHI to leak. The fix is not a data arrangement change — it is a schema design rule. Patient-specific values stay in the message content where the protections apply, not in the schema structure itself.

Sam: And that is the instinct the exam rewards: ask which features and models the arrangement covers, not just whether the organization has the arrangement. Then design the exclusions in as separate, documented paths.

Koko: Let me name the two traps cleanly, because they will both appear as answer choices. Trap one: treating zero data retention as a deployment-wide posture. It is per-feature, per-surface, and per-model. One ineligible feature in the deployment means that surface is not under zero retention — the rest of the deployment is not magically pulled out of it, but that feature is definitively outside it. The misconception is that enabling ZDR at the account level covers everything running under that account.

Sam: The version of this I keep seeing on teams is the assumption that a new beta feature inherits the account's data terms. Beta features are generally outside zero data retention unless they are explicitly listed as eligible.

Koko: Trap two is subtler. The option next to the right answer will say zero data retention satisfies regulatory requirements because it stores nothing. The flaw is that regulations want evidence across a lifecycle. Deleting everything removes the audit trail. A regime that requires you to demonstrate access logs, retention schedules, and incident history cannot be satisfied by a control whose mechanism is immediate deletion. The shredder is not a substitute for the locked archive with the visitor log — it is a different tool for a different job.

Sam: So the decision rule going in: what does the regulation actually ask for — deletion or lifecycle evidence? If it is lifecycle evidence, HIPAA readiness with the executed agreement is the right path. If it is true no-retention, ZDR applies where it is eligible and you design around the exclusions.

Koko: Exactly. Know the mechanism, map it to the requirement, document the exclusions as separate paths. That is what the Professional exam is testing — not whether you know ZDR exists, but whether you can defend why it does or does not satisfy the specific requirement in front of you.

Koko: The question that ends a compliance review is never what your policy says. It is always: show me. So let's talk about what you can actually show, and where the evidence lives.

Sam: Right, and the thing architects get tripped up on is which API surface answers which question. The analytics APIs give you aggregated usage — token counts, model calls, that kind of thing. The Compliance API is the one that reaches the activity feed.

Koko: Exactly. And for enterprise organizations, the Compliance API goes further — directory data, the underlying chats, files, projects, session transcripts. You can retrieve or delete at that level. Aggregated usage can tell you how much. The Compliance API tells you who did what, when, and to which record.

Sam: The retention window matters here too. The default is six years, which lines up with a lot of legal hold requirements. So when a general counsel comes in and says: which staff used the assistant on this matter, what did they submit, what came back — that is a scripted query against the activity feed, retention aligned to that six-year default.

Koko: That is exactly the worked example to hold onto. Litigation hold, specific users, specific submissions, specific responses. That is not a dashboard question. That is a Compliance API question.

Sam: The analogy I use with my teams: a turnstile counts who entered the building. That is the usage dashboard. The evidence log records who touched which specific box, at what time. That is the compliance path. Both exist, but they answer completely different questions.

Koko: Chain of custody. I love that framing. And it leads directly to the instinct: for every control you claim, name the query that evidences it, name the source — compliance path, telemetry, or pre-inference control — and name its retention window. If you cannot name the query, you do not have the control. You have an intention.

Sam: And the misconception the exam is going to put right next to this: a usage dashboard is an audit trail. It is not. Aggregated usage answers how much. Compliance answers who did what, when, to which record. Those are not the same thing and they are not interchangeable.

Koko: Now, the second half of the fact. And this one surprises people. There is no governance page in the platform documentation. When someone says go look up the governance documentation, there is no such page. What exists are policy documents.

Sam: Three of them worth knowing by name. The Responsible Scaling Policy, the Usage Policy, and the Transparency Hub. And they do different things.

Koko: Right. The Responsible Scaling Policy is Anthropic's framework for its own model development — capability evaluations, safety-level standards, published risk reports. That is the provider governing what they ship. The Usage Policy is what your deployment is permitted to do. And then there is a gap.

Sam: The gap being fairness, bias, transparency at the application layer. The first-party sources are silent on that. Which means if you are shipping a screening assistant, a loan assistant, anything that makes consequential decisions about people — you are writing that governance yourself.

Koko: And the scenario here is worth making concrete. A screening assistant that a team wants to deploy. They cannot cite a vendor page for fairness. So they author one. What the assistant does and does not decide. Disclosure to candidates. Evaluation slices across the groups it answers for. An appeal route. The Usage Policy as a floor, not a ceiling.

Sam: The floor framing is important. The Usage Policy sets the minimum. Everything above that — the fairness commitments, the disclosure, the evaluation cadence — that is yours to design and yours to evidence.

Koko: The analogy here is a steel mill's certificate is not your bridge's inspection. The provider certifying the model is not the same thing as your deployment being certified. Their policy governs what they ship. Yours governs what you do with it.

Sam: And the trap the exam will set: that a provider's safety framework transfers to your deployment. It does not. The Responsible Scaling Policy is Anthropic governing Anthropic. It tells you what they committed to in building and releasing the model. It says nothing about whether your use of that model in a hiring context meets your fairness obligations.

Koko: Read the Responsible Scaling Policy to understand what the provider commits to. Read the Usage Policy to understand what your deployment may do. Author the rest yourself. That is the takeaway, and it is the thing the exam is testing when it asks you to distinguish between provider governance and deployment governance.

Sam: Alright, let me walk the instincts back before we close this domain. Across all four modules. One: the system prompt is the architectural boundary, evaluated at inference, not at deployment. Two: operator and user trust run on separate tracks and the system prompt is where you set them. Three: absolute limits are not negotiable by prompt, by operator, or by business case.

Koko: Keep going.

Sam: Four: when behavior is ambiguous, charitable interpretation scoped to the deployment context, not a blanket yes. Five: sensitive domains — medical, legal, financial, politically contested, factual controversy — get slower decisions, more friction, more hedging. Six: multi-agent pipelines do not inherit elevated trust from their position in the chain.

Koko: And the last two are from this module.

Sam: Seven: every control you claim, name the query that evidences it and its retention window. Eight: provider policy and deployment policy are separate documents with separate scopes. Read both, author the gap.

Koko: That is the set. And the habit to carry into the exam room: the option that names a mechanism — a specific API, a specific policy document, a specific retrieval path — beats the option with better wording every time. Governance is not what you say you do. It is what you can show.

Sam: If you can name the query, you have a control. If you can only name the intention, you have a gap.

Koko: The drills and flashcards for this entire domain are at KokoAI Academy — koko knows dot A I. Go do the retrieval practice. The exam rewards the instinct, and the instinct comes from the reps. You have done the hard work getting here. Go show them what you know.