Integration

Koko: Every tool you attach to an agent is paid for on every request, whether it gets called or not. That is the single instinct this domain keeps testing, and it is the right place to start.

Chris: That framing is exactly what I want sharp. Integration is nineteen percent of the Professional exam — biggest single domain on the test — and I have wired enough of these systems that the how-to is not what I need. What I want is the rule that decides which connection, because that is the question the exam keeps asking.

Koko: Right. And the rule comes from one idea: a connection is rent, not a purchase. You pay it every turn, whether or not anyone uses the thing. So the Professional question is never can this be connected — it is what does this connection cost when nothing is using it, and who pays that, every single time.

Chris: And the most literal place that shows up first is the tool list itself.

Koko: Exactly. So here is the fact. Tool definitions live in the context window and are sent on every request. Used or not. The model sees all of them before it reads a single word of user input. Anthropic's own guidance puts the threshold somewhere between ten and twenty tools as the point where a loaded toolset starts costing more than it is worth — the tool search page says ten or more, the tool-context guide says roughly twenty — and above that threshold, the recommended alternative is to send one tool-search tool and let the model discover the rest on demand.

Chris: One extra turn of latency, but the baseline context drops to a single definition instead of the full catalog.

Koko: That is the trade exactly. Now the scenario, because the numbers make it real. An operations agent had accumulated fifty-one tools. Teams kept adding integrations over time — totally reasonable individually — and by the time anyone measured, the baseline context was substantial before a single user word arrived. And tool selection had visibly degraded.

Chris: That degradation is the part that is easy to miss at first. Fifty-one tools means the model is choosing among fifty-one near-neighbors on every turn, many of which are plausibly relevant.

Koko: And when they switched to on-demand discovery — one definition standing, the rest fetched by the search tool — selection got better, not just cheaper. Because the model was no longer staring at a wall of options.

Chris: The discriminator I'd press on here: what is the actual criterion for deciding whether to defer? Because ten versus twenty is a range, not a number.

Koko: Good question. The criterion is not how many tools the platform supports — it is how many are plausibly relevant in a single turn. Under about ten stable tools that virtually every request needs, cache the definitions and leave them loaded. Somewhere in that ten-to-twenty band, start measuring whether you are seeing selection errors. Past twenty, defer. Judge by relevance density per turn, not by catalog size.

Chris: Which is basically the tradesman's van problem.

Koko: That is the analogy, yes. If you load every tool you have ever owned into the van on every job, you pay fuel for all of it every day, and you spend more time hunting the right wrench than turning it. The solution is not a bigger van — it is bringing what the job actually calls for.

Chris: And the takeaway for the exam room is: count the tools, measure the baseline, and use the plausible-relevance-per-turn criterion to decide where you sit on that spectrum.

Koko: That is exactly the instinct to carry in. Now the trap, and I will name it directly. The misconception is that tool bloat is only a cost problem, which leads people to reach for caching as the fix. Caching changes the price of those tokens — it does not change their presence. And presence is what degrades selection. If the symptom is the wrong tool being chosen, cheaper definitions do not help. Fewer visible ones do.

Chris: The option sitting next to this on the exam will say something like 'enable prompt caching on the tool definitions to reduce the performance impact of a large toolset.' That sounds reasonable. It is addressing a real cost. But it is not addressing the selection problem at all.

Koko: Right. And the counterweight — which the exam will also test — is that progressive discovery is a real trade with a real downside. Discovery is an extra step that can itself fail. If that search tool errors, the model cannot find what it needs. So you are not just optimizing cost and selection — you are also taking on a new failure mode.

Chris: So the decision is: stable small toolset, load and cache. Large or growing toolset, defer and accept the discovery latency and the extra failure surface. Those are genuinely different architectures with genuinely different risk profiles.

Koko: And that is the kind of choice-and-defense question this domain is built around. Every connection is rent. Count what you are paying before you add the next line item.

Koko: So Domain Three gives you three ways to wire things together, and the exam will absolutely ask you to pick the right one. Here is how each actually works. MCP exposes tools, resources, and prompts through a protocol layer — one server, and any client that speaks the protocol can discover and use it, no redeploy required when the surface changes. A direct API or CLI call is code you own: no protocol, no discovery, cheaper, tighter, shaped exactly for one consumer. And agent-to-agent delegation hands over a goal — not a function call — and gets back a summary, not a payload.

Chris: So the unit being exchanged is fundamentally different across the three. A direct call returns structured data, MCP returns a tool result through the protocol, and delegation returns a conclusion.

Koko: Exactly. And that difference in the unit is what drives the choice. Let me make it concrete.

Koko: A company has a ticketing system. Five different assistants need to touch it, and the tool surface changes monthly — new fields, new actions. They put it behind MCP. One server, all five consumers discover updates automatically, no client redeploys when the schema shifts. Their pricing service, on the other hand, has exactly one consumer and a contract that has not changed in two years. They call it with a plain typed client. Direct, fast, no protocol overhead. And when someone needs a competitive landscape summary — forty pages of research condensed to a recommendation — they delegate that to a specialized agent. The caller cannot decompose that work from the outside. They want the conclusion.

Chris: The pricing service one is the sharp edge. What stops you from putting it behind MCP anyway? It would still work.

Koko: It would work. You would also be operating a server, maintaining an authorization boundary, and paying a protocol tax for a relationship that already has a perfectly good typed contract. MCP earns its keep when one capability serves several agents or the surface moves. For one stable internal call, it is overhead you chose.

Chris: The framing I use internally is: a standard protocol changes distribution, not comprehension. It does not make the capability smarter or easier to reason about — it just makes it reachable by more consumers more cheaply over time.

Koko: That is the discriminator. Write that down. Now, the analogy that makes this stick for me: plumbing. A standard threaded fitting is worth machining when a dozen appliances are going to connect over the years. One pipe between two fixed points in your basement — you weld it. A full bathroom remodel you hand over as an outcome. You do not give the contractor a list of pipe cuts.

Chris: The weld-versus-fitting distinction is exactly the consumer count and churn question in physical form.

Koko: Right. So the takeaway, and this is the instinct the exam rewards: pick by consumer count and churn. Many consumers or a moving surface means a protocol. One consumer and a fixed contract means a direct call. A goal you cannot decompose from the outside means delegation.

Chris: The option sitting next to this on the exam will say MCP is the modern replacement for calling an API — like it is just the newer, better version of a REST call and you should default to it.

Koko: That is the misconception, and it is a seductive one. MCP is an interoperability and discovery layer. It earns its place when the many-consumers or moving-surface condition is true. When neither condition is true, you have added a protocol, a server to operate, and an authorization boundary — none of which you needed. The exam will make the MCP option look clean and modern. The question to ask is: clean for whom, and how many of them are there?

Koko: So in MCP's authorization model, the server is an OAuth 2.1 resource server. Two requirements do most of the security work. First, the client sends a resource parameter — that's the canonical URI of the server it actually intends to call. Second, the server validates that the incoming token was issued specifically for it as the intended audience. The spec is explicit: no other token may be accepted or transited. That's the whole mechanism in one sentence.

Chris: Right — and the word 'transited' matters there. An agent can't just forward a token it received for one server over to a different server, even if the user is the same and the agent is the same. The token's audience binding is to that one canonical URI.

Koko: Exactly. Let's make that concrete. Say an assistant holds a token for an internal analytics server, and it also communicates with a third-party server. Each token is audience-bound to one URI. If that analytics token gets misrouted — either by accident or by prompt injection — the wrong server rejects it. It doesn't silently honor it. And if the scope is insufficient for the operation being attempted, you get a 403 naming what's missing, not a quiet failure.

Chris: That 403 with the named scope is actually useful. In systems I've shipped, surfacing the specific missing scope is what lets the orchestration layer decide whether to request elevation or abort cleanly instead of just hanging.

Koko: Good operational point. Now here's the analogy I want you to carry in: a hotel key card. It opens one room in one hotel. The lock checks the card's binding to that specific door — it doesn't look at the guest's face. That's why it refuses in the wrong pocket. The binding is in the credential, checked by the receiver.

Chris: And the design instinct that follows from that is: you draw the token boundary per server, not per user. The question you ask at design time is — which server is each token minted for, does that server validate the audience claim, and what could an agent reach if it received a token it was never supposed to have.

Koko: That's the takeaway exactly. Token boundary per server. Audience validation on the receiving end. Minimum scope for what's actually needed. That's the mental checklist.

Chris: There's a second identity layer I want to flag, because this trips up real teams. An application credential — what the agent is running as — says nothing about which end user is behind the request. Those are two separate identities. User permission has to be enforced in your own code, before context assembly and again before execution.

Koko: Which sets up the trap perfectly. The misconception the exam is testing here is that authenticating the user is the security design. It isn't. The real risk is the confused deputy — a correctly authenticated agent that gets induced to use its authority on the wrong object or the wrong server. The agent isn't compromised. The user is authenticated. And it still goes wrong.

Chris: The version of this I keep seeing on teams is: we have login, so we're covered. But a login page stops an unauthenticated outsider. It does nothing about an authenticated agent that's been manipulated into calling the wrong endpoint with a valid token it legitimately holds.

Koko: Right. What addresses the confused deputy is audience binding — so misrouted tokens get rejected — scope minimization, so the blast radius is small, and human approval on sharp actions. Not another login screen. The exam will put 'enforce strong user authentication' right next to the correct answer, and the discriminator is: authentication is about identity, audience binding is about authority. Those are different problems.

Koko: Before we get into the mechanics here, one thing you need to know going in: Anthropic publishes no official guide to retrieval-augmented generation, chunking, or indexing. A documentation search on those terms returns nothing. The Cookbooks have recipes, but this objective measures reasoning you construct, not a page you memorize.

Chris: That's worth flagging early. So the exam is testing whether you can apply first principles to a retrieval design, not whether you read a spec.

Koko: Exactly. And the first principle is this: chunk to the unit that answers a question. Not to a fixed character count, not to a paragraph, not to a page. To whatever a user is actually asking about.

Chris: The policy assistant scenario is a clean illustration of that. We chunk by clause, not by section, because a user asking about a specific obligation is asking about a clause.

Koko: And critically — every chunk carries the section heading and effective date. Because if the chunk loses its heading, it cannot be cited. You retrieve it, you surface an answer, and no one can tell which policy it came from.

Chris: The photocopier analogy is the one I use with teams. You copy a page out of a bound manual and cut it just below the section heading. The text is perfectly legible. It is completely unusable.

Koko: That is exactly it. Legible and unusable. The metadata is what makes a chunk citable, and provenance is the whole point of retrieval in a regulated or auditable context.

Chris: There's a second design decision in that scenario that I think the exam probes hard: lookups by policy number go to an exact-match lexical index, not a vector search. An identifier has no semantic neighborhood.

Koko: Right. Embeddings find things that mean something similar. A policy number doesn't mean something similar to another policy number — it either matches or it doesn't. Sending an identifier through a vector index is spending money to introduce noise.

Chris: And the third leg: aggregates don't belong in either index. If a question requires counting or summing across records, you want a query over structured data, not prose retrieval.

Koko: So the instinct to carry out of this room: match the strategy to both the data and the query type. Conceptual questions go to embeddings. Exact identifiers go to lexical lookup. Aggregates go to structured queries. And whatever you retrieve, ground the answer with the citation mechanism the API actually documents.

Chris: The trap the exam runs at this point is the big context window argument. And I see the same reasoning on teams: model windows are large now, so you paste the whole corpus in and skip retrieval entirely.

Koko: The misconception is that retrieval is only a capacity workaround — that once the window is big enough, you don't need it. But retrieval is a precision and provenance mechanism. Pasting a whole corpus into every call costs more per call, dilutes attention across irrelevant passages, and destroys the audit trail. You lose the ability to say which passage a specific sentence came from.

Chris: The exam also tests the other direction though: whether retrieval is warranted at all. A small, stable corpus that fits comfortably in a prompt probably belongs in the prompt. Retrieval adds engineering complexity, and that complexity should earn its keep.

Koko: So the full decision: small and stable, just put it in the prompt. Large or dynamic, retrieve — but retrieve to the right unit, carry the metadata, and route each query type to the index that fits it. That's the reasoning the exam rewards.

Koko: Here is the thing the exam is actually testing when it hands you a configuration question: it is not asking whether you know the levers exist. It is asking whether you can justify the lever you pulled. And justification means naming the budget the choice was made against.

Chris: Which is why a vague answer like 'we used a lower-effort setting for speed' fails. The examiner wants to see what number you were trying to hit and for which path.

Koko: Exactly. So let's walk the levers quickly so we are all on the same page. Effort trades intelligence for latency and cost — you get answers faster but you are leaving some reasoning on the table. Fast mode, on the models that support it, buys raw output speed at a premium price. Batch trades immediacy for a discount. Streaming changes perceived latency, not total time — the wall clock is the same, but the user sees tokens arriving. And caching cuts time to first token by reusing earlier computation.

Chris: The priority service tier used to be another lever. That one is legacy now, no longer sold, so if the exam lists it as an option it is a distractor.

Koko: Good call. Now the scenario, because this is where it gets concrete. A trading-desk assistant has a stated requirement: ninety-fifth percentile response under two seconds for the interactive path. The team takes a lower-effort setting and adds caching. That combination gets them under the line. Same firm, overnight reconciliation job — no latency target, just finish by morning. That goes to batch and collects the discount. Two configurations, one written budget each.

Chris: And those two jobs probably share infrastructure. The point is you can't collapse them into one config and average your way to a number that satisfies either of them.

Koko: That is the analogy I want to use here. Think of a kitchen running a lunch counter and a catering order off one stove. Two clocks. The average completion time across both tells the person standing at the counter absolutely nothing about when their sandwich arrives.

Chris: And on the team level, I have seen this mistake made with dashboards. Someone puts up a system-wide p95 and it looks fine, but the interactive path is at three seconds and the batch job is bringing the average down.

Koko: That is exactly the trap the exam is going to set. The misconception is that latency is one number for the system. It is not. It is per user-visible path. Averaging across paths hides the tail. And here is the sharper edge: a p50 target lets a quarter of your users have a bad time and still be inside the number.

Chris: So the option sitting next to the right answer on the exam will say something like 'set effort to low and measure average response time.' It sounds operational. It is not.

Koko: Right. The instinct to carry in: write the budget as a percentile with a window before you touch any lever. Split interactive paths from asynchronous ones explicitly. Then when you pull a lever, record which one and what it cost on accuracy. That record is the justification.

Chris: And the second half of this pairs directly with the first. Once you have per-path budgets, you need monitoring that can actually see whether the system is answering correctly — not just whether it is answering.

Koko: Same kitchen. The smoke alarm sounds for a fire and stays completely silent for a sandwich that is merely wrong. Uptime, error rate, p95 — those are your smoke alarm. A system can be fully up, blazing fast, and returning confidently bad answers, and none of those metrics flinch.

Chris: The version of this I keep seeing on teams is: someone builds a beautiful dashboard showing green across the board, and the first they hear about a retrieval failure is from a user. The system was healthy by every metric they were tracking.

Koko: So what does monitoring that can see a wrong answer actually log? The assembled prompt. The parameters you sent. The stop reason. The retrievals and tool results that fed the answer. Every piece of what the system decided, not just whether it responded.

Chris: I think of it as a record of what the system decided, not more dashboards. The dashboards tell you the system is alive. The decision log tells you whether it was right.

Koko: That framing is exactly right and it is also the exam instinct. If the question gives you a monitoring design choice and one option logs response time and status codes while another logs the prompt, parameters, stop reason, and retrievals — the second one is the answer, because only that one can surface a quality failure.

Chris: The trap the exam will place next to it is something like 'add more alerting on error rate.' That sounds like better observability. It is just more of the same blind spot.

Koko: And the misconception underneath it is that correctness is visible from infrastructure metrics. It is not. The takeaway: log what the system decided, at the level of the decision — prompt, parameters, stop reason, retrieved context, tool outputs. That is the only record that lets you audit a wrong answer after the fact.

Chris: Alright, so we've been through the whole domain. Let me try to say what I actually carry out of this into a design session, and you tell me if I have it right.

Koko: Go for it.

Chris: Six moves, in the order you actually hit them. First, count the tools and measure the baseline — you need a number before you have a problem. Second, pick the coupling style by who the consumers are and how fast things churn. Third, bind the token to the server, enforce user permission in your own code, and do it twice. Fourth, chunk to the unit that answers the question — not smaller, not bigger. Fifth, write a latency budget per path before you write the code. And sixth, log what the system decided, not just what it returned.

Koko: That's exactly right. And I'd sharpen one thing on the third one: the doubling isn't paranoia, it's architecture. The token side and the permission side fail in different ways, so checking once doesn't cover both.

Chris: Right — one is about what the credential allows, the other is about what this specific user is allowed to see in your system. Different layers entirely.

Koko: And the through-line connecting all six is cost. Every one of those moves is really asking: what is this connection going to charge me on an ongoing basis?

Chris: Context spent every turn, a trust boundary you now have to maintain, a subsystem you have to operate. Those aren't setup costs, they're recurring.

Koko: Which is exactly what the exam is testing. Here's the habit to name before you walk in: integration questions almost always describe something already connected and then ask what it costs. The scenario is already running. They're not asking you to design from scratch.

Chris: And when two options both look like they'd work, the one that names the standing price is the one they want.

Koko: Yes. And the recurring right answer in this domain skews narrower rather than cleverer. Fewer visible tools. A tighter token audience. A chunk that carries its own heading. A budget per path rather than one budget for everything.

Chris: The exam is going to put a clever option right next to the correct one, and the clever option will usually be doing more — more tools exposed, wider trust scope, bigger chunks for efficiency. It looks like it's solving harder problems.

Koko: And the question you ask is: what does maintaining that cost every single turn? If you can name the price and the narrow option doesn't pay it, the narrow option wins.

Chris: A connection is rent.

Koko: That's the one. A connection is rent. You pay it every turn whether or not the feature is worth it that day. Design like a landlord is watching.

Chris: I like that a lot more than I expected to.

Koko: Drills and flashcards for the full Domain Three are at KokoAI Academy — find them at kokoknows dot A I. They're built around exactly the decision rules we covered today, including the cost-naming habit, so practice with those before exam day.

Chris: If you can say out loud what each connection costs, you're already thinking the way this exam wants you to think.

Koko: You've got the instincts. Go show them.