Skills Load Only When Needed

Koko: Here is the scariest sentence I have read about enterprise AI in a while: a badly worded description throws no error, breaks no test, and produces no log line. The skill simply never fires. And everyone assumes it is working — because it exists, and is well written — while the model quietly reconstructs the procedure from general knowledge every time.

Sam: So the failure mode is perfect silence.

Koko: Perfect silence. And you find out, if you ever find out, by noticing that your supposedly governed process produces slightly different outputs every run.

Sam: Which just looks like, quote, the AI being a bit inconsistent. Not a broken trigger.

Koko: Exactly. And that is the entry point into the piece we are going deep on today — Skills Load Only When Needed — which is a very precise mechanical explanation of how skill libraries actually work inside an agent, and why almost every default design decision costs you more than it should.

Sam: The subtitle calls the context window a budget line. Which, the first time I read that, I thought was a bit dramatic.

Koko: And the second time?

Sam: I went and looked at our own tooling and felt slightly ill.

Koko: So let's get the core argument on the table. Skills Load Only When Needed is making a claim that sounds obvious until you see the arithmetic behind it. A skill has two parts: a name and a description — that is all that sits in context by default — and then a body, which is the full procedure, the playbook, the links to reference documents.

Sam: And the body only loads when the description matches the work.

Koko: Right. So you can have thirty, forty skills in a library, and a routine run pays only for the descriptions — one sentence each — until something actually matches. The piece calls this a sublinear cost curve. Add a fortieth procedure and the baseline moves by one sentence.

Sam: That's the good news version. The bad news version is: if you break that property — if you always-load things instead of letting them load on demand — you lose the whole benefit at once.

Koko: And you lose it for everything simultaneously, because every always-loaded procedure is riding along on every other procedure's runs.

Sam: It's not additive waste. It's multiplicative waste.

Koko: That's the thesis. And the reason it matters right now is agents. The piece makes a point that hit me pretty hard: a prompting-era assistant pays a bloated baseline once per question. An agent loop pays it once per turn.

Sam: So the same design mistake that was mildly wasteful before agents is — how does the piece put it — an order of magnitude more expensive after them.

Koko: Nothing about the model got more expensive. You're just paying for your own context more times per task.

Sam: Walk me through the actual arithmetic, because the piece does this in a way that is hard to argue with.

Koko: Okay. Take a procedure of roughly two thousand tokens — the piece uses a close checklist with escalation rules as the example, which is realistic. Loaded on demand: it appears in the runs that need it. Say that is one run in fifty. Each such run carries it across, say, twelve turns. So the cost is two thousand tokens times twelve, once every fifty runs.

Sam: Fine. That feels manageable.

Koko: Now always-load it. Fifty runs at twelve turns is six hundred turns, every single one carrying two thousand tokens. Versus twelve turns in the on-demand case.

Sam: That's a fifty-times multiple.

Koko: The piece puts it cleanly: the multiple is the reciprocal of the hit rate. One in fifty hit rate, fifty-times tax. And then here's the part that the piece flags as counter-intuitive.

Sam: The tax is worst for the procedures that feel most important.

Koko: Yes. The instinct to always-load is strongest for high-stakes, rarely-used material. Capital-approvals playbook. Regulatory-filing checklist. And those are precisely the entries with the lowest hit rate and therefore the highest multiple.

Sam: So the CFO says, this one is critical, make sure it's always there — and by doing that they've just created the most expensive line in the context budget.

Koko: Every single run. Whether or not anyone is filing a regulatory document that day.

Sam: I want to linger on the description-as-trigger point, because that is where the silent failure lives and I think it's the most practically important thing in the piece.

Koko: Go ahead.

Sam: The description is not documentation. It is a matching surface. It has to use the exact phrases a person would actually say, not the phrases the procedure uses about itself.

Koko: And the failure mode is that most people write it the way they'd label an internal document — formal, categorical — which removes exactly the vocabulary that matches.

Sam: So if someone says 'can you help me close the books' and your skill is named 'Period-End Financial Reconciliation Procedure'—

Koko: It might just not fire.

Sam: And then the model does it anyway from general knowledge, differently each time, and your audit trail says everything is governed.

Koko: The piece calls that the real damage. Not just wasted effort — it silently returns the function to an ungoverned path while presenting as governed.

Sam: Okay, I want to push back on something, or at least name the tension I feel reading this.

Koko: Please.

Sam: The piece is very confident that a skill should not execute — it's knowledge plus a trigger, and the actual running happens in a subagent or a workflow. And the distinction between subagent and workflow is model-driven versus deterministic. But in practice, how many teams building this stuff have explicitly made that call?

Koko: The piece addresses this directly. It says a design discussion that never establishes which of those two is doing the work has not produced a design.

Sam: Right, which is a polite way of saying: most of the things being called agent designs right now are not designs.

Koko: Fair. And the table in the piece is genuinely useful here. Skill, subagent, workflow — each has a control path and a 'reach for it when.' The workflow wins on cost, latency, auditability, and — this is the bit I liked — the number of ways it can surprise you, when the sequence is fixed and correct.

Sam: Fewer ways to be surprised is underrated as a design goal.

Koko: Very underrated. But here is where I'd push back slightly on the piece's framing. It says the subagent's value is specialization plus tool scoping plus context isolation, not raw power, and that a specialist handed every tool has surrendered two of those three.

Sam: True but hard. Because the path of least resistance when you're building fast is: give it everything and see what happens.

Koko: And the piece sort of acknowledges this in the section on honest effort. It says the first skill will be slow — not because packaging is hard, but because writing down what is actually done, as opposed to what the policy document says is done, is the part organizations have been deferring for years.

Sam: That's almost a therapy observation dressed as an engineering note.

Koko: It is. And I think it's the most honest line in the piece. The artifact produced is useful whether or not an agent ever runs it. And it is the one input no vendor can supply.

Sam: So even if you never deploy the skill, you've been forced to write down your actual close process — which might be the first time that's happened.

Koko: Which is worth something entirely independent of the AI question.

Sam: Okay. Executive in the room, busy, twenty seconds. What does this piece want them to leave with?

Koko: Three things. First: your context window is a cost center with a line-item structure you probably cannot currently see. The piece makes the case for recording which skill fired on each run — that one field turns a single monthly model invoice into cost by procedure. Suddenly you can ask: what does our close actually cost to run this way? Which procedures are worth packaging next?

Sam: And the answer to that second question is frequently not the procedure that feels most strategic.

Koko: The piece says rank by frequency times cost per run. Feeling strategic is not in the formula.

Sam: What's the second thing?

Koko: Audit your always-loaded set. Every entry in it should have a named reason it cannot be on-demand. The piece says the default answer is that it can be. So if nobody has a named reason, it should probably move.

Sam: And the third?

Koko: Build for the third metric, not the first two. A skill that never fires wastes the effort of writing it. A skill that fires constantly and is wrong wastes the effort of everyone downstream, at volume, and looks like adoption on every chart that counts invocations. The piece calls that the most expensive thing in the portfolio — and says it's invisible without both the load rate and the acceptance rate.

Sam: Load rate and acceptance rate. Most teams are measuring one of those at best.

Koko: And the piece is pretty direct that a skill that makes the model consistent but consistently wrong is genuinely worse than an inconsistent human — because inconsistency is at least occasionally noticed.

Sam: Reliably wrong at scale. That is a bad place to be.

Koko: It is. And the thing is — none of the four context management disciplines the piece lists are exotic. What loads by default and why. What a tool returns. What survives across turns. What is retrieved versus remembered. These are all things that get decided by default in the first fortnight of a build and then set the unit economics of everything built afterwards.

Sam: The first fortnight decides the cost structure for everything downstream. That is not a sentence most teams are taking seriously.

Sam: So to close: Skills Load Only When Needed is making a case that the engineering decisions that look the most boring — what sits in context, what the trigger description says, whether a thing is a subagent or a workflow — are actually the financial decisions.

Koko: They are the financial decisions. Dressed as technical defaults. Made by someone who didn't know they were setting a budget.

Sam: And the question I keep coming back to — the one I'd leave the listener with — is: if you pulled the trace from your current agent build right now, could you tell me which skill fired and what it cost? And if the answer is no, what are you actually managing?

Koko: That is the right question. And it doesn't require a new tool or a new vendor. It requires a field in your trace log and the discipline to look at it.

Sam: Low bar. High leverage.

Koko: The full piece — Skills Load Only When Needed — is on Koko Knows, and I'd particularly recommend the packaging table and the context management checklist if you are walking into a design review. Everything in it is source-verified, and it tells you exactly when the figures were last checked. Worth the read.