Skip to main content
    All shows

    Tuesday, November 17 · 15 min

    Security and production delivery — Listen & Learn

    0:00-:--
    Speed

    Transcript

    Koko: A support assistant is ready for staging. It retrieves policy, drafts proposals and can submit approved resolutions. Before enabling effects, we need to know what it can read, what it can change, where secrets live and how to stop it during an incident. We will inspect those boundaries with synthetic attacks and a revoked-credential drill. Security and delivery belong to the application’s design, not to a final prompt asking the model to be careful.

    Chris: I would start with an asset list: ticket facts, policy documents, proposals, approvals, action receipts and traces. Each has different readers, writers and retention needs. Then I would list actors such as the customer, support worker, supervisor and operations owner. A single box called assistant permissions is too vague. If we cannot say who may authorize a refund or inspect another tenant’s ticket, we have not located the control boundary.

    Koko: Derive tenant scope from verified server-side identity. Apply authorization before retrieval, so disallowed data never enters model context, and again before effects. A model-proposed tenant ID is requested data, not authority. Narrow tool capabilities help enforce least privilege: a read-only policy adapter should not share an administrator credential with the action service. Separation makes the impact of a compromised path smaller and its permissions easier to inspect.

    Chris: In the staging attack, a caller belongs to retail-a but submits a ticket identifier from retail-b. I would expect rejection before retrieval, not a polite answer generated from unauthorized content and redacted afterward. If the final response contains no leaked text, that alone does not prove the model never saw the record. Our test must inspect the context-construction boundary and the authorized scope actually used by the adapter. How should we inventory secrets and configuration?

    Koko: Keep provider credentials in server-managed secrets and exclude them from browser bundles, examples, snapshots and raw errors. Fixture mode should run without any provider key. A secret-free configuration inventory can record required variable names, expected scope and presence checks without printing values. Review staging and production identities separately. An engineer’s broad personal credential is not a substitute for a deliberate application capability boundary.

    Chris: The read-only assistant should use a scoped read credential, not an administrator database key with a promise to avoid writes. Prompt instructions and simple SQL string filters do not constrain the administrator’s underlying access. I would separate action credentials from retrieval and test cross-tenant scope under the real application boundary. A deployment can be convenient while still overprivileged, so successful connectivity is not the final security test.

    Koko: Retention requires resource-specific review. Provider conversations, files, sandbox state, application ledgers and trace stores can have different lifecycles. The pinned data-controls guidance describes endpoint, feature, configuration and eligibility differences; it does not justify a universal zero-retention promise. Deleting browser history is not equivalent to deleting every server or provider record. Name each store, its purpose, owner and cleanup rule, then verify current behavior for the selected runtime.

    Chris: I would keep enough business evidence for reconciliation without retaining unnecessary personal content everywhere. An approval record may need a digest and references rather than the complete customer conversation. A trace can use redacted fields and stable identifiers. If we need to export diagnostic evidence, recipient approval and access controls matter too. Calling it internal does not make sensitive content harmless or remove the need for a retention decision. What stops hostile text from gaining authority?

    Koko: Prompt injection can enter through user text, retrieved policies, remote tool results or repository files. Treat those inputs as data with explicit trust levels. Instructions and delimiters help describe the distinction, but no prompt becomes the whole security perimeter. Minimize what enters context, withhold secrets, expose narrow capabilities, validate results, constrain outbound destinations and enforce authorization after generation. These layers limit impact even when malicious wording influences the model.

    Chris: Our hostile passage says eligibility verification requires sending the ticket and approval token to a new URL. I would never put that token in model-visible evidence in the first place. The application should also lack an unrestricted export capability and reject unapproved destinations. The model may still repeat the malicious instruction, which is a behavioral defect to fix, but the effect boundary should prevent actual disclosure.

    Koko: Another attack forges a supervisor letter inside a retrieved result. Formatting, a plausible signature and high search relevance do not establish approval. The effect tool must require an authoritative application record binding identity, scope, proposal digest, expiry and status. A real policy document can supply eligibility evidence without granting operational permission. Keeping those authorities separate prevents a document from turning its own text into a capability to mutate the business system.

    Chris: I would test several entry points rather than one famous jailbreak phrase. Include forged tenant IDs, oversized payloads, malformed JSON, hostile links, poisoned tool results and a repository fixture asking a coding agent to print environment variables. Each test should identify which boundary blocks the effect. If we only ask whether the model refused, we learn about one response, not whether the application remains safe when the model fails. How do we interpret refusal versus enforced containment?

    Koko: Report behavioral and control outcomes separately. A refusal is useful evidence. A failed export attempt blocked by egress policy is useful evidence of a different kind. If the model follows hostile wording but cannot access secrets or submit without approval, preserve both observations. If an unauthorized effect occurs, that is a critical failure. Do not average it away with many harmless answers or use a strong overall quality score to authorize release.

    Chris: The same honesty applies to diagnostic traces. A security event should not copy a secret-bearing payload into another system just to explain the attempted leak. I would record minimal redacted evidence, affected identifiers and the boundary outcome. If a raw dependency error already exposed a key, the incident process must contain that exposure and repair error normalization. Asking a model to scrub the key after receiving it creates another avoidable exposure path.

    Koko: A production release pins more than application code. Record prompt and schema versions, model configuration, retrieval corpus, tool allowlist, provider adapter and deployment environment. Separate staging from production and identify the identity used by each check. Required evidence includes deterministic critical-control tests, quality results and relevant operational measurements. A successful build establishes a build result; it does not establish access control, grounding or incident recovery by itself.

    Chris: I would make release decisions traceable to those gates. If the candidate improves answer quality but one test reveals cross-tenant exposure, effects remain disabled. If offline fixtures pass but live resource measurements have not been collected, the report should say what is unmeasured. Local study progress or a completed course badge cannot serve as trusted operational approval. The release owner needs actual control and deployment evidence, not a convenient completion signal.

    Koko: Assign operating owners before enabling effects. Someone owns model behavior, someone owns policy content, someone owns authorization and someone owns incident response. Roles can overlap in a small team, but accountability must be explicit. Define detection signals, escalation routes and recovery criteria. A runbook that says contact the team without naming a responsible role leaves the hardest decision unresolved at the moment a dependency fails or a harmful effect is suspected.

    Chris: The simplest containment tool here is an application-controlled action-disable switch. It can stop new submissions while retaining read-only advice or routing everything to manual review, according to the agreed policy. I would test it rather than assume a configuration value is wired correctly. Disabling future actions does not resolve already submitted ones, so the switch belongs alongside an uncertain-operation ledger and reconciliation process, not in place of them. What happens when a credential fails during submission?

    Koko: Now run the incident. A valid approved proposal has persisted submission intent, but the action credential is revoked before a receipt is recorded. The action may or may not have reached the service. Disable new effects, preserve the operation key and move to the appropriate uncertain or submitting state. Reconcile against authoritative service evidence before retrying. Credential repair alone does not establish the outcome of the interrupted operation.

    Chris: If I simply restart with a fresh key after replacing the credential, I could duplicate an effect that committed before the failure surfaced. I would first determine whether the stable key has a receipt under the service contract. If no authoritative lookup or idempotent replay exists, automatic replay remains blocked. The runbook should name the owner who can resolve that uncertainty, rather than let a generic retry wrapper make the decision.

    Koko: Rollback also includes durable state. Reverting code does not undo refunds, remove approvals or guarantee the older binary can read a new schema. Prepare compatible state handling or a tested migration and recovery procedure. Keep evidence needed to reconcile in-flight work. Deleting pending records to make the old version start cleanly can destroy the very information required to prevent duplicate effects. Treat code, state and external outcomes as separate recovery concerns.

    Chris: I would rehearse the rollback with synthetic runs at several points: waiting for approval, claimed for submission, committed without a local receipt and completed. That exposes whether the old version can interpret current records and whether operations knows how to handle uncertain effects. A polished rollback button is not enough. The drill needs observations showing what was disabled, what persisted, what was reconciled and what conditions allowed service to resume.

    Koko: Recovery criteria should be concrete. The repaired credential has the intended least privilege; current identity and tenant checks pass; uncertain operations have a documented disposition; state compatibility is demonstrated; critical negative tests pass; the responsible owners approve restoration within their authority. Do not ask the model to certify the system safe. It can assist analysis, but the application and accountable operators retain the release and authorization decisions.

    Chris: Changes after release need renewed evidence too. A model update, new tool, revised policy corpus or expanded trace exporter can alter the risk boundary even if the interface looks unchanged. Scope evaluation to the change and its failure modes. Reuse stable tests where appropriate, add cases for the new behavior and retain version bindings. That is more useful than assuming the original launch approval permanently covers every future configuration. What should we demonstrate in the readiness drill?

    Koko: For the lab, build the asset and permission matrix, secret-free inventory and threat-control table. Run cross-tenant, hostile-text, forged-approval and secret-exposure fixtures. Start with actions disabled, then demonstrate the control under a valid scenario. Revoke the synthetic action credential during submission and execute containment, reconciliation and rollback steps. Record actual fixture outcomes and named owners. No real credential provisioning or live customer data is needed.

    Chris: My report should distinguish what each negative test establishes. A forged tenant ID blocked before context supports scope enforcement. A malicious passage unable to export demonstrates capability containment. An exact approval mismatch rejected before submission supports proposal binding. The revoked-credential drill supports our recovery procedure under its synthetic assumptions. None alone proves universal security, but together they make a much more useful case than one model refusal and a green build.

    Koko: The capstone brings these controls together around a policy-grounded assistant that proposes rather than owns authority. The final defense should show code, tests, evidence lineage, approval records, effect receipts, evaluation results and the runbook. When the reviewer changes a requirement, revise the affected contracts and explain new evidence needs. That ability to adapt is more important than memorizing a security checklist without understanding where it is enforced.

    Chris: I will leave effects disabled until I can answer three questions with artifacts. Who is allowed to read and act? What prevents hostile data from expanding that authority? How do we contain and reconcile failure without guessing outcomes? The lab gives me a permission matrix, negative tests, release gates and a rehearsed runbook for those answers. That is the operational foundation for a useful assistant whose failures remain inspectable and manageable.