AI Engineer

Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic

1978 summary words 9 min summary Watch video

Start with the signal

9 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: Production agents should receive bounded, identity-enforced operational budgets—not broad binary tokens—using reversible actions, per-write rate limits, aggregate tripwires, and human-held escalation keys.
  • Why it matters: This is a concrete control-plane pattern for allowing agents to perform useful production work without letting a single mistaken loop silently create an unbounded blast radius.
  • Best use: Use it as a design review checklist for OpenClaw or other agentic systems that can write to infrastructure, deploy software, alter flags, or trigger operational workflows.

Executive Summary

Sachin Malhotra argues that the common demo pattern—give an agent a powerful token and a tool list—fails once agents operate in production. His motivating incident was an agent cleaning up workloads: an empty filter caused a selector to match everything, deleting roughly 200 workloads affecting about 20 engineers in 90 seconds. The central failure was not malicious behavior or necessarily a model defect; it was granting unbounded power to an unattended system that can act continuously and confidently while wrong.

His replacement for binary authorization is a multidimensional “budget”: how much an agent can do, how quickly it can do it, what it can undo itself, and who notices while it acts. The operating model consists of asymmetric verbs, refillable rate limits on every write, aggregate tripwires, and an “undo test” that determines when a human second key is required. It deliberately resembles how organizations onboard junior engineers: useful authority by default, catastrophic actions structurally out of reach, and escalation paths for exceptions.

The implementation argument is as important as the policy framework. Prompts and context files convey intent and can be updated cheaply, but they are only advice and reportedly work about 80% of the time. Deterministic infrastructure must enforce the hard limits. Anthropic’s pattern is a per-agent-session proxy that owns real credentials, stamps every outbound action with a non-forgeable agent/session identity, and lets downstream systems apply quotas, approvals, ownership, and monitoring from that identity.

The strongest practical takeaway is to make the agent autonomous inside a small, automatically replenishing envelope while preserving human overrides outside it. Let agents safely perform actions whose errors become visible and are cheap to reverse; require an auditable human-held second key for quiet, irreversible, or high-blast-radius actions. Monitor behavioral aggregates rather than scrutinizing individual calls, then improve the agent’s guidance from observed failure patterns.

Key Takeaways

  • Claim: Replace broad, static token scopes with operational budgets that bound an agent’s authority across magnitude, rate, reversibility, and observation. | Evidence: A cleanup agent encountered an empty pipeline stage; its selector then matched everything and deleted about 200 workloads affecting roughly 20 engineers in 90 seconds. Malhotra contrasts this with a token’s binary scope: too narrow makes the agent useless, while too broad creates postmortems. | Implication: For every agent write capability, define a quantitative envelope and escalation path instead of deciding only whether the agent has a verb at all. | Caveat: Budgets do not eliminate the need for authorization boundaries; they complement scoped access by constraining the damage possible within authorized operations.
  • Claim: Authorize asymmetric verbs differently: give agents actions that fail visibly, but gate actions that can fail silently. | Evidence: In CI test quarantining, incorrectly unskipping a test turns builds red and is cheaply correctable, so the agent can do it. Incorrectly skipping a test can let a real bug reach production under green checks, so that break-glass action requires a human and an audit trail. | Implication: Classify tools by the consequence of a wrong action—not merely resource type or API name—and reserve quiet-failure verbs for approval workflows. | Caveat: Visibility is not sufficient by itself; reversibility and blast radius must also be evaluated through the undo test.
  • Claim: Apply a refillable rate limit to every agent write, with the limit sized by resource sensitivity and scope. | Evidence: After the deletion incident, a neighboring Anthropic team built an admission webhook that caps deletes by fixed number per hour, resource kind, and namespace. An agent can have a higher deletion budget in its own namespace and a smaller one in a shared namespace; the original incident would have been capped at a couple of tens of workloads. | Implication: Build rate enforcement into the write path, so a malfunctioning loop is bounded without forcing humans to pre-approve routine actions or file tickets for temporary permissions. | Caveat: Emergency bypasses are necessary, but the bypass must not execute inside an agent session; it should instruct the agent to request that a human run the command.
  • Claim: Use tripwires to learn from aggregate agent behavior after bounded actions occur, rather than relying on static allow lists or inspecting isolated calls. | Evidence: Anthropic tracks investigation threads launched per hour for a given test-job failure. A spike revealed that the agent had opened separate, individually reasonable investigations across dozens of jobs sharing one infrastructure failure signature. The remediation was a context instruction to correlate failures before opening separate threads. | Implication: Instrument agent behavior at the aggregate level, route triggered alerts to an owner, and turn recurring patterns into concise operating guidance rather than prematurely expanding static allow lists. | Caveat: A tripwire is a smoke detector, not a preventive lock: it pages after the write or threshold crossing, so it must be paired with hard limits and must alert someone who will respond.
  • Claim: Use the undo test to determine whether an agent may act alone: can it restore the change itself, and is a wrong outcome tolerable? | Evidence: An agent can fully ramp and roll back a feature flag in canary traffic, but it can only propose promotion to real production. Similarly, it cannot undelete an uncheckpointed running job in another team’s namespace. | Implication: Set autonomy boundaries around reversibility and blast radius. Canary rollout and rollback can be agent-owned; irreversible production promotion, destructive cross-namespace changes, and similar operations should be human-authorized. | Caveat: When either self-recovery or acceptable impact is absent, the decision requires a second key held by someone other than the agent, plus an audit record.
  • Claim: Agent identity must be infrastructure-issued and propagated independently of the agent’s request, or budget controls are bypassable. | Evidence: Each agent session runs beside a proxy that holds real credentials and stamps outbound calls with the known agent and per-session identity. In the Kubernetes example, the cluster applies that stamp as a job label inherited by child jobs; quotas, approvals, rate limits, ownership, and tripwires all key off it. If an agent can set its own identity header, it can simply change the name after reaching a limit and receive a fresh budget. | Implication: Treat provenance as a control-plane primitive: route all agent egress through an identity-bound proxy and make downstream safeguards consume that immutable identity rather than agent-supplied metadata. | Caveat: The proxy intentionally does not interpret the agent’s rationale; it is a narrow deterministic enforcement layer that counts, compares, and allows or returns a 403.

Detailed Brief

Policy split: guidance in text, enforcement in infrastructure

  • Claims: Policy needs two layers with different jobs: context explains intent and operational rationale, while infrastructure deterministically bounds actions.; Prompt/context policy is useful because it can be changed quickly and can tell an agent how to reason, but it must not be treated as a security boundary.; The proxy is deliberately unable to reason about intent or negotiate exceptions; this reduces the ability of prompt injection or model behavior to override controls.
  • Evidence: Malhotra says the relevant instruction in a Markdown context file works about 80% of the time, but requires ongoing maintenance as such files grow.; The enforcement proxy sees a concrete operation such as a delete and a crossed budget, then returns a 403 rather than evaluating why the agent wanted the action.; Every outbound call from an agent session is routed through the colocated proxy.
  • Caveats: Textual policy can become stale, overly long, or internally conflicting; maintaining it is an operational requirement rather than a one-time setup.; A deterministic proxy cannot supply nuanced approval reasoning, so systems still need explicit human escalation paths for legitimate exceptional work.
  • Implications: Maintain agent instructions as an editable operational playbook, but translate non-negotiable constraints into external policy enforcement.; Keep the enforcement layer small and auditable: identity issuance, operation classification, counters, limits, and approval checks are more robust than a proxy attempting to emulate agent reasoning.

Composable controls rather than one universal gate

  • Claims: The controls answer distinct questions: asymmetric verbs address what is touched, rate limits address how often, tripwires reveal behavioral outcomes, and the undo test sizes the applicable safeguards.; Not every write requires every safeguard; controls should be selected based on the operation’s failure mode and recovery characteristics.; The framework is an explicit translation of an engineering onboarding checklist into machine policy.
  • Evidence: The speaker frames the corresponding human questions as: what can a new engineer touch, how much rope do they get, who signs off, and how does the organization know the process is working.
  • Caveats: The talk provides design principles and Anthropic examples, not a complete implementation specification for policy schemas, approval UX, or cross-service identity standards.
  • Implications: Use a per-operation review matrix rather than a single blanket 'agent mode' permission level.; Evaluate new tools and workflows against the same onboarding questions before exposing them to autonomous agents.

Notable Concepts & Terms

  • Agent budget: A multidimensional alternative to binary token scopes: limits on amount, rate, reversibility, and observation of agent actions.
  • Asymmetric verbs: Operations that look similar but have fundamentally different failure consequences; agents should preferentially receive verbs whose errors fail visibly.
  • Undo test: A decision lens asking whether the agent can restore the change itself and whether a wrong outcome is acceptable; failure on either condition requires a second key.
  • Second key: An approval or separately scoped production credential held outside the agent, used for high-impact or irreversible operations.
  • Refillable rate limit: A per-write ceiling that restores over time, preserving routine autonomy while capping damage from a single runaway loop.
  • Tripwire: Post-action aggregate monitoring that detects unusual agent behavior and pages an owner; it learns from production behavior rather than predicting all valid actions upfront.
  • Infrastructure-issued identity stamp: A proxy-applied agent/session provenance label that downstream services use for enforcement; it cannot be forged or rotated by the agent to evade controls.
  • Canary versus production scoped keys: A practical second-key pattern: the agent controls feature-flag rollout and rollback in canary environments but cannot promote the flag to production.

Operator Notes / Why Ken Should Care

  • Create a write-capability inventory for each agent tool, classifying the operation by silent versus loud failure, reversibility, maximum blast radius, and required escalation owner.
  • Require every agent-originated outbound write to traverse a credential-holding proxy that injects immutable agent and session identity; reject caller-provided identity as an authorization input.
  • Implement per-identity, per-resource, and per-scope refillable write budgets, beginning with destructive actions, shared namespaces, production flag changes, and high-fanout workflow triggers.
  • Make bypass mechanisms unavailable to the agent runtime; route budget-exceeding requests into a human-run, auditable escalation flow.
  • Add aggregate behavioral tripwires with accountable paging—for example, duplicate investigations, action volume by failure signature, destructive action counts, and unusual fan-out—and feed resolved incidents back into concise agent context.
  • Separate policy artifacts into advisory context files and enforced control-plane rules; periodically prune and test the context files rather than treating prompts as durable enforcement.

Source/Metadata

  • Title: Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic
  • Transcript words: 5550
  • Duration seconds: 1193
  • Timestamp note: No timestamps or chapter markers were present in the supplied transcript. The latter portion repeats earlier material, so the effective unique content is shorter than the supplied word count.
Search this video Resume transcript at 0%
Full transcript 3925 words · 25 min read
0:12

Hello. All right. Let's get started. Welcome to the talk, Give the Agent a Budget, Not a Token. My name is Sachin. I'm an engineer on the CI team at Anthropic. We build and manage all of the test machinery for all the code that people write. Everything from test quarantining to merge automation to CI auto scaling to merge queues, everything and anything in between. It's the plumbing that allows a few thousand engineers every day to ship code safely. I've been at the company for a little over 10 months. I'm based out of Seattle. Outside of work, I love dialing my espressos and climbing some crazy mountains.

0:21

A lot of cool agent demos start the same way, really. Someone gives an agent a God token and gives it access to some sort of tool list and just watches it go. And it does go. In this particular example, it'll spin up a cute little coffee website. It'll build all the pages, deploy them, bring it up live, and it's done in three seconds. People will nod, people will clap, and someone will ship it into production. This talk is about what happens after that, after the demo ships, once the agent is starting to do some real work in production.

0:51

So here's what after looks like in one of our scenarios. What you're looking at is a real command. The agent was trying to clean up after itself. It was listing a bunch of workloads that were no longer interesting or useful to itself. Then it found them and deleted them. Now, that's completely reasonable, except one stage in the pipeline evaluated to nothing. The filter dropped out, and now the selector matched everything. So you can see it took out about 200 workloads, which ended up impacting about 20 engineers' worth of stuff. And all of that was gone in 90 seconds. Nobody was being malicious in this case. The agent genuinely thought it was tidying up after itself. Some of these workloads were long-running training jobs and stuff. Maybe some of these were not even checkpointed. It was just hours of progress that was gone. And poof, 90 seconds and stuff, right?

1:21

The problem in this case is that this idea of, here's a token, and here's a tool list, is just not enough. It doesn't scale when the agent is starting to do some real work in production. So here's a brief summary of what we're going to talk about today. I'm going to walk through three primitives and asymmetric verbs. We're going to look at rate limits. We're going to look at this idea of trip wires over allow lists. And then I'm going to talk about this one lens, which I call the undo test, that you can use to size up the other three primitives. So, three things that you enforce and then one question you ask about all of them.

2:18

So here's what bugged me after the cleanup was done for the incident that I was just showing you. The agent technically hadn't done anything that I couldn't have done. It was using my token, after all. The failure wasn't the model itself. The failure was that I was giving the agent an unbounded amount of power to do something that I wasn't watching super intently.

2:22

And we've solved the same kind of problem, except not for agents, the first time you onboarded a junior engineer on your own teams. Right? So just think about it. We don't sit around watching every engineer or newly boarded person type out every keystroke on their keyboard. We're not behind their chairs just watching everything that they're doing. There is always a path whenever a new engineer wants to ask for something. There is always an escalation path. The catastrophic stuff is just structurally out of reach for them. So the other side of the coin is that the agents are very different. They never get tired. They never sleep. And every so often, they're just very confidently wrong.

2:37

And if I'm being honest, that's maybe most of us in the first month of our job anyways. What we're going to talk about is this idea of your onboarding checklist, but written down as policy for agents.

3:07

And that brings me to the word in the title of my talk here. The standard fix for an incident like that is basically that you narrow the token scope. You just take the deletes away, effectively. You would technically never do that for a new hire. You wouldn't take the whole verb away from them. And it also doesn't work for an agent either.

3:28

Maybe it works for about a week, maybe two, but then eventually you end up in a situation where the agent is genuinely trying to delete something that it feels is just not in the infrastructure. And you will just be there sitting and pressing enter by hand all over again. The core concept of the token that I feel is wrong is that a token is a boolean. It's just a yes or no. It's a static list of scopes. You either have it or you don't have it. So if the token list is too tight, then your agent is effectively useless. If the token list is too wide, then you're maybe writing a postmortem. A budget is a very different shape.

4:08

A budget is not just one number. It has four different dimensions. How much can the agent do? How fast can it do it? What can it undo on its own? And then who's noticing while it's actually taking those actions? That's roughly the umbrella for my four primitives, that each of the things that follow is basically one of the things along these dimensions.

4:23

And they're going to replace a yes or no question with a budget. So I'm going to start off with this first verb, which I call asymmetric verbs. And by verbs, I simply mean operations or actions that an agent can take. It could be API calls, it could be CLI commands, it could be really anything. And they're asymmetric because the same-sized action, even though it looks the same size, can have a very different blast radius in actuality, depending on which direction it goes. So the core point is that you need to stop thinking about resources for a second and think about verbs. Specifically, think about what happens when one of the verbs goes wrong.

4:44

Some verbs fail out loud. So in this particular example, let's say if an agent decides to unskip a test and it's the wrong call, the worst that would happen is CI would go red for a bunch of people. Same with paging. If an agent decides to page a human and it's the wrong call, the worst that's happening is that it's a nuisance for the on-call, but there is always a human to correct it. There are other verbs that fail silently, right? So if the agent decides to skip a test due to whatever reason it shouldn't have, nothing technically turns red. A real bug can actually walk into production with green checks, and nobody would notice it until much later. So unskip and skip, in this example, they're effectively the same kind of action, but the difference is which of the failures would show up on a dashboard and which one wouldn't.

4:52

And so the core idea is that you give access to verbs that can fail out loudly on a dashboard to your agent, and for the other ones, just involve a human.

5:03

Just a bit of context on CI stuff and how this plays out for us is that we have a test quarantining service behind the scenes, and it basically holds a list of all the tests that are currently skipped because an on-call decided that they had to break glass in a certain situation whenever there was an incident or something. Now, the agent has the ability to re-enable any one of these zones depending on when it evaluates that the tests are working fine and stuff. Again, if it's the wrong call, the worst that would happen is a bunch of different builds and tests would just start showing up red, and then a human can actually put them back very cheaply. The skip is a break-glass verb itself, right? It's basically, as I mentioned, what our on-call would reach for during a very critical situation, under pressure and stuff. And so, in that scenario, if an agent actually gets it wrong, a real production bug can actually, a real bug can walk into production.

5:13

So this needs a human, and it will always leave an audit trail. And the key detail is that the agent itself is not responsible for writing the row or the audit trail itself. There is a proxy in the middle that I've highlighted that I'm going to talk about in a little bit, which is responsible for stamping the caller's main identity on every call, whether it's a skip or an unskip. The agent technically never holds the pen on its own provenance.

5:19

The second primitive is about rate limits. It's a pretty standard concept, but this is the most concrete form of the budget idea as a whole. So, a ceiling that refills. So every caller gets a small amount of disruptive actions per time window, during under pressure and stuff. And so, in that scenario, if an agent actually gets it wrong, a real production bug can actually, a real bug can walk into production. So this needs a human, and it will always leave an audit trail. And the key detail is that the agent itself is not responsible for writing the row or the audit trail itself.

5:42

There is a proxy in the middle that I've highlighted that I'm going to talk about in a little bit, which is responsible for stamping the caller's main identity on every call, whether it's a skip or an unskip. The agent technically never holds the pen on its own provenance. The second primitive is about rate limits. It's a pretty standard concept, but this is the most concrete form of the budget idea as a whole. So, a ceiling that refills. So every caller gets a small amount of disruptive actions per time window, so you can spend them however you want. There's no approval, there's no waiting.

6:04

And if you cross the line, the request simply bounces back with the count, saying that you're actually exceeding your account, your budget. You wait a bit, and then the limit essentially refills. And that's the whole thing. So the agent gets full autonomy within the limit, and there is a hard ceiling on how bad a single loop can get. And every write effectively gets a rate limit. There are no exceptions to that. What changes is the size of the rate limit. So if I'm trying to delete a bunch of workloads in my own namespace, my rate limit might be higher. But if I'm trying to touch resources in a shared namespace, I guess my budget or my rate limit might be smaller.

6:35

So this is effectively the solution for the incident that I was showing earlier.

6:41

After the incident, the team that sits next to mine built an admission webhook of sorts, whose sole job is to cap the number of deletes at a fixed number per hour, per resource kind, per namespace. There is always a bypass flag, because sometimes you genuinely want to delete more than you're allowed, in maybe some on-call scenario or whatever. And the part that I absolutely love in this case is that inside a cloud code session or inside an agent session, effectively, the bypass flag simply refuses to do anything. All it's going to do is tell the agent to ask the human to run the command itself. So the agent effectively gets the rate limit, and the human keeps the override. And nobody effectively has to file a ticket for the limit, because it just refills. The third primitive is about this concept of tripwires over allow lists.

6:47

So in my mind, an allow list is effectively a guess that you're making upfront about what the agent needs, or about model behavior or agent behavior itself. I feel like it's pretty static, and you write it upfront before you have any data on how the agent is behaving in different situations. A tripwire, on the other hand, is how you get that data after the fact.

6:59

So for cheap actions, you let the agent act, and every action gets recorded with the actor stamp, identity stamp. And these two go well together. Rate limits are the enforcement; they put a hard limit on the rate itself. Tripwires are how you find out what actually happened, so you've got something to react to. So you effectively watch the aggregate and not individual calls. And usually when a tripwire goes off, the fix is maybe one or two lines in the agent's context, and not really a big code change. The core point is that allow lists don't really get better over time. They can get stale, but tripwires do get better over time. So here's a loop in practice for us.

7:14

So we track one number, which is the number of investigation threads that our agent is launching per hour for a given test job failure. So one morning, say the number was way above the baseline, and the tripwire pages on call. And that part's important, because a tripwire that nobody sees is practically useless. It pages after the write has already happened, after the limit has been crossed, not before. It's effectively the smoke detector, not the lock on the door.

7:30

The agent has spun up, in this particular scenario, a bunch of investigation threads for dozens of jobs that were all failing with the same kind of error signature. Each thread effectively looked reasonable on its own, but if you took them in aggregate, you would realize that it was actually an infrastructure failure that was causing the same test failure signatures across the board. So the fix in this case was as simple as telling the agent, hey, the next time you encounter something like this, maybe try to correlate a bunch of different failures and test jobs and stuff before launching a separate investigation thread.

7:42

So that correlation is important. And the next time when this happened, it did exactly that, because that example and that line of how it should react and how it should debug things was right there to guide it. So that's three rough primitives. Now, the lens I mentioned at the start is this idea of undo test. And this one's slightly different. It's not something you effectively enforce in code. It's the question you ask when you're sizing the other three. So it's two questions, really. One, can the agent put it back by itself? And how bad would the impact be if it actually got it wrong?

8:06

And this sounds like asymmetric verbs in a sense, but it's kind of different because the verbs ask whether you would notice the failure. And undo asks whether you can recover from it. So if the agent can effectively roll back its own change and the blast radius is acceptable to you, you effectively log it and you let it go. If either of the answers is no, then you effectively need a second key. And the second key is not something that the agent holds itself. It has to be someone else. And there has to be an audit record so that you can track what happened, why the second key was involved, and stuff like that.

8:35

Let me just show you another example of how this works outside of the CI domain specifically. So our agent has a key for one of our feature flag services. So on the canary side, which is basically staging traffic and a bunch of dog food and customers, the agent effectively has the full dial. It has the ability to ramp up a specific feature flag, roll it out to all of the canary traffic, and it can take it all the way from zero to 100. And it also has the ability to look at any bugs that are being filed and toggle it back off and on. What the agent's key is not scoped to do is promote the flag to real production.

8:57

The best that an agent can do for now in a lot of cases is that it can propose that someone actually promote the feature flag in production because it's been tested out in canary. But that's pretty much it. The second key in this scenario is not necessarily a new auth system. It's a scoped key for production and a scoped key for canary. So this is what it looks like day to day for me personally. It's like I ask Cloud Tag in our Slack channel to own the entire loop on rolling out a flag. It can basically ask me a clarifying question on whether this is for internal dog fooding group or early access or whatever.

9:19

And the important part is that I'm not in the middle of any of these things. Every action that the agent is taking is stamped with its own identity and not mine. And that stamp is basically what ties the whole thing together. So these primitives, they compose together nicely because each one asks a different question. The verbs ask what the agent is touching. The rate limit asks how often. And the tripfires catch what happened afterwards. And the undo test is the lens that I personally use to size all these three.

9:48

Now the original cold open delete that I was showing you, the incident itself, the rate limit itself would have capped it at a couple of tens of workloads itself. The undo test, if you would be thinking about it, would basically tell you that you can't undelete a running job in someone else's namespace. So anything past the cap basically needs a human with a second key. So the important part is that you don't need all the checks in every write scenario. Only some of these might be relevant for whatever kind of action you're trying to evaluate.

10:15

And if any of these really sound familiar, they should, because this is effectively some sort of onboarding checklist for your engineers. What can a new engineer touch? How much rope do they get? Who signs off on their operations? And how do we know it's effectively working? We just wrote it for people. It's now the same checklist that we want for agents. So where does all the policy that I've been talking about really live? It's two places, and I feel like you need both. The first one is text. That's prompts. That's your context files. Markdown the agent reads before it can act. And this is where you can explain your why, the intent.

10:51

We've got exactly the same sentence that I was showing earlier written in a markdown file. It works about 80% of the time. The upside is that it's very cheap to change, and you can explain the reasoning. The downside is you have to garden it because the files can grow over time. And at the end of the day, it's just advice. What can a new engineer touch? How much rope do they get? Who signs off on their operations? And how do we know it's effectively working? We just wrote it for people. It's now the same checklist that we want for agents. So where does all the policy that I've been talking about really live? It's two places, and I feel like you need both.

11:30

The first one is text. That's prompts. That's your context files, markdown the agent reads before it can act. And this is where you can explain your why, the intent. We've got exactly the same sentence that I was showing earlier written in a markdown file. It works about 80% of the time. The upside is that it's very cheap to change, and you can explain the reasoning. The downside is you have to garden it because the files can grow over time. And at the end of the day, it's just advice. Text can shape the intent, but there is no enforcement anywhere. The second place is infrastructure, which is the proxy layer for us.

11:44

So the proxy is not reading the prompt. It doesn't know why the agent wants to do something. And it doesn't really care. It will see a delete happening. It will see a budget being crossed, and it will simply return a 403, and that's the whole conversation, really. It's narrow. It's deterministic. It counts, compares, it can allow or deny. What it can't do is explain the why. And a clever prompt injection cannot really talk it out of the rule itself. So you need both. The text shapes what an agent is trying to do, and infra is bounding how wrong it can go.

11:53

So zooming out, this is where the infrastructure really lives for us. Every agent session has its own proxy running right next to it. The agent starts off by reading its own context file, the markdown files that I was showing earlier. That's the text layer. And every outbound call goes through the proxy after that point, which is the infrastructure layer. So the proxy is not, whatever the agent is trying to do, the proxy is only responsible for stamping that action with the agent's identity. If the agent is trying to launch a bunch of jobs in a Kubernetes cluster, the proxy is not following it. What is following is the stamp itself, the identity itself.

12:04

So in this example, the cluster would write the stamp onto the job as a label. And every child job, or anything that's happening afterwards, simply inherits the same identity. And every safeguard that is there in the rest of our systems, they're simply reading that one label. Whether it's ownership, whether it's quotas, rate limits, approvals, tripwires, whatever it is, they're all keyed on the same stamp, and the agent never got to touch it.

12:12

So that's roughly the shape of it. But before I wrap up, I want to spend 30 seconds on why it has to be a proxy layer, a different layer that is doing the stamping on this identity, and not the caller itself. Because without it, the agent can roughly pick a different name. And every limit that you've set resets.

12:17

So say the agent has the ability to set its own identity in a header, right? So let's say it hits a specific limit. And what's the easiest fix from an agent's point of view that it can do? It will just change the header. In this case, it will just say instead of such and such and two. And voila, you just have a fresh budget to work with. Now, in this case, you technically don't have a rate limit. You just have a suggestion.

12:21

With the proxy in the path, the agent never gets to say who it is. The proxy already knows. It's the thing that's holding real credentials. And it stamps every call with the identity that it already knows, not the one that the agent claims. And because the proxy is the one that is stamping, you also get this per-session ID so that you're able to differentiate different sessions that are running, all for you, and see which one's overreacting or which one's not acting as it's supposed to be. So again, stuff that I want you to take home with is, give the agent access to verbs that would fail out loud. And keep the human on the ones that can fail out quietly.

12:28

Two, you put a ceiling on every write, and that has the ability to refill on its own. So nobody's effectively filing tickets. Three, you watch the aggregate and not the individual calls. That helps you understand the agent's behavior. And then you fix whatever you can with the sentence. And four, you use the undo test to size all of these three, or any other primitives that you have, for your own write operations. And underneath all of this is this concept of identity, which has to come from the infrastructure, not from the request. I feel like that's one rule. If you get that one rule right, everything else is just tuning. Thank you.

12:55

And the second key is not something that the agent holds itself. It has to be someone else. And there has to be an audit record so that you can track on what happened, why the second key was involved and stuff like that. Let me just show you another example of how this works outside of the CI domain specifically. So our agent has a key for one of our feature flag services. So on the canary side, which is basically a staging traffic and a bunch of dog food and customers, the agent effectively has the full dial. It has the ability to ramp up a specific feature flag, roll it out to all of the canary traffic, and it can take it all the way from zero to 100.

13:30

And it also has the ability to sort of look at any bugs that are being filed and toggle it back off and on. What the agent's key is not scope to do is for promoting the flag to real production. The best that an agent can do for now in a lot of cases is that it can propose that someone actually promote the feature flag in production because it's been tested out in canary. But that's pretty much it. The second key in this scenario is not necessarily a new auth system. It's a scoped key for production and a scoped key for canary.

13:59

So this is what it looks like day to day for me personally. It's like I asked Cloud Tag in our Slack channel to own the entire loop on rolling out a flag. It can basically ask me a clarifying question on whether this is for internal dog fooding group or early access or whatever. And the important part is that I'm not in the middle of any of these things. Every action that the agent is taking is stamped with its own identity and not mine. And that stamp is basically what ties the whole thing together. So these primitives, they compose together nicely because each one asks a different question. The verbs ask what the agent is touching. The rate limit asks how often.

14:37

And the tripfires catch what happened afterwards. And the undo test is sort of the lens that I personally use to size all these three. Now the original sort of cold open delete that I was showing you, the incident itself, the rate limit itself would have capped it at a couple of tens of workloads itself. The undo test, if you would be thinking about it, it would basically tell you that you can't undelete a running job in someone else's namespace. So anything past the cap basically needs a human with a second key. So the important part is that you don't need all the checks in every write scenario.

15:07

Only some of these might be relevant for whatever kind of action you're trying to evaluate. And if any of these really sound familiar, it should, because this is effectively some sort of onboarding checklist for your engineers. Like what can a new engineer touch? How much rope do they get? Who signs off on their operations? And how do we know it's effectively working? We just wrote it for people. It's now the same checklist that we want for agents. So where does all the policy that I've been talking about really live? It's two places and I feel like you need both. The first one is text. That's prompts. That's your context files.

15:40

Mark down the agent reads before it can act. And this is where you can explain your why, the intent. We've got exactly the same sentence that I was showing earlier written on in a markdown file. It works about 80% of the time. The upside is that it's very cheap to change and you can explain the reasoning. The downside is you have to garden it because the files can grow over time. And at the end of the day, it's just advice. Text can shape the intent, but there is no enforcement anywhere. The second place is infrastructure, which is the proxy layer for us. So the proxy is not reading the prompt. It doesn't know why the agent wants to do something.

16:14

And it doesn't really care. It will see a delete happening. It will see like a budget being crossed and it will simply return a 403 and that's the whole conversation really. It's narrow. It's deterministic. It counts, compares, it can allow or delete or deny. What it can't do is explain the why. And a clever prompt injection cannot really talk it out of the rule itself. So you need both. The text shapes what an agent is trying to do and infra is bounding how wrong can it go. So zooming out, this is where the infrastructure sort of really lives for us. So like every agent session has its own proxy running right next to it.

16:50

The agent starts off by reading its own context file, the markdown files that I was showing earlier. That's the text layer. And every outbound call goes through the proxy after that point, which is the infrastructure layer. So the proxy is not like whatever the agent is trying to do, the proxy is only responsible for stamping that action with the agent's identity. Like if the agent is trying to launch a bunch of jobs in a Kubernetes cluster, the proxy is not following it. What is following is the stamp itself, the identity itself. So in this example, the cluster would write the stamp onto the job as a label.

17:22

And every child job or anything that's happening afterwards simply inherits the same identity. And every safeguard that is there in the rest of our systems, they're simply reading that like one label. Whether it's ownership, whether it's quotas, rate limits, approvals, tripwires, whatever it is, they're all keyed on the same stamp and the agent never got to touch it.

17:43

So that's roughly the shape of it. But before I wrap up, I want to spend like 30 seconds on why it has to be a proxy layer, like a different layer that is doing the stamping on this identity and not the caller itself. Because without it, the agent can roughly pick a different name. And every limit that you've basically set resets.

18:01

So say the agent has the ability to set its own identity in a header, right? So let's say it hits a specific limit. And what's the easiest fix from an agent's point of view that it can do? Like it will just change the header. In this case, it will just say instead of such and such and two. And voila, you just have a fresh budget to work with. Now, in this case, you technically don't have a rate limit. You just have a suggestion. With the proxy in the path, the agent never gets to say who it is. The proxy already knows. It's the thing that's holding real credentials. And it stamps every call with the identity that it already knows.

18:33

Not the one that the agent claims. And because the proxy is the one that is stamping, you also get this per session ID so that you're able to differentiate different sessions that are running all for you and see which one's overreacting or which one's not acting as it's supposed to be.

18:49

So again, just stuff that I want you to take home with is give the agent access to verbs that would fail out loud. And keep the human on the ones that can fail out quietly. Two, you put a ceiling on every right and that has the ability to refill on its own. So nobody's effectively filing tickets. Three, you watch the aggregate and not the individual calls that help you understand the agent's behavior. And then you fix whatever you can with the sentence. And four, you use the undo test to sort of size all of these three or any other primitives that you have for your own write operations.

19:21

And underneath all of this is this concept of identity which has to come from the infrastructure, not from the request. I feel like that's one rule. If you get that one rule right, everything else is just tuning. Thank you.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note