---
title: "Balancing Acceleration and Safety for AI Agents"
speakers:
  - name: John Feminella
    affiliation: Two Sigma
room: "Simons Room (Metropolitan East)"
starts_at: "2026-10-13T09:36:00-04:00"
ends_at: "2026-10-13T09:54:00-04:00"
format: "keynote"
recording: "2026-10-13_0938_Two-Sigma"
duration: "19m 16s"
source: "qwen3-asr-1.7b"
---

# Balancing Acceleration and Safety for AI Agents

> Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): John Feminella (3 vs 0), Two Sigma (3 vs 0). Check them against the audio before quoting.

### 00:00:00 — Introduction & Regulatory Disclaimers

To accelerate innovation without compromising trust, please welcome John Feminella, SVP of architecture at Two Sigma.

Hey everyone. I carelessly balanced my water cinder. My name is John Feminella. I work on AI architecture and engineering at Two Sigma. I'm based here in New York. Before I get too far into things, I have a legal disclaimer slide that I have to show you. It says two things. One, I promise not to sell or market or offer you any at any point, would stop. And two, everything, and this is my own opinion, and not the opinion of Two Sigma. Thanks for indulging me. If I don't say those magic words, the 5G microchips implanted in me will explode.

### 00:00:58 — When Controls Fail: Citibank's $900 Million Revlon Error

So that slide is a good example, actually, of how our industry is regulated, and as a regulated company with lots of controls, I think one thing that's always interesting is to explore what happens when controls fail, because that gives you a good idea about why the controls were there in the first place. So I'd like to tell you a story about that by showing you another people slide. After August two thousand and twenty, a very interesting contract clause suddenly started showing up in a bunch of commercial loan agreements, and this clause essentially tries to prevent a certain scenario from happening.

And it's complicated, but I'll explain it like this. Let's say that you're a lemonade stand business, and you borrow money from a lender. That's why companies stand. And the bank that's overseeing that transaction is in the middle between the borrower and the lender, sends the lender all of the money that you owe in a catastrophic dispute. You probably be mad and want the money back, but the lender might also say, "No thanks, you paid us back and we're all good," so we'd rather.

Something like this happened in August 2020. Revlon, the cosmetics company, had borrowed money under a big 2016 term loan, and that loan was split up into a bunch of pieces across 300 plus lenders at different request. Those other banks, and the middle is Citibank, the administrator agent. Being the agent in this standard transaction means that we're performing the plumbing that connects the borrowers with the lenders and managing the life cycle of the loan, so this transaction: Redline pays City, and then City pays the lenders. A city's not lending its own money. It's just in the middle, moving people's money around.

So here's the plan for August eleventh, twenty. Redline would pay about eight million bucks of accrued interest to both groups, and the lenders' principal would roll into this new facility. That's an exchange of paper, not cash. So the remaining lender's principal would stay put, and the idea here is that only interest should be coming out. Principal should not be moving at all.

The bank had a piece of internal processing software called Flexview. Isn't that a fun kind of a software name? So, InflexQ, the easiest way to do this kind of transaction is to pretend to, like redline, we're paying off the entire loan, and that triggers the correct interest payment calculations to our — you probably guess what's about to happen next. So you're supposed to send the principal portion to an internal wash account that you own as the bank, so that it's closed and recorded correctly, but it never leaves the bank. So keep your eye on that Washington.

What was supposed to happen is that the boxes labeled front and fund and principal should all include the wash account number and should all have a checkbox checked next to them. So unfortunately, it turned out that was what needed to happen, but for various arcane reasons, the boxes were not checked, and although Flexview pops up a warning notification and says, "Oh, money's about to leave the bank. Are you sure you want to do this?" That was where that money was in those billions of dollars because they were making the interest payments. So the three approvers who were supposed to approve that looked at it and said, "Yep, that's the right thing that's supposed to happen." And although that warning notification popped up, the human operator is carrying out the transaction over the phone. Remember, Grandma was being that interesting.

So the principal never went to the washing out, and instead, Fletchview sent the entire outstanding principal of the loan to all of the 300 lenders, about 900 million dollars instead of 80 million dollars. So every lender got the amount that they would have been owed if red line had paid the loan in full, but red line didn't want to pay off the loan, they wanted to hold on to its money. That was a great deal for the lenders because remember, Revlon was in trouble at the time. So as long as we're in trading at face value and getting all your money back on a deal that you thought was only worth sixty cents on a dollar. Is it readable for you?

So they didn't want to give the money back, even after City called them and said, "These types of mistakes give us the money back," and they're arguing why shouldn't have to get the money back because you paid us in full, and that how loans work: You eventually pay them back. And just because you paid it back at a different time than you were supposed to, doesn't mean that we should have to get it back now. So to prevent that from happening, commercial loan agreements everywhere are starting to include including a clause like that one.

### 00:05:13 — Why Safety and Agent Autonomy Aren't Opposites

So, what does a payment error at a bank have to do with AI agents? Well, the bank had this procedure, had these controls, and everybody did what they thought the right thing to do. What's the problem I want to talk about for the rest of this talk? Every day we're giving real agents, real autonomy over systems like Flaskier, and how do we prevent accidents like that one from ever happening without locking everything down, or just so people get lucky that they walk out.

So here's the class of the issue: you can't really know in advance what an agent will do. Right, they're these non-deterministic systems, and that sounds scary until you remember that humans are also complex systems. So they're going to make mistakes too, and so most firms don't choose to say, well, in order to reduce the risk of humans making a mistake, you know, like lock them in rooms and prevent them from talking to each other, and they're all going to double check everybody else. We allow them a certain amount of autonomy, along with a certain amount of control. We give them boundaries, visibility, and we put them in systems and controls that let them do their jobs. So in this sort of model, safety and innovation are not opposite ends of the spectrum, and you must trade off. There are things that you can have at the same time. So, I can even have your game and either two. And when we were thinking about what controls we wanted to apply to agents, we also looked at some approaches that didn't work for us. We don't want a sandbox to lock down everything. The sandbox with no real access to systems is not, is saved, but it's not very interesting or useful.

If you let agents go wild and do whatever they want, that's, you know, maybe powerful and freeing from a cognitive perspective, but the isolation is very real. That push certainly not suitable for highly regulated devices. And then, if you just have policy documents that you just look at, or that are part of your specification for the prompts, that's also not very forceful at runtime. You can't guarantee that will be part of the enforcement controls. So what did work for us was thinking about what mix of controls would get us both safety and innovation at the same time.

So how do we provide boundaries and stop problems, but don't hinder folks from operating correctly? So we have a number of these controls, but today I'll talk about biometrics. When I say controls here, what I mean are the guardrails and shared capabilities that let teams move in ways that don't hinder what they're trying to do, and we want to — we don't want to reach some chasm that we need to cross to let people get their work done before, and then make everyone wait while we build the bridge to get there. So these controls are composable. Each one does one job, and together did let us provide agents with real autonomy at an appropriate level of safety.

### 00:07:48 — Making Agents Legible with an Automatic Registry

So the first problem is that agents were showing up everywhere. We had big teams with lots of agents running. We have small teams of a few. We have individual people running stuff in an uncontrolled way, and these companies appreciate. It could be chatbots. It could be NPC servers glued together in this Frankenstein. It'd be cool though to experiment. It's going to be really useful web tools for folks.

But it's hard to answer when you look at that, a few simple questions like: who owns this? What tools does have access to? What should be doing? What should not be doing? If you can't answer those kinds of questions, you can't govern them. Then you can't really also reuse that. So you're tempted to build multiple solutions to the same problem. You don't know that one exists already. I mean, the firm can't also so govern those in any kind of legible way. So one thing that firms sometimes do here is act some kind of like, don't let anything run until everything is registered and approved and has gone through this very onerous process, and that sounds easy, but if you look at what it does, every idea, even some simple Nobel experiment, it's been a waiting line behind something that's really risky, and we don't — you don't, you don't want to subject everybody who's give some diversity of agents to exactly the same process, so most people are going to give up, or they'll just get pushed to some shadow IT, less governed path that is less visible, and we don't get to see anything at all.

So that's what we're going to get in the first control, is an Asian question. You can't talk to him. You can't see. So the agent registry makes agents legible, and it registers which reviews and discovers the key facts about agents that we want to know for any given. So those are things like which tools it declares, what endpoints it's going to talk to, which product is belong to, who owns it. And humans, other programs, other agents can then use the agent registry to decide what's available at the firm, whether that's useful for them, and to go use that as a source of truth for other questions.

So it doesn't require extra effort on your part. It just happens when you use one of the firms approved agent building frameworks. It's automatically registered. All that information is recorded there for you, and if you want to do a manual registration, you can certainly do that.

### 00:10:00 — Risk-Scored Review for Purpose-Built Enterprise Agents

So the second problem we have is that now we've got all these agents doing all these things, we quickly noticed that what they're doing is really different, and that means our agents carry really different risks, right? So I think that summarizes tiered or linear tickets is really different than a production deployment agent that can touch a running system. And if we use the same, we try to use the same review process to cover all of these agents, then every agent is waiting in the same queue for weeks for human review. And if people are going to wrap around slow processes, then we wind up in the same situation that we have before. So again, you might be tempted to say the answer is something like really detailed, having manual review by all of these different constituencies. Again, that feels safe, but now we've got a trivial summarizer waiting two months for the same sign-offs as an agent-based captioning production system.

And most of them do an H I D. Most of the ones that have value right away, start out as small experiments. You don't want those people to wait in the same line as the stuff that's such a production rate thing. So the reviewers are drowning in these like low risk requests, and the high risk ones have to share that attention pool. So we don't want to trade away that speed if we can help it.

So again, that's not every agent needs the same review. And because most agents are purpose-built, radical, open-ended — when I say open-ended here, you can imagine coding agents like Py or Claude Code, when you open them and you start typing, Claude, builder, AI doesn't know what you're about to ask. It doesn't — it doesn't satisfy this purpose in any particular way. But most of the agents that are deployed in enterprises do have very specific purposes. When you're engaging with that agent, it's to do something specific that's well known in our industry. So they have a very specific reason for existing. They're very service-like. And you can tell these agents apart from each other by looking at what they declare about their tools, the endpoints they access, the controls that they're applying, etc.

### 00:11:58 — Declared Tool Access, Guardrails & Shared Evaluations

So the third problem is that if you know about your agents and what they're doing, you'll quickly notice that many agents don't talk to each other. I'm sorry. They don't talk to just one thing or just one capacity. The more open-ended the HSM board looks like a clock, closer to Pi, the more it can do, a variety and a diversity of different things. 搞個journal API， run做一個web search，那種嘢。 And each one of those is a different opening, a different door into a different kind of risk. And if you give an agent an entitlement for a roll account, then it can do a lot more damage than something that's just going to be a public webpage. The question is not: Can this agent do X, but what's the granularity that we need to solve for the agent to act successfully?

And again, one tempting answer is kind of be heavy-handed here, or to give every agent the same short allowed list of stuff that seems to do, then block everything else. That's safe, but it's also useless. An agent that can't, that can't search or can't call an API or can't do any of those things without asking somebody first can't really do the work that it was intended for. And then teams either give up on those agents, where they find a way around the wall, and both outcomes are costly. So a wide kind of blanket block on everything trades away the innovation that we want to preserve, and avoid making a real decision about this, and as other John checks both a ship and harbor is safe, but that is not what ships.

So the fix here is to let agents clear their purpose upfront, and say what tools they're going to use, or what tools immediate access to. So in the second panel here, we might say that the research summarizer is going to be allowed to use a web tool to access research, but for a release deployer agent, we're not going to allow them to do that because they have no reason to use that, and so, if they try to ask for it, they won't get it.

So we don't need to make ten decisions for every tool. The agents just have to say which tools they want, and you're as the Asian author, you're in the best position to say, I need these four things out of the buyers that are available. So the way we need to enforce that is that when agents run in our managed environments, they look at the registry we talked about before to look up what policies apply to it at initialization time, and that policy gets translated into enforceable statements. Language: Who is allowed to access? Which hosts are allowed to access? What other types of resources are available or not available?

And in this way, nobody's reinventing it every time they need to change the access. Nobody's going to talk to IT or some other group outside of your team. As long as you conform to the risk scores that we talked about before, and you're in the Asian registry, you're good you're okay. So this is how we keep the cost to teams really low for deploying, building, running agents, and we see control architectural works in everybody settings. The next thing we do is we set teams, we're building them a ton of different ways to evaluate agents, and every stack looks different. And some teams don't build anything at all. Sometimes you build like this really detailed staff or how your models work. Sometimes it's sometimes there's nothing.

And I think in general we'd be upset if teams were not writing unit tests, right? We consider revals to be kind of the same thing. If models are changing, that's like a runtime dependency that your application has, and if your agent is and it change models, it doesn't know without B vowels what's going to happen to its to its outputs and to the quality of those outputs? So again, one tempting answer here is let's have some kind of central evaluation methodology that waits for people to be asked to even actually sign up. But now every like prompt tweak, every one line change is going to sit in that same queue. So we don't want to put more decisions on more reviewers, or things like that.

We were we don't want to trade away that innovation. So again, we wouldn't ship software without unit tests. So agents need evaluations for the same reason. And we want to provide this stuff at the level of the firm that's going to be the same for everybody. So team A running evals, team B running evals, they're both going to have many of the same features. So we want to have one shared approach to how we do evals. And I would sure approach be something that's run by the firm and available to everybody for doing their analysis. So that covers all the common parts of evaluation. All you've got to do is describe what's true about your agent.

### 00:16:32 — Preventing Review Fatigue with Runtime Classifiers

And finally, maybe you've had the experience of being asked to lots and lots of individual agent actions when there's a human in the loop. That's fine at a small scale, but once you're operating agents in parallel or in production, this becomes completely onerous and review requests. I mean, basically anybody can be expected to approve them, when after the first few times it is happening, you succumb to review fatigue. You just see yes, yes, yes, yes, yes, and eventually pick yes on something that's you know actually the nuclear bomb. Right, so you don't want to okay, yes, five reflexes, but that's going to happen eventually.

One recent example of this is earlier this year, director of the AI Research Lab told for OpenAI mission to suggest emails to delete from her inbox, then wait for her approval. But her inbox was so large that eventually the agent compacted its contents and lost the part that said, "Wait for my removal." I mean, it deleted hundreds of emails. It didn't stop when she told it to stop.

So again, it might be tempting to try to make us human improve every action, but that's not the right way to do it. I better refer to select these independent, conservative classifiers make decisions about routine calls, and then send the person only the requests that are most important, the ones that most — if you're saying yes, 99.9999% of the time, and that's probably just a rule that the agent should follow, not a question you need to ask to get information from. And the stuff that's unclear or ambiguous or potentially dangerous — yes, surface that you would need to inform — it is definitely just dangerous, deny it outright. So this is a way to not have to do all the review work all the time. So we don't want to take the same approach for both explicit — when explicit rules apply, we want deterministic fast paths. When explicit rules don't apply and there's some ambiguity, you want a classifier to do that work instead.

So putting those five pieces back together, you create a registry so that every agent's visible when the data is created. You score these agents, so human review goes to the right spot. You buy agents on dark rails so that they only touch what they need. You share evals and metrics so everybody is reaching the same bar without having to reinvent the wheel. Then, by the Docker and your runtime, to make sure that classification happens there and not Samuel House, so each of these controls works together. So I'll leave you with one question: Why are there breaks? It's not just so you can stop, so you can go fast and stop safely. So controls for our agents work the same way. Thank you very much. I'm John Feminella.