Process Supervision for Agents That Do Long-Horizon Work

Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): Mitchell Troyanovsky (1 vs 0), Basis (4 vs 1). Check them against the audio before quoting.

00:00:00 — Enterprise Productivity & Human Process Supervision

Please welcome Mitchell Troyanovsky, co-founder of Basis.

How long do you work? Hi Mitch, I'm one of the two co-founders of Basis. We are based right here in New York City, and the thing that we focus on is how can we get agents to do production scale work reliably enough so that we are trusted and used end to end by accounting firms across the country. So a lot of our research really focuses on this: how do we get these agents to do real work in the real world, so.

I wanted to start with a question, which is, if you're at this conference, honestly, if you're watching this talk, you probably increased your own productivity a lot. You're using co-work or Codex or Thoughtcode, whatever it is — and yet, if you go look at the average company, the average enterprise, the average firm, their productivity hasn't gone up ten x, a hundred x. I know my productivity has gone up maybe 100x, but if I were working inside of a company, it would be anywhere near that order of magnitude, and so why is that?

At Basis, one of the things we have is that humans are already used to working with non-deterministic systems. It's just the systems are normally their cores, not their computers, and so I actually think to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies. So, how do organizations scale their own productivity? Right.

Organizations need to trust that the people doing the work at every level are doing it in a way that is going to produce good outputs consistently at scale, right? If you look at this image here, here I'm showing an accounting firm, but this could honestly be any company, traditional company. The person who is the head of the firm, they have literally no idea about the exact day to day of the people at the very bottom, and yet somehow, they trust the work is going to be done. How does that happen?

It doesn't happen through deterministic systems, right? People are not deterministic; they're intelligent entities. And instead, there are processes that are put in place, and those processes — if we believe the people who you know put them in place think that those processes will generalize to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process.

This is true, by the way, even in the smallest steps. So in this example, earlier, I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm, but even within that process, say you wanted to see how do I reconcile cash? Well, I need to get the bank data, and even that has a process. The process could be really basic. It could just be used approved software to extract the data, but there's still some process that is put in place. You wouldn't expect the firm to just let anyone Google, you know, extract PDF, and then just use whatever they want to go extract it, because now you wouldn't have the process.

00:03:42 — The Competency Gap & the Process Gap

Now let's talk about agents. As I said earlier, everyone already has cowork, ChatGPT, Codex, Claude. But where is that productivity? Our opinion is there are two main problems that come with what I'll call coding style Asian products where you're working with an agent and you just tell it to go and do something.

The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user. Right. If I'm the user, I now have to manage my skill and memories, and think about how to — when I should use the agent, I have a theory of mind over it. This is really hard. For those who are here, maybe you guys find it easy, but I promise, for the wide majority of knowledge workers, they have no idea how to do any of this.

And there's the second part, which is there is a process step, right? If you are saying, "Hey user, here's your agent." Do what you want. Well, now suddenly the way the work is being done is essentially up to whoever the user is to customize the way the engine behavior, which goes entirely against the way that organizations. The process should be coming from senior leadership down, not many individual employee deciding how they're going to do their job, right? If you going back to my example, if every employee just made up their own way of extracting data, you know, even back in 2018, that wouldn't be a good idea.

And so we believe strongly that the key to building production agents that actually do work end to end is solving these two problems. Is to solve the competency gap by making it simple to the user. You shouldn't have to know it's an agent, or you probably know, but you shouldn't have to really feel it. It should be as simple as just working with a coworker: you just tell it what you want it to do, and all the things you expect to work should work. And the second is the process gap. You need agents to be reliable, not having a showbiz genius one day and forgetting something the other day. It's not about being perfect. Humans are not perfect, right? Nobody is looking for perfect. What you're looking for is reliability and consistency and trustworthiness.

00:06:10 — Why Reliable Agents Require Opinionated Processes

So, what does that actually look like? We think that to be simple and reliable, where you no, you don't have to think about what to do. The agent just do it for you without the user knowing the internals. To do it the same way consistently, you have to be — let me introduce, just to show you, actually, one slide back here. I want to point out that getting it right isn't actually the main task, is again, humans are imperfect themselves. The point is to do it in the similar way. It's okay if you can't do something, but you should be consistently able to not do something, and then you get better, just like a person does — if you suddenly one day are an expert at something, and then you miss something very basic that would be expected, now you're not relaxed. So you have to give me an example. What's an example in a minute?

Let's say we have an agent, and we need to go and read a PDF. So in this case, what is the best way to read a PDF? There's many ways: you can throw it into a VLM. You can go write some Python code, use an open-source OCR library. You can work with one of the many OCR specific model providers. You could do an OCR, and then you could have the VLM check its work. You can have five DLNs that work. You can do a hundred DLNs and take the best of them. There's literally infinite possibilities here.

And so, if you just said, "Agent, extract Excel," or "extract PDF," you might get good outcomes, but they're not going to be the line. Right? Not because the process it did was bad, but because the process it did was not opinionated. So the way we develop opinions is you use initial intuition, right? Some things you know what opinions are from expertise. For example, if I'm trying to get an agent to do tax work well, I know that if the agent goes and reads the primary source before returning to the user, it is more likely to be accurate. I don't need a set of emails to tell me that. I don't need to divine that from first principles. We know that, right? I know that if I'm an Asian, make a PowerPoint, if it takes a picture of the PowerPoint before it returns to the user, it is less likely to have a formatting issue. That is obvious.

And so there are lots of ways to get all opinions. And one of the key points here, I'll just make before I move on to talking about how to use the opinions, is that the pains are not just about accuracy, right? We just talked about the example of taking a picture of a park. Well, that adds latency, right? You may not want that. Maybe the agent only makes formatting mistakes one out of a hundred times, and you're okay with that, and so therefore you don't have to take a picture of the PowerPoint. The point isn't exactly what the opinion is, and this talk we're going to focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale into production.

00:09:16 — Why Outcome Evals Fall Short on Long Horizons

So, say you have a bunch of outcome-based evals. And when I say alphabet-based evals, I mean what we traditionally think of as evals. It has a right answer, we have some rubric we're trying to judge the things you did. How do you know that the emails generalize to production? Right. You might have a thousand cases or a hundred cases, but in production, if you're doing real work, you're going to have hundreds of thousands of situations, and that's only going to increase as the trajectories get much longer because the amount of decisions that are made in trajectory just grow exponentially as the like long horizon abilities of agents increase.

And so I can't prove — if I if you look at all this distribution of trajectories, we can't prove every one of them is correct. You don't have to round to forever to get it. But you can potentially look at the trajectories to decide if the behavior, the opinion that you have, is actually happening, and even then infer that if it's actually happening, it's more likely to generalize to production. It's not perfect. You don't know for sure, but at least you are having an opinion.

Let's run through a real example. So let's say that I for behavior, this is an accounting example where if there is a mistake for in the bank reconciliation, I don't want the agents like Post and Mitchell to make a tie. I wanted to sort of explain the difference instead of putting a plug. Again, this is subjective. You might want the other way, but what you don't want is a customer getting one way one time and another way the other time.

And so, what would you do? Well, first, let's understand what governs the behavior. Right, the behavior is not just you write the two sentences into your system prompt, and it just works. That might work for very simple narrow agents, but when you're building these big agent systems, the behavior is actually a combination of the system prompt, your progressive disclosures, yeah, potentially tool descriptions, obviously, the training of the model. There's a lot of things that govern the behavior. And one tip, by the way, I have for when you think about building a production agency: really think of the context that agent is going to see as it's working as runtime training data. The word prompt, I think, trivializes it a bit, right? At the end of the day, this is the data that the agent is being trained on at runtime — not literally turned on, but effectively turned on, and so you want to care about every single token and understand how that data caused the agent to behave in certain ways.

00:12:01 — Behavior Specs as Ground-Truth-Free Trajectory Rubrics

So, when you're putting behaviors in the production, it might be very difficult for one behavior to put it in exactly one you know paragraph, which you know exactly because again the behavior is all of the data at once, not just one thing. And one of the things that we developed at Basis was this concept of a behavior stack. And the idea of a behavior step is essentially saying, before we update the context or write the context, what do we actually want? What is our intent? You can kind of think of this a little bit as like a product set for the age era.

And you can't write — you shouldn't be writing every detail down. You should be taking the most important opinions and putting because if you put everything, you'll be duplicating a lot of states. So you find the ones that matter the most, and then you can use them as a way to align your team from a spec perspective, but then you can also use them as a rubric. And you can say, in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant? And that, if so, did it actually exhibit the behavior? And this way, you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome-based evals around truths and Rubric's path. Those are really important, but you cannot scale those anywhere near as fast as you can scale the ability to supervise what the behaviors of the agent are. Let's just take a quick example here.

00:13:40 — Trajectory Forking & Subagent Simulations

This is — so this is actually the second concept, which is called forking, which we combine here with behavior specs. To think about forking, if you've ever used block code or Codex the web apps, there's this little fort icon under each message where you can essentially create a copy of the agent using the same thing, which works really well on your local computer. It's hard to build a system that does this in the cloud perfectly with recreating the database environment and recreating the prompts, all the all the different environments.

But forking is another tool that we use to check the behavior. So flow that you can imagine is saying, we have trajectory. It could be in a debug set. It could be in testing or in production, and it failed the behavior eval. And now you're trying to understand why: why did it fail, failed behavior we got? Why did not exhibit the behavior? And then you can go into the trajectory, go right before the point in which the agent made the decision that was in contrast to the behavior — assuming again your behaviors, and it's actually producing signal, and then you can board it, and when you forward it, you can ask the agent, or you would work it right after you made the decision — you could ask it, "Why did you do that?" Or you can go and update the context of the agent, and see if that improves the behavior. And so this 14 even start to actually take the signal and start to figure out, okay, how can we then improve the harness to now leverage the signal?

Versus, if you're just looking at signal from an outcome-based eval, you can't do that anywhere near as easily because it failed the outcomes. How do you know what? You have potentially a thousand inference calls in your five-hour-long agent trajectory run.

And then the last tool that I'd like to talk about — this is actually my favorite — which is called simulations. It's gonna sound silly, but I promise it's very effective, which is that you can, when working with coding agents, ask the coding agent to make a subagent that pretends to be your production agent. And you can tell it to not cheat, and give it the system prompt and the skills, and you render everything perfectly. And what you can do is you can ask it what it would do in some scenario. You can give the situation and see if you can recreate the situation that you saw in your eval reproduction. And if you can recreate it, now you can tweak your context to see if the subagent would change its behavior based on your tweaks.

And the reason to do this—it's obviously far worse than working, but you do this because this is much cheaper, much faster, and allows you as an agent builder to really easily iterate on ideas, without needing to go through the expensive process of actually forking an entire environment. And you can do this today without building it. I did this for literally year and a half plus before we built forking into our internal system.

And the main idea here is understanding that if you look at the distribution from earlier, you're trying to understand is, but he is the agent generalizing to that distribution? And so you don't know purely based alchemy bells. You can try to choose the best alchemy bells that should represent the curve, but you don't really know, and so supervising the process through some of these methods, gives you some additional signal in order to understand if it's generalizing problem. And what's really cool is as you start to get more signal, you can actually start to close to look. And an opinion is what lets you do that, right? If you don't have opinions about your system, you're not going to be able to develop that thing.

00:17:21 — Earning Organizational Trust Over Move-37 Creativity

And now, if we go back to the firm, we talk about, talked about supervision, we talked about how the agents, if your opinion, it can be more reliable, and it actually can be simpler.

What that means is the agent maybe it starts doing something narrow because people will work with it. They see that it's easy to use, and they see that it's reliable. As it gets better and better, they will trust it more. It's just like a coworker. You have a coworker, Daymon: Imagine someone sends you a PR. You're not gonna — if it's you're looking at two axes. Whenever someone sends you a PR, it's these two axes: What's the blast radius of the PR, and what is your perception of how competent that person is, right? If the least competent person at your company is setting your PR that's editing a core service, you're going to look at it. If the most competent person is sending you a PR that's editing an internal document, you may not look at it. And so this idea is true with humans working with agents, just like it is with humans working with you.

And I think I just want to make one last point to close here, which is that one of the things that people say when they hear this is, "Well, wait a second! If you let the agents just do what they want, won't they find a better way? Won't they move 37 your work?" And the answer is, they might — but that's not how you scale GDP and scale output. Nobody wants their agent to move thirty-seven their tax return. They want it done reliably at scale in a way that they can trust, and in a way that their people know how to work with.

And so the thing I would leave you with is for agents to diffuse, they need to be simple; for teams to trust them, they need to be reliable. The way you get to simple and reliable is you develop opinions on how the agent behaves, and then opinions allow you to build that additional signal that helps you start to close the loop so that you can achieve self-improving agents of production. Thank you very much.

What if AI could do more than generate?