Who verifies the software factory?

Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): Sonar (4 vs 3), Addy Osmani (1 vs 0), Anthropic (5 vs 1). Check them against the audio before quoting.

00:00:00 — The Review Gap & Speaker Introductions

Having a good event, I'm very excited to be here and talk about who verifies the software factory. Before we get started, can you give me a quick show of hands? Raise your hand if your quoting agent has pulled opened any pull requests in the last one week. Almost everyone. Thanks, and keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged.

I do see a few people. That's excellent. So what I'm going to talk about is the gap between the two — merging requests without reviewing manually, and you know merging requests after doing some reviews. So my name is Manish Kapoor. I'm the product and developer relations team at Sonar, and I'm excited to be here with my guest speaker, Addy Osmani from Anthropic. Addy has been, had been at Google for about 14 years. Most of you probably know him, and he's written lots and lots of content and articles. I learned a lot from Addy, so I'm really fortunate to have him here stand next to me and share his wisdom with all of us here today. He recently joined Anthropic, and Ali, so congratulations on your role. We like to tell you what you're working on and also what your role is.

Great developers, curators, building with live code, and anytime that we discover there are opportunities make it even smoother, we bring that right back into the product. We ship a couple of times every single day, so we're very interested in trying to make sure that you, the best possible experience is possible.

00:01:45 — CI Merge Failures & Lights-On vs. Dark Factories

Thank you. All right. So let's start with some problem and context settings. Like, there are two numbers on the slide. Circle CI analyzed twenty-eight million CI workflows this year, and they noticed that the throughput jumped by about fifty nine percent, but yet three in ten attempts to merge that production arcane, and this is started from one independence that from CircleCI, and I think I'm seeing the same trend from most of our customers that I speak to every day.

This is a quote from Addy: A software factory is harnessing loops at scale. This resonated quite a bit for me. If you have not read that article, you can look it up. It's a pretty educational article and talking about future, how the future will look like in terms of software packages. So I would like to expand on what we were thinking when we wrote this article about loops and harnessing the loops that's here. Absolutely. So a lot of folks here probably heard this term soft factories, and there's two types of factories that are at least worth keeping in mind.

You know, software factory is harnessing loops and scale. A lights on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person so apply your judgment and taste inside the picture, and then we have a dark software factory, which is a little bit closer to how you know car factory, the robotics might be working. It's constantly doing the work, so it's scoping, building, checking, and you as the person move down further to think about intent, and you're pulling, you're really raising the exception. That's sort of what I was thinking. I think a lot of teams are somewhere on this spectrum, and our own journey to figure out like how do I get further and further into automation, the way that I can trust it. So.

00:03:41 — Shipping Faster with Confidence & Verification Loops

AI coding is now basically a solved problem. We all know agents are there. Eighty-five percent of the organizations are already using coding agents. They go to 95 percent, and 95 percent of the people, seventy-five percent of the seventy-six percent of the developers that write AI code, they are saying that we reduce that is impossible for them to verify, so they are spending more and more time reviewing the code. So, what do you think is the most pressing question that engineering leaders have right now? So I think that teams have gotten better at working with agents over the last year. Models harnesses have also gotten better, and what I see a lot of leaders being interested in is how do we enable you to ship code faster with confidence, and that confidence piece is very important.

Agents can now generate ways for you to check your code much more deeply than ever before. If you think about the things that enabled us to all good code, you know, high quality, those were things like good tests, benchmarks, anything that gives us confidence. So agents are able to help us generate that a little bit more easily these days, a little more quickly. We get around those things, but I think that those checks are one of the most important things we should be spending our time on these days. Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. If you've got that in place, you can make sure that the code is both correct and complete. That can easily two to three by all of your outcomes. Absolutely, I think we're confident, we've been running an ad for consensus class, and you really need to build it up now that the coding problem is solved.

00:05:26 — Merging 30,000 Daily Changes at Anthropic

So I actually had yesterday night, so this might be a surprise. Yesterday night, I read a post from Anthropic, and I encourage all of you to read it. It also depends on the scale at which anthropic is operating. They are merging about thirty thousand code changes every single day. They used to do about three thousand or so, twenty five hundred or so code changes just a few months back. They have from 3,500 quote changes to 3,000 or 30,000 quote changes per day. That is mind-boggling. So, what is possible at an topic and you?

Yeah, these are great questions. So we were we were merging about 2,500 changes all the way back in May, and even for our recent refresh with Cloud AI, we put out a case study saying that we kind of three by the performance of it. That was accomplished by merging about three thousand requests in a very short space of time, with engineers, product designers, a number of different functions contributing to it. So our workflow for being able to get verification and code review in place is, we first of all have laws to first pass review. What's very important in there is making sure that it has a way to actually try the experience out, similar to a real user. So that can mean end-to-end tasks, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence.

People usually ask us, okay, well, what happens if you know you just discover a bug or a gap in functionality? Well, we have hooks that will then go and run on any edits. They'll study the dips and allow us to get this group in place where we are correcting the PR as quickly as possible, so we have Claudio, doing the review. We have a code review product that you can also check out is pretty good, and the idea there is we're trying to automate code review as much as possible. This is something that we use ourselves.

And increasingly, what's important is for your teams to be thinking about, how can you, we talk about taste and judgment so much these days, right? It's important for you to be trying to codify what you consider best practice and take in your team's shared play, Claude, me, agents anywhere where the agent can then consult during code review, and then people are responsible for checking on the agent's evidence. So, how do we know? If I'm building out a piece of UI that's going to be you know used by a user in the browser, I want to make sure that it's tested in the browser. Maybe generated me a video of that experience where it's been clicking around, and gives me confidence. Well, okay, well, this works. If I care about mobile or desktop experience, maybe it generates me screenshots or videos of that, but basically, you want to get to a place where the AI is giving you as much verification evidence as possible.

Yeah, verification is key. I think you need to verify every single the software products produce, and that's what we are going to talk about today. So another discussion.

00:08:20 — Sonar Survey Insights & Moving Toward Dark Factories

These stats, by the way, are from the State of the World Survey report that will be coming out very soon from Sonar. We surveyed about thousand more than thousand developers, and we found some interesting data, which will give us insights as to what's happening in the industry today, and then we are today. So 84% say AI teachers gets features and fixes to production faster. The poll requests are getting larger and more complex, and you know, yet only thirty one percent more code is reaching the production without review. So that's a little bit of a scary fact, particularly for production applications in the regulated industries, in the mission critical applications, where you absolutely need to make sure that you know, you have to verify that code, and you're confident in the code that is being shipped. So, what are the, where are the next games going to come from, Addy?

I think the next game is going to be a combination of us really nailing verification right, as well as model quality. But models are already, like you take a look at Opus 5.5, model quality has already gotten to a place where it's pretty good. And I would say that us being able to share best practices about that verification step is going to become increasingly more important because if you tell people, yeah, just like verify the work, the next question is, what does that even mean? Yeah, right. And so I think that the conversation needs to evolve into specifically what kinds of tests are useful. We lean a lot on end-to end tests, property tests, any visual verification tests—things that can help us just get that confidence to merge those changeset automatically.

You mentioned light factories. I want to go dark, and that's the way the direction we are headed. Our survey results show that you know today five percent of the companies are using software factories. They can get to 34 percent, and surprisingly, 29% in the financial services. How many of you are using software factories? Please just hear us. A handful of you, and how many are planning to use Zapf factories by next year? Yeah, that's a good question. How do you define a software company?

We'd like to take that question. Yeah, yeah, sure. So we talked about definition of software factories just a little bit earlier. A software factory is basically moves scale, and when we think about this, it's all about like, what end of the spectrum are you at in terms of how much people are still involved in the process? The more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark, and I think that yeah, most people are still somewhere in that spectrum, are not quite as this data shows at dark points.

Yeah. Just to add to what Ali said, I think software actually think of it like a production line, where you start defining the intent and your specifications, then you start writing the code. So it's an assembly line where each of these software development lifecycle steps happen, and automating them at scale, where this pretty much in the dark factory, there would be no lights on. It's autonomously working, fully autonomously working. A light factory has, is where people are involved in the process at every step, or most steps. Now, given the dark factory, the goal would be to turn on the lights and let the factory run independently, 24 by 7.

00:11:26 — Zero-Trust Multi-Layered Verification & Enterprise Guide Rails

All right. So you talked about verification. I'll talk a little bit about what we think at Sonar about verification. If you saw a team of earlier today from our CEO, he talked about verification should be multi-layered. We absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach, and verification should be done in every loop, whether you are operating inside the agentic loop, or whether you're working inside the CI verification room, or whether you're doing code maintenance. In each and every loop, it is important to verify the output of the factories or the agents that, output that the agents are producing. So we look at it in terms of layers and think of it like you know this first layer, which is a deterministic layer, which catches bugs, security issues, things like SQL injection, buffer overflow bugs and all of those are caught, and then there's a second layer where agentic review happens, where agents are used to like, Andy was talking about agents review that code, and in that agentic review, they catch logic and intent issues which the deterministic layer is not well suited to capture.

So we use each layer is targeted at catching a specific types of issues. And then the third layer could be a software supply chain security layer, where you look at all the CVEs, all your malware detection happens, all your malicious package detection happens, all of that happens in the third layer, so it's a layered approach where each layer is tuned and optimized to catch a certain type of risk, so we absolutely recommend doing a multi-layer verification.

So there's another stack here. Twenty-eight percent, you know companies have approved agents and shared controls, very few shared controls, and five percent are operating software factories around the world. What are your thoughts there, Ali?

In terms of, so I've talked to so many companies where a couple of months ago, one of the biggest worries was they're allowing their teams to go off and play around with agents, and they got to a place where there were hundreds, thousands of agents in some cases. We just didn't know what they were doing. What are these agents actually doing? Touching all of our systems, accessing all kinds of data. Maybe there's a breakthrough. And so, a lot of enterprises think about concerns like data identity and telemetry, being able to track what's actually happening. And I'd say that the three biggest things for a lot of companies are how to think about control, how do we think about permissions, and how do we think about the sandbox? Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that shouldn't be there?

Yeah, we so now we call it guide rails. So instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the rail tracks, right, a specific path to take and prevent them from going out of bounds, or so to say, going out of the sandbox, like hiding the system. So it's very important to have benchmarks for the engines and the world of the engines that you're working with.

00:14:29 — Lightning Round: Labeling PRs, Skills & Safe Merges

So let's be coming to a close now. As my timer is running, we just have a few minutes left. Let's get into a lightning round. One sentence response for each of these questions, and you've seen the questions already on the slide, so anybody wants analysis? As long as the time permits, light or dark factory for most by end of 2027? Yes or no? I think we're going to see a light factor with some dark corners. I don't think we're going to get to everybody in this room using dark factories entirely, but it's going to be — I would guess Antopolis will do it this year or next year.

All right. Second question. Should agent-written code be labeled in the PR? I personally think it should. Historically, we have felt a great deal of ownership around our code, pride around the fact that hey I wrote this thing I'm putting my name on it and at Anthropic like we think about this a little bit different. We've evolved to the point where we don't really mind if Claude is like labeled as, hey, I'm the one that wrote this, because we're a little bit more focused on the outcomes and making sure that the code, the features, and what we're shipping to users is the right. And so, I personally think it's fine for the code to be written, labeled as being coming from an agent, and that can also help us with things like figuring out: okay, well, the agent wrote it. What are some of the constraints and extra checks that we need to keep in mind?

I have the same point of view. I think in the short run, at least in the foreseeable you know six to nine months, we should be able to PR. They will be confident enough in our factories. We do want to know and track it, and it's also important for the financial customers here in terms of auditability. If you know if you label it, you know who's writing the code because we are at the end of the day. Yeah, absolutely. Accountability is the most important. Like we still label it. Who's the owner for this? I might have kicked off a cloud run to complete a piece of work, but I'm still on the hook. If something goes wrong, I have to. Yep. Yep. Keyboard. Yep.

The next question, actually, I'll involve you here because this is a leadership session. One skill a junior developer or leaders should build on this year. I think that you should push your teams. If you're a leader, you should push your teams to really think hard about what verification means. How are you already approaching things like testing? What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough, so merge on their own, and so figuring that out. If you're a junior, it's the other side of that, like actually getting more experience and different flavors of tasks, whether it's like end-to-end tests, property tests, etc. And the last, you actually marked following the rules and going beyond one's limits, but that's the last point.

Thirty seconds left. We can go a minute or two. The agent opened the PR — it is safe to merge men — you have full confidence that things are going to go wrong, or that you've mitigated those as much as possible.

All right. Here's my last slide, and then we'll wrap it up. So, what's the trade — teams that are shipping fast but not shipping blind? I think that three key takeaways for you: A, writing code is the fastest fast to write is not fast to ship. You have to basically count what is getting merged. Look at what is getting merged, and make sure it doesn't escape the sandbox and things like that before it gets into the production. Verify everything, and autonomy is on you. We will get to software factories very soon, if not next year. You need to put an independent automated verifications inside each loop of your factory, make sure that you can trust the output that the factory is producing.

With that, I'll wrap it up. Thanks for attending our talk today, a conversation with Addy. Again, I really appreciate you taking the time and joining us here in San Francisco. Visit us at the big red booth, the Sonar booth, in the expo hall, if you have any further questions or would like to talk more with them. Thank you.