Because coding agents generate pull requests far faster than humans can read them—and reviewers accept wrong AI output four times out of five—shipping confidence must come from independent, multi-layered verification harnesses and runnable behavioral proofs rather than manual line-by-line diff inspection.
Because coding agents generate pull requests far faster than humans can read them—and reviewers accept wrong AI output four times out of five—shipping confidence must come from independent, multi-layered verification harnesses and runnable behavioral proofs rather than manual line-by-line diff inspection.
Software delivery talks confronted the review bottleneck head-on: Sonar reported a 2,450% explosion in AI-generated code yielding only 30% more releases because human reviewers succumb to 'cognitive surrender' (agreeing with wrong AI output 80% of the time) while multi-step agent accuracy decays exponentially across long horizons. Teams moving toward lights-on and dark software factories replaced manual line reading with executable proof loops. In the verification workshop, external MCP gates (`Ctx`) refuse agent self-reports until deterministic CLI and Playwright checks verify backend persistence, making verifier suites more durable than the application code itself. At Anthropic, merging 30,000 daily changes relies on automated first-pass reviews, edit hooks, and browser-driven screenshot and video evidence; Sonar pairs CPU static analysis with GPU generative checks and CVE scanning across every loop; Greptile inverts author and reviewer model families (+10% bug catch) and boots full-stack VM sandboxes for vision-model UI QA; Linear wires verification skills and product-history review agents into unattended bug fixes; GEICO enforces deterministic nine-step artifact gates; Untangle caps builder-reviewer loops at three iterations; and Coinbase runs parallel agent reviews inside gated DAGs. Against pure automation, Untangle insisted that a human product owner watching each feature video remains irreducible, and Featherless warned that unowned generation lobs 'slop grenades' onto peers.
“A lights on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person so apply your judgment and taste inside the picture”Addy Osmani · Who verifies the software factory? · 00:01:45
In Who verifies the software factory? (00:03:41), Manish Kapur, Addy Osmani argued that with raw code generation largely solved, the highest-leverage engineering investment is building automated tests and benchmarks that let agents self-verify correctness and completeness for a two-to-threefold outcome gain. In Who verifies the software factory? (00:05:26), Manish Kapur, Addy Osmani argued that scaling from roughly 2,500 to 30,000 daily merged code changes at Anthropic relies on automated first-pass Claude reviews, diff-triggered edit hooks, and agents exercising UIs in browsers to produce screenshot and video verification evidence.
In Who verifies the software factory? (00:08:20), Manish Kapur, Addy Osmani argued that achieving high-confidence automated merges as unreviewed production code rises requires concrete behavioral checks—specifically end-to-end tests, property-based tests, and visual verification—rather than relying on manual line-by-line diff inspection. In Who verifies the software factory? (00:11:26), Manish Kapur, Addy Osmani argued that enterprises should enforce zero-trust, multi-layered verification across the agentic loop, CI, and maintenance by combining deterministic static analysis for bugs and SQL injections, agentic review for logic and intent, and supply-chain scanning for CVEs and malware.
In Who verifies the software factory? (00:14:29), Manish Kapur, Addy Osmani argued that explicitly labeling agent-authored pull requests preserves regulated auditability and triggers stricter automated verification constraints while keeping a named human engineer accountable for production outcomes. In The Machine that Changed the World, vAI (00:05:15), Tariq Shaukat argued that human-in-the-loop review fails to convert a 2,450% surge in AI-written lines of code into proportional delivery (yielding only 30% more releases) because reviewers exhibit 'cognitive surrender,' accepting wrong AI output 80% of the time.
In The Machine that Changed the World, vAI (00:08:01), Tariq Shaukat argued that compounding step-level errors flatten long-horizon agent reliability—dropping 95% per-step accuracy to roughly 36% by 20 steps and sharply curtailing METR's 16-hour task horizon when required accuracy rises from 50% to 80%. In The Machine that Changed the World, vAI (00:10:56), Tariq Shaukat argued that applying Toyota-style quality engineering to agentic software factories requires four multiplicative pillars—explicit intent, in-loop guide rails, independent verification, and auditable evidence—where missing any one pillar collapses overall confidence to zero.
In The Machine that Changed the World, vAI (00:13:29), Tariq Shaukat argued that pairing CPU-based deterministic static analysis with targeted GPU generative checks provides independent verification that left surveyed SonarQube users 81% more likely to see positive security impact and 76% on outage frequency.
“Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct.”Addy Osmani · Who verifies the software factory? · 00:03:41
“If I care about mobile or desktop experience, maybe it generates me screenshots or videos of that, but basically, you want to get to a place where the AI is giving you as much verification evidence as possible.”Addy Osmani · Who verifies the software factory? · 00:05:26
“We lean a lot on end-to end tests, property tests, any visual verification tests—things that can help us just get that confidence to merge those changeset automatically.”Addy Osmani · Who verifies the software factory? · 00:08:20
In Your Coding Agent Has a Context Problem (00:12:47), Tom Moor argued that combining enriched intake context with agent verification skills and failure feedback loops makes unattended implementation reliable enough to auto-fix small triaged bugs locally or in the cloud. In Your Coding Agent Has a Context Problem (00:14:07), Tom Moor argued that rather than checking code logic in isolation, a context-aware review agent joins PR diffs with original customer intent, session decisions, ownership, and historical reverts—catching rejected UI patterns, inverted label hierarchies, and DB transaction risks.
In How to build an AI code review agent: from the creators of Greptile (00:08:42), Daksh Gupta argued that because model families exhibit distinct post-training blind spots and fail identically during generation and review, pairing an authoring agent with a different model family for review increases bug catch rates by 10%. In How to build an AI code review agent: from the creators of Greptile (00:14:04), Daksh Gupta argued that booting the full application stack inside configurable dependency-rich VM sandboxes with documented user-flow maps allows computer-use and vision models to catch layout and UI regressions that static diff inspection misses.
In Beyond “Done”: Building Verification Loops for AI Agents (00:19:52), Nick Nisi, Zack Proser argued that application-specific deterministic verifier CLIs become more durable assets than the application code itself because they allow teams to rewrite or port implementations while proving behavioral equivalence. In Beyond “Done”: Building Verification Loops for AI Agents (00:02:47), Nick Nisi, Zack Proser argued that shallow UI-only assertions produce false-positive agent completions when optimistic DOM updates mask missing backend or storage persistence across page reloads.
In Beyond “Done”: Building Verification Loops for AI Agents (00:30:02), Nick Nisi, Zack Proser argued that emitting explicit, state-detailed error messages from application and test tooling turns failed verification runs into closed feedback loops that agents can diagnose and self-heal autonomously. In Beyond “Done”: Building Verification Loops for AI Agents (00:33:40), Nick Nisi, Zack Proser argued that defining tasks by an executable proof command rather than step-by-step procedural instructions improves frontier agent performance by constraining the target outcome while leaving exploration paths open.
In Beyond “Done”: Building Verification Loops for AI Agents (00:38:07), Nick Nisi, Zack Proser argued that pairing deterministic proof scripts with fast System 1 judge models evaluates semantic UI correctness without brittle string-matching contracts breaking whenever confirmation copy changes. In Beyond “Done”: Building Verification Loops for AI Agents (00:10:45), Nick Nisi, Zack Proser argued that centralizing verification authority behind an external MCP web service decouples workflow completion gates from agent self-reporting, marking tasks satisfied only after independent CLI and API checks pass.
“Between the base, the base context, being able to verify and being able to see failures, we see a massive amount of success in our team.”Tom Moor · Your Coding Agent Has a Context Problem · 00:12:47
“It starts with that original issue and that base context that we created with the customers, the original intent, and then it goes and looks for product history, so things like previous decisions, reverts, regressions and incidents in the same area.”Tom Moor · Your Coding Agent Has a Context Problem · 00:14:07
“And so you want to invert the model. If you use hard code to write the code, you want to use the GPT model to review it.”Daksh Gupta · How to build an AI code review agent: from the creators of Greptile · 00:08:42
In Inside the Fable War Room (00:13:46), Ford Prior argued that executing the nine-step modernization workflow across cloud code-dive agents and laptop-hosted enterprise MCPs relies on deterministic gate checks verifying that each step produced its required artifacts before advancing. In Trusting Agents in Policy-Driven Workflows (00:03:19), Evan Kormos argued that modeling gated approval flows as directed acyclic graphs (DAGs) lets agents execute previously cost-prohibitive independent reviews in parallel and aggregate their results without bypassing required compliance gates.
In Your job is now "agent manager" (00:00:07), Ryan Carson argued that builder agents run automated review loops capped at three iterations alongside child QA processes, escalating unresolved bugs to a manager agent and only then to the human.
“And just an interesting detail, and this is something that the code monetization plugin does: is it does like a deterministic analysis of like, did the previous step create the artifacts that needed to include the file?”Ford Prior · Inside the Fable War Room · 00:13:46
“Previously, some work used to cost too much or be too complex to coordinate in this way. Now we can run independent reviews alongside each other, bring results together, but keep the required checks in place.”Evan Kormos · Trusting Agents in Policy-Driven Workflows · 00:03:19
“We typically stick to a three-review max, where we allow the builders to loop about three times with reviews.”Ryan Carson · Your job is now "agent manager" · 00:00:07
No operational pattern at AI Engineer NYC went unqualified. Across the corpus, 2 counter-claims pushed back on naive versions of this shift—highlighting where judges introduce variance, where caches break, or where human judgment stays irreducible.
“to prevent the slop grenade from happening, enforce that people should own what they hand over or generate.”Eugene Cheah · The AI Bill Is Too Damn High – Tokenomics 101 · 00:41:48
“Every builder records a video of the feature being used and passing tests.”Ryan Carson · Your job is now "agent manager" · 00:01:40
Where does human review belong once coding and workflow agents run autonomously?
Scaling AI agents safely without line-by-line human code review requires giving each application deterministic, outcome-driven verification harnesses, explicit state-rich error output, centralized completion gates, and persisted run receipts so agents must prove real behavior rather than self-report success.
Building a production-grade AI code review agent requires moving beyond naive diff prompting to combine historical incident eval datasets, cross-family author-reviewer model inversion, a living codebase wiki instead of raw grep, dependency-rich sandbox UI testing, and per-subtask model routing.
Because coding agents accelerated inner-loop implementation while leaving enterprise legacy modernization bottlenecked on cross-functional alignment and domain context, GEICO built a multiplayer AI-native war room steered by Markdown FleetKit protocols, human-in-the-loop worksheets, and deterministic nine-step gates to ship legacy claims APIs in a single day.
Because abundant AI-generated code overwhelms human reviewers, compounds multi-step errors, and defeats inconsistent LLM judges, enterprise software factories must engineer shipping confidence multiplicatively through clear intent, in-loop guide rails, hybrid deterministic-generative verification, and auditable evidence.
Deploying agents safely in regulated enterprise workflows requires treating written policy as normative build-time intent separated from the continuously changing runtime system, and embedding agents inside directed acyclic graphs (DAGs) that parallelize independent reviews while preserving mandatory compliance gates.
Because coding agents generate large pull requests far faster than humans review every line, scaling toward autonomous software factories requires zero-trust, multi-layered verification inside every loop paired with rich agent-generated behavioral evidence.
As inference costs plummet and coding agents run unattended in the outer loop, engineering's primary failure mode shifts from implementation capacity to product incoherence, requiring automated pipelines that thread customer intent, product history, and ownership through triage, implementation, review, and daily feedback distillation.
Software engineers should abandon local coding to manage communicating fleets of cloud builder, reviewer, child-QA, manager, and merge-manager agents—keeping humans in the loop via mandatory feature-demo videos and hiring strictly through recorded agent-operating trials.
Reducing enterprise AI inference spend at scale requires splitting API keys by department and workload, matching model size and per-request tokens-per-second to human latency sensitivity, sizing dedicated GPU capacity to base load with cloud spillover, and measuring real business outcomes rather than raw token volume.