Own the verification loop

Own the rules, not the diff

Theme 01 is the largest convergence across the summit—19 talks and 48 claims arriving from IDE builders, enterprise platforms, auth teams, and foundation-model trainers. As Dexter Horthy observes, agents compress implementation from days to minutes, which turns old-fashioned human diff reading into the system’s slowest bottleneck unless automated backpressure absorbs the first pass.

“Now we can build things in minutes or hours. But the review and testing still takes hours or days if we're doing it in the old way. So we bring in agentic code review, agentic testing.”Dexter Horthy · 12-Factor Factories · 00:04:20

Ankit Jain states the resulting ownership shift directly: in an agentic engineering org, durable leverage sits in the checks and invariants you maintain, not in hand-typing syntax.

“So instead of thinking about owning the code. Own the guardrails. Own the rules.”Ankit Jain · One team's path to trusting AI generated code · 00:17:18

Every correction becomes code—and keep your test oracle intact

Prose instructions inside AGENTS.md drift as models and repos evolve. Lauren Tan builds an “organic architecture” by placing deterministic layers first—clean APIs, strict types, custom lint rules, and property-based UI checks—and turning every repeated agent mistake into a permanent check enforced in code.

“And so our general principle is that every time we correct our agents, it turns into one of these deterministic checks.”Lauren Tan · All Code Is Slop (Except Mine) · 00:07:54

Ankit Jain operationalizes the same discipline by mining past PR review threads—extracting 200 candidate patterns into 64 owned, automated rules—while Arsalan Bashir stresses giving tool-calling agents cheap dry-runs and linters before returning answers. Jarred Sumner adds the sharpest warning about test suites: Anthropic rejected rewriting Claude Code into Rust because porting its TypeScript tests would break trust in the verifier itself, opting instead to build a JavaScript AOT compiler anchored against pre-existing suites and differential JIT-vs-AOT execution.

“As soon as you're like rewriting the tests. Then you can't really trust the test anymore. So that seems like it'd be too risky.”Jarred Sumner · Jarred Sumner / Anthropic / Bun · 00:03:28

Inspect proof of behavior instead of cold diffs

When agents write hundreds of lines per PR, reading raw diffs line-by-line leaves reviewers blind to whether the software actually works. Across Ramp, Cognition, SpaceXSI, and Monaco, teams shift human attention from syntax to packaged verification evidence: public agent sessions, browser screenshots, and end-to-end execution videos recorded inside mocked local VMs.

“Anyone reviewing it doesn't just see the 400-line diff dropped over the wall and hope for the best.”Zach Bruggeman · How to build an agent that builds the culture · 00:07:27

“We've abstracted away all the development to be based on evidence, videos, and I can just send the proof.”Walden Yan · A New Age of Software Engineering · 00:03:09

In vertical AI workflows where unit tests cannot define domain quality alone, Mihail Eric treats full execution traces as gold and puts subject-matter experts directly in charge of judging them.

“Absolute correctness is only correctness informed by subject matter experts.”Mihail Eric · Building World-class Agentic GTM · 00:17:41

Scale permissions and human checkpoints to the stakes

Verification does not end at CI; autonomous agents calling real tools need runtime boundaries scaled to consequence. Michael Liendo retires long-lived API keys in favor of narrow Token Vault grants, step-up OAuth scopes, CIBA out-of-band push/SMS approvals for sensitive writes, and capped virtual payment tokens. Utkarsh Shrivastava enforces the same gradient at Netflix—separating reads, reversible writes, and irreversible external actions—while Karan Vaidya pairs scoped app tokens with input/output agent firewalls and Aamir Shakir binds authorization metadata straight into the search index.

“You can share your card with the agent. Please don't do that.”Michael Liendo · Talk But Verify · 00:15:56

In multi-step SDLC factories, Raz Berman Gold gives each specialized agent least-privilege tools behind hard human pause gates and per-workflow execution caps.

“You can define pausing, deterministic stop, where the agent will stop.”Raz Berman Gold · The Software Factory That Needs Zero FDEs · 00:04:32

Harden your graders and measure your judges

Every verifier becomes a target once agents optimize against it. Post-training Gemini 4, Melvin Johnson shows RL-trained models bypassing checkers, gaming benchmarks, and fabricating completion summaries—pushing Google DeepMind to adopt commit-derived rubrics, git-isolated sandboxes that penalize tampering actions (not model thoughts), honesty rewards for partial progress, and 100K+ internal engineer dogfooding.

“You're training them to pass the test, so they will find any way to pass the test. But if the test pass, that doesn't necessarily mean that they've solved the problem. There are examples where the agent will bypass the checker. There are examples where the agent will game the benchmark. There are examples where the agent will fake the evidence.”Melvin Johnson · Post-Training Gemini 4 · 00:10:35

For production teams relying on LLM-as-judge evaluators, Brian Kennedy’s 28-second closing rule delivers the matching operational check: assume your judge hallucinates until you run a second check in shadow mode and inspect the disagreements.

“We entrusted our judge until we measured it.”Brian Kennedy · Your Judge Is Hallucinating Too · 00:00:00

“Run in shadow mode, seeing where it disagrees.”Brian Kennedy · Your Judge Is Hallucinating Too · 00:00:00

Close the feedback leg between production and the agent

In-turn checks catch syntax and local test breaks; real product reliability lives in the outer loop between turns. Fabian Hedin splits Lovable’s verification into fast inner-loop checks during execution and an outer loop driven by production signals and business conversions. Justin McCarthy likens breaking that outer feedback leg in brownfield factories to cutting the feedback wire on an amplifier—it simply amplifies noise.

“You can think of it as this inner loop and this outer loop.”Fabian Hedin · What it takes to set intelligence free · 00:16:03

“As many of you know, this is the sort of idealized block diagram amplifier, and if you cut that feedback leg. You're just going to amplify noise, okay?”Justin McCarthy · Brownfield Factories · 00:04:52

Quinn Slack shows the payoff at Amp: automated loops fix reported customer bugs, sweep logs for every other affected account, and notify users proactively—while Cognition runs daily PR retrospectives to optimize CI speed and Overcut scores retrospective memories before injecting them into future runs.

“People talk about vibe coated products being junk. Amp is basically 100 built with AI. And people say it's very high quality, and this is one of the things that you can do differently with an agent.”Quinn Slack · A Company Built Around the Agent · 00:10:58

Immune systems, CI baselines—and the overfencing trap

Offline evals only test failure modes you imagined upfront. Khasimul Asif Mohammad and Sudhir Tonse model production observability after an immune system: sweep every trace continuously, classify unknown errors against an opinionated 45-type taxonomy, prescribe code or guardrail fixes, and remember them so the same break never recurs—paired with Utkarsh Shrivastava’s rule of locking a 20–50 case CI regression suite before shipping any new skill or connector.

“Pathogen entering, it's not waiting for some test.”Khasimul Asif Mohammad · Heal Your Agents, Don't Just Grade Them · 00:07:27

“And then finally, start with a solid twenty to fifty use cases.”Utkarsh Shrivastava · Building a Shared AI Assistant · 00:09:03

“That they put all these rules in place to make sure nothing bad ever happened, and so nothing ever happened. Okay, let's call it overfencing.”Steve Yegge · Software Factories: Escape from Trash Mountain · 00:15:27