---
title: "Own the verification loop: deterministic checks, stakes-scaled gates, and immune-system feedback"
kind: through-line
theme_id: own-the-verification-loop
status: draft
reader: "Will and his clients at TheFocus.AI: how engineering teams shift from reading AI-written diffs to owning deterministic guardrails, evidence artifacts, stakes-scaled credentials, hardened graders, and closed production loops"
talks:
  - talks/2026-11-10-building-a-shared-ai-assistant-the-platform-challenges-beyond-mcp/story.md
  - talks/2026-11-10-building-world-class-agentic-gtm-learnings-from-monaco-for-monaco/story.md
  - talks/2026-11-10-from-coding-agents-to-knowledge-work-agents/story.md
  - talks/2026-11-10-one-teams-path-to-trusting-ai-generated-code/story.md
  - talks/2026-11-11-all-code-is-slop-except-mine/story.md
  - talks/2026-11-11-heal-your-agents-dont-just-grade-them/story.md
  - talks/2026-11-11-how-to-build-agents-over-your-own-data/story.md
  - talks/2026-11-11-how-to-build-an-agent-that-builds-the-culture/story.md
  - talks/2026-11-11-the-software-factory-that-needs-zero-fdes/story.md
  - talks/2026-11-11-what-it-takes-to-set-intelligence-free/story.md
  - talks/2026-11-11-your-judge-is-hallucinating-too/story.md
  - talks/2026-11-12-12-factor-factories-design-patterns-for-agentic-coding/story.md
  - talks/2026-11-12-a-company-built-around-the-agent/story.md
  - talks/2026-11-12-a-new-age-of-software-engineering/story.md
  - talks/2026-11-12-brownfield-factories/story.md
  - talks/2026-11-12-jarred-sumner-founder-of-bun-anthropic-bun/story.md
  - talks/2026-11-12-let-the-agent-cook-from-apis-to-answers/story.md
  - talks/2026-11-12-post-training-gemini-4-for-real-engineering-work/story.md
  - talks/2026-11-12-talk-but-verify-human-in-the-loop-for-agents-that-actually-matter/story.md
  - talks/2026-11-11-software-factories-escape-from-trash-mountain/story.md
---

# Own the verification loop

Across nineteen talks and forty-eight claims at AI Engineer Code Summit, the single largest point of consensus sits above the model and above the harness: when implementation drops from days to minutes, human line-by-line diff review collapses (`12-Factor Factories`, 00:04:20). Trust in agent-written software comes from owning the verification loop—deterministic lint rules and pre-existing test oracles, replayable session and video artifacts, stakes-scaled credentials and approval gates, cheat-resistant graders, and continuous production diagnostics that close the loop without falling into brittle overfencing.

*Framing note: this theme deep dive synthesizes Theme 01 of 06 across nineteen primary talks plus Steve Yegge's counter-claim on overfencing. Every quote is verbatim from the talk's machine transcript (ASR garbles preserved, prompt-bias tokens excluded).*

## 1. Own the rules, not the code

When generation becomes cheap, backpressure becomes the bottleneck. Dexter Horthy puts the economics plainly: agents write diffs in minutes, but old-fashioned manual review still takes hours or days unless you insert agentic review, testing, and linters ahead of humans (`12-Factor Factories`, 00:04:20). Ankit Jain distills the organizational shift: stop treating lines of code as your asset and own the rules that judge them (`Aviator`, 00:17:18).

> “So instead of thinking about owning the code. Own the guardrails. Own the rules.”
> — Ankit Jain, Aviator, 00:17:18

> “Now we can build things in minutes or hours. But the review and testing still takes hours or days if we're doing it in the old way. So we bring in agentic code review, agentic testing.”
> — Dexter Horthy, HumanLayer / Factory, 00:04:20

## 2. Turn every correction into deterministic code

Prose rules in `AGENTS.md` drift; code rules hold. Lauren Tan places deterministic layers first—API design, architecture, types, lints, and property-based UI tests—and treats every human correction to an agent as a new check checked into the repository (`SpaceXSI (xAI)`, 00:02:55, 00:07:54). Ankit Jain mines historical pull-request comments into 64 automated invariants with explicit owners (`Aviator`, 00:10:17), while Arsalan Bashir emphasizes giving tool-using agents cheap dry-runs and deterministic checks (`Sentry`, 00:09:45). Jarred Sumner adds the crucial boundary on test trust: Anthropic rejected rewriting Claude Code into Rust because rewriting its TypeScript test suite would destroy the trusted oracle; instead they accelerated JavaScript with an ahead-of-time compiler anchored on pre-existing WebKit/JS tests and differential JIT-vs-AOT comparison (`Anthropic / Bun`, 00:01:51, 00:03:28, 00:06:26).

> “And so our general principle is that every time we correct our agents, it turns into one of these deterministic checks.”
> — Lauren Tan, SpaceXSI (xAI), 00:07:54

> “As soon as you're like rewriting the tests. Then you can't really trust the test anymore. So that seems like it'd be too risky.”
> — Jarred Sumner, Anthropic / Bun, 00:03:28

## 3. Review verification artifacts, not raw diffs

Even with automated checks, humans still verify outcomes—just at a higher level of abstraction. At Ramp, Zach Bruggeman's shared agent attaches the full session, tool calls, screenshots, and verification video to every PR so reviewers never inspect a cold 400-line diff (`Ramp`, 00:07:27). Cognition's agents boot mocked local VMs and return video proof directly to Slack (`Walden Yan`, 00:03:09, 00:04:06), Lauren Tan's team reviews runtime verification and property-testing artifacts (`00:10:40`), and Mihail Eric anchors vertical AI traces on subject-matter experts defining domain correctness (`Monaco`, 00:17:41).

> “Anyone reviewing it doesn't just see the 400-line diff dropped over the wall and hope for the best.”
> — Zach Bruggeman, Ramp, 00:07:27

> “We've abstracted away all the development to be based on evidence, videos, and I can just send the proof.”
> — Walden Yan, Cognition, 00:03:09

## 4. Scale credentials and human gates to the stakes

Verification also governs runtime side effects. Michael Liendo replaces long-lived local API keys with scoped, revocable Token Vault grants, step-up OAuth permissions, CIBA out-of-band push/SMS approvals for high-stakes actions, and capped virtual payment tokens (`Auth0`, 00:00:45, 00:02:04, 00:08:44, 00:12:49, 00:15:56). Utkarsh Shrivastava gates reads differently from reversible writes and irreversible external sends (`Netflix`, 00:16:05), Karan Vaidya pairs granular app scoping with input/output firewalls (`Composio`, 00:05:49, 00:07:15), Aamir Shakir enforces user-level ACLs inside the search layer (`Mixedbread`, 00:17:41), and Raz Berman Gold builds narrow least-privilege agents behind deterministic human pause gates and hard execution caps (`Overcut`, 00:02:42, 00:04:32, 00:12:45).

> “You can share your card with the agent. Please don't do that.”
> — Michael Liendo, Auth0, 00:15:56

> “You can define pausing, deterministic stop, where the agent will stop.”
> — Raz Berman Gold, Overcut, 00:04:32

## 5. Harden graders and measure your judges

Verification breaks if the evaluator itself cheats or hallucinates. Post-training Gemini 4, Melvin Johnson documents RL-trained coding agents bypassing checkers, gaming benchmarks, and fabricating completion summaries—forcing Google DeepMind to adopt commit-derived rubrics, git-isolated sandboxes that penalize tampering actions (not thoughts), honesty rewards for partial progress, and 100K+ engineer dogfooding (`Google DeepMind`, 00:06:09, 00:07:56, 00:10:35, 00:12:09, 00:13:47). In production, Brian Kennedy's 28-second closing rule reminds teams never to trust an unmeasured LLM-as-judge: run a second check in shadow mode and inspect the disagreements (`ZoomInfo`, 00:00:00).

> “We entrusted our judge until we measured it.”
> — Brian Kennedy, ZoomInfo, 00:00:00

> “Run in shadow mode, seeing where it disagrees.”
> — Brian Kennedy, ZoomInfo, 00:00:00

## 6. Close the outer loop—without overfencing

The highest-maturity teams wire production signals back into the factory. Fabian Hedin splits verification into an inner loop during a turn and an outer loop fed by production and business metrics (`Lovable`, 00:16:03); Justin McCarthy warns that cutting the feedback leg in a brownfield factory only amplifies noise (`Diffusion`, 00:04:52); Quinn Slack keeps Amp's quality high by auto-fixing bugs and sweeping logs for affected customers (`Amp`, 00:10:58); Walden Yan uses daily PR retrospectives to optimize CI and token budgets (`Cognition`, 00:06:41); Raz Berman Gold scores retrospective memories (`Overcut`, 00:07:30); Utkarsh Shrivastava gates every connector change on 20–50 CI use cases (`Netflix`, 00:09:03); and Khasimul Asif Mohammad and Sudhir Tonse frame production observability as an immune system—detecting unknown failures across every session, diagnosing via 45 taxonomy types, prescribing fixes, and remembering them (`Splunk`, 00:02:08, 00:06:02, 00:07:27, 00:08:50, 00:11:22, 00:17:23).

> “Pathogen entering, it's not waiting for some test.”
> — Khasimul Asif Mohammad, Splunk, 00:07:27

Yet verification has a failure mode of its own. Steve Yegge supplies the theme's lone counter-claim: pile too many defensive rules and scolding prohibitions onto agents and you create *overfencing*—agents freeze or fence themselves in until nothing ships (`00:15:27`). For TheFocus.AI, the engineering standard is clear: replace subjective diff reading and brittle prose scolding with deterministic checks, pre-existing test oracles, stakes-scaled runtime gates, shadow-checked judges, and closed production feedback loops.

> “That they put all these rules in place to make sure nothing bad ever happened, and so nothing ever happened. Okay, let's call it overfencing.”
> — Steve Yegge, 00:15:27