AI Engineer New York · Oct 12–14, 2026 · Synthesis
Eight shifts from AI Engineer New York 2026
Across 31 recorded talks, 95,224 spoken words, and 176 transcript-verified claims, New York’s engineering rooms converged on one reckoning: generating code and prose is now cheap, so the real engineering work has moved into synthetic domain evals, sandboxed computers, deterministic verification gates, runtime risk governance, and self-improving agent fleets.
Bootstrapping Domain Evals Without Public Benchmarks
Public benchmarks never measure a team's real task, so builders must engineer bespoke evaluation loops—synthesizing inputs from known answers, replaying historical outages, backtesting dated web snapshots, and grading trajectories with behavior rubrics—or redesign the agent until it is evaluable.
Across enterprise RAG, legal review, creative writing, code review, accounting, and quantitative investing, speakers converged on the same wall: off-the-shelf benchmarks tell you nothing about whether your agent works, and domain experts cannot hand-grade thousands of runs. Air Canada inverted dataset creation by generating questions from known source pages and populating legal templates with clauses engineered to breach or pass rules, even re-architecting an agent purely to make it deterministically evaluable. Greptile benchmarks prompt changes against historical outage-causing PRs; Bridgewater hand-labeled 200 investment queries in a week to lift precision@10 from 60% to 95%; Exa Snapshot backtests predictive agents against 400 billion dated web pages without lookahead leakage; Towards AI runs five drafts per prompt under a two-stage cross-family rubric; and Basis grades ground-truth-free production runs with behavior specs and trajectory forking. The sharp counter-position centers on the grader itself: Sonar and Air Canada both showed that uncalibrated LLM-as-judge verdicts flip across identical runs and add noise rivaling the agent's own error rate.
“And you can say, in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant?”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · ### 00:12:01 — Behavior Specs as Ground-Truth-Free Trajectory Rubrics
Token Economics, Right-Sizing, and Open Foundations
As frontier intelligence commoditizes while aggregate token bills surge, durable economics come from attributing spend per workload, routing subtasks to fast specialist or open-weight models, tuning serving concurrency and KV caches, and owning a model-swappable stack rather than renting lock-in from frontier labs.
Practitioners attacked runaway inference costs across every layer below default frontier API calls. Featherless showed that splitting API keys by department, reserving superlinear low-latency serving (3x cost per 2x speed) for interactive paths, and leveraging higher tokens-per-second to entice developers onto smaller worker models deliver immediate ROI. Runpod's BenchPod tunes vLLM nightly builds, batch-token concurrency, and 4xH200 DeepSeek deployments live without 30-minute cold starts; Greptile routes 20 review subtasks across small and frontier models; Exa cuts trajectory tokens 95% via dynamic snippet highlights; Towards AI merges ten cheap GLM/Flash drafts with one high-thinking pass; and Ramp pairs frontier orchestrators with post-trained 35B Qwen models, 5x-cheaper open-weight security swarms, and semantic spend taxonomies—even as Apify warned that heavily subsidized flat-rate plans distort build-versus-buy incentives. Strategically, Nous Research argued that shrinking token margins force frontier labs to seek lock-in at the enterprise agent layer and compete vertically, making open, model-agnostic foundations essential. The counter-arguments cut both ways: naive turn-by-turn multi-model routing destroys prefix KV-cache hit rates, while falling frontier unit prices tempt teams to scale on frontier platforms rather than maintain custom post-training.
“If you think back to those two slides I showed, where the choice of harness did not make a significant difference in eval scores, both of those studies actually found that there was a significant difference in costs for tasks.”Jediah Katz · The Model Ate the Coding Harness. Now What? · ### 00:11:59 — Cloud Runtime, Scoped Security, and Token Efficiency
Sandboxed Computers, Context Architecture, and the Shrinking Harness
As frontier models absorb prompt-level coding tricks through a user-trajectory data flywheel, the durable agent harness converges on two layers models cannot internalize: isolated create-run-delete VM sandboxes and outer-loop context engines that supply institutional intent, living codebase maps, and precomputed KV caches.
OpenHands framed the central paradox of 2026 harness design: coding scaffolding makes harder tasks tractable, user trajectories feed model post-training, and models absorb those habits until harness choice barely moves coding benchmark scores—motivating self-cleaning guardrail shims that watcher bots delete once newer models stop triggering them. What models cannot absorb defines the lasting architecture. First is the execution boundary: Daytona showed that every competing framework reduces to create-run-delete VM primitives and is actively deleting local executors because embedded runtimes still expose host tools and a single allowed `python -c` command compromises the host; Arize and Greptile isolate every run in disposable sandboxes while persisting memory externally. Second is outer-loop context: Linear showed that five-cent intelligence shifts failure from coding capacity to product incoherence unless triage agents assemble customer value, logs, and prior decisions upfront; Greptile replaces grep with a living company wiki; Sonar Vortex feeds structural maps in-loop to cut token spend 36%; Ramp compresses 20–30x month-end context into offline KV-cache cartridges; GEICO routes edge-case HTML worksheets to domain owners; and Basis treats every working-window token as runtime training data. Yet as OpenHands noted in counter-point, entering unmastered domains like autonomous multi-agent colleagues immediately forces teams to re-add heavy prompting, coordinator hierarchies, and guardrails.
“really think of the context that agent is going to see as it's working as runtime training data. The word prompt, I think, trivializes it a bit, right?”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · ### 00:09:16 — Why Outcome Evals Fall Short on Long Horizons
Bounded Autonomy, Runtime Governance, and the Enterprise Diffusion Gap
Outside software engineering, enterprise agents remain near 5% production adoption not for lack of raw model intelligence, but because non-deterministic workers require institutional trust infrastructure—automatic registries, build-time policy DAGs, risk-tiered entitlements, and opinionated workflows—between reckless autonomy and paralyzing lockdown.
Tenex quantified the conference's starkest macro reality: across roughly 100 large enterprises, general-purpose LLM access sits near 90% and coding agents near 50%, yet autonomous agents in non-engineering business workflows sit around 5%. Anthropic, Apollo, GEICO, Basis, and Apify all traced that diffusion gap to governance, process, and trust friction rather than model IQ—with Apify showing that even after eighteen months of agent payment standards, 30-day volume sits near $1M on x402 and $1,500 on MPP because static API keys, untrusted endpoints requiring marketplace middlemen, and deferred-settlement double-spend risks dominate the funnel. The architectural remedy across regulated firms was bounded autonomy: Two Sigma replaces months-long manual review queues and rubber-stamp approval fatigue with automatic framework registration, risk-scored scrutiny, registry-compiled runtime entitlements, and conservative classifiers that escalate only ambiguous calls; Coinbase separates build-time normative policy ('the original institutional system prompt') from mutating runtime state inside compliance DAGs; Basis encodes opinionated, auditable tax and accounting workflows rather than 'Move 37' creativity; and Anthropic scopes financial agents with least-privilege identities and deterministic hooks gated on advisor or SME sign-off. Qualifying this optimism, Daytona showed that host-level policy allowlists collapse if an interpreter command is permitted, while GEICO's one-day war rooms proved outer-loop enterprise inertia can be broken when decision-makers sit inside the loop.
“Nobody wants their agent to move thirty-seven their tax return. They want it done reliably at scale in a way that they can trust, and in a way that their people know how to work with.”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · ### 00:17:21 — Earning Organizational Trust Over Move-37 Creativity
Deterministic Verification Loops Over Line-by-Line Review
Because coding agents generate pull requests far faster than humans can read them—and reviewers accept wrong AI output four times out of five—shipping confidence must come from independent, multi-layered verification harnesses and runnable behavioral proofs rather than manual line-by-line diff inspection.
Software delivery talks confronted the review bottleneck head-on: Sonar reported a 2,450% explosion in AI-generated code yielding only 30% more releases because human reviewers succumb to 'cognitive surrender' (agreeing with wrong AI output 80% of the time) while multi-step agent accuracy decays exponentially across long horizons. Teams moving toward lights-on and dark software factories replaced manual line reading with executable proof loops. In the verification workshop, external MCP gates (`Ctx`) refuse agent self-reports until deterministic CLI and Playwright checks verify backend persistence, making verifier suites more durable than the application code itself. At Anthropic, merging 30,000 daily changes relies on automated first-pass reviews, edit hooks, and browser-driven screenshot and video evidence; Sonar pairs CPU static analysis with GPU generative checks and CVE scanning across every loop; Greptile inverts author and reviewer model families (+10% bug catch) and boots full-stack VM sandboxes for vision-model UI QA; Linear wires verification skills and product-history review agents into unattended bug fixes; GEICO enforces deterministic nine-step artifact gates; Untangle caps builder-reviewer loops at three iterations; and Coinbase runs parallel agent reviews inside gated DAGs. Against pure automation, Untangle insisted that a human product owner watching each feature video remains irreducible, and Featherless warned that unowned generation lobs 'slop grenades' onto peers.
“A lights on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person so apply your judgment and taste inside the picture”Addy Osmani · Who verifies the software factory? · ### 00:01:45 — CI Merge Failures & Lights-On vs. Dark Factories
Software engineering is shifting from writing code locally to managing asynchronous cloud agent fleets and closing the trace-to-skill loop—centralizing transcripts and tool calls, grading runs post-merge, and letting meta-agents distill recurring failures and human corrections into versioned skills.
Across Warp, Arize, Linear, Untangle, GEICO, Anthropic, and Runpod, the unit of engineering work moved from crafting individual features to operating and improving the agent factory itself. Warp demonstrated 'factory engineering': shared cloud Docker sandboxes automatically centralize conversation transcripts, MCP/tool calls, and diffs, exposing silent agent workarounds (like brute-force port probing) to post-merge markdown judge rubrics and daily meta-agents (`Skill Doctor`) that open measurable skill PRs. Linear runs a daily post-merge distillation job that turns PR review corrections and reverts into `agents.md` updates; Arize designs agent-first telemetry to wake scheduled sandbox sweeps while tracing both production agents and the coding harnesses behind them; Anthropic treats SME-authored plain-text skills as living, versioned code co-iterated by business teams and engineers; GEICO coordinates legacy transformations through Markdown FleetKit protocols; Runpod exposes GPU benchmark controls as FastAPI endpoints for agents to drive directly; and Untangle replaces local development with communicating builder, reviewer, and merge-manager sessions—extending the paradigm to hiring by stack-ranking candidates on 45-minute recorded agent-operating trials, echoing the Day 2 keynote's call to pair human company curation with career-research agents. In tension with universal 'agent management,' Basis warned that pushing prompt and skill authorship onto non-technical enterprise users creates a severe competency gap, and Day 2's keynote cautioned against letting agents automate personal outreach like resume submissions.
“You can set up some scoring agents to score the quality of how those conversations went, so think of this like an LLM as a judge reviewing against rubrics that you define across metrics like efficiency, or code quality, or test coverage.”Ben Holmes · Your agent needs a skill to improve its skills · ### 00:01:44 — Factory Engineering & the Skill Improvement Loop
Governed Ontologies, Provenance, and Financial Workbenches
In institutional investing and private markets, off-the-shelf web chatbots fail on stale or conflicting inputs; agents earn front-office trust only atop governed firm ontologies, validated golden sources, unified MCP connectors, and hybrid code-ML-LLM-human pipelines that write synthesis back into the vault.
Every institutional finance talk located competitive advantage in governed data infrastructure and workflow design rather than raw model selection. Bridgewater contrasted an open-web chatbot answering a Korean won query from a single three-month-old note with Vault—which aligns 100M+ documents to a bitemporal ontology, serves 1,000+ QPS agent storms via turbopuffer cluster-local native filtering, migrates researcher email habits into saved collections, and writes human-agent workbench synthesis back into the corpus so knowledge compounds. At Apollo's Redding Ridge credit platform, engineers sat daily with portfolio managers to codify golden-source precedence across 20 conflicting systems, turning multi-hour weekly loan reconciliations into one-minute automated QA reports and cutting 200–400-name CLO portfolio construction from days in Excel to minutes for the first 80–90%. Coatue modeled the hedge fund as a composite state-machine function `(external_context, internal_context) -> (stock, size, timing)`, insisting that every pixel a human sees must be API-accessible while assigning each step whichever runner—deterministic code, classical ML, LLM, or human—maximizes quality. Exa supplied the external temporal layer with 400 billion dated web snapshots built for dense exploratory agent queries, while Anthropic standardized multi-system enterprise access behind a single MCP connectivity layer.
“We essentially use agents to do the work of data engineering, right information or reconciled data from across a variety of systems to the portfolio managers, traders, and so forth can make decisions faster and more accurately.”Michael Manapat · AI in Practice: An Inside Look at Apollo's Approach · ### 00:00:00 — Introducing Rowspace and Apollo's Redding Ridge Credit Business
Spatiotemporal World Models and Real-Time Simulation
Scaling laws extend beyond turn-based text to continuous video and action streams, turning pixels-to-actions policies and next-frame predictors into real-time world simulators that internalize physics, control robots, and replace static 3D engines with interactive visual interfaces.
General Intuition and Runway pushed AI engineering past autoregressive text into continuous space and time. General Intuition argued that real-time agents must observe while acting, leveraging Medal's gameplay pipeline—scaling toward three billion videos a year and 1.5 trillion monthly controller actions—because high-APM game controllers share the exact action space of low-DOF robot teleoperation and plug neatly as reactive control layers beneath frontier LLM planners. Runway bet on compute scaling laws for next-frame video prediction, demonstrating that models trained at scale internalize geometry, gravity, occlusion, and object permanence directly from pixels—unlocking action-conditioned robotics simulators, engine-free World Interface Models with no Unity or Blender mesh underneath, interactive Continuum video portals, and visual scratchpads where coding agents simulate spatial designs alongside code. Their sharpest technical disagreement sits on supervision: General Intuition contended that explicit controller action labels transfer far better to real-time control than passive video-only latent inference because subtle inputs like aircraft rudder adjustments leave almost no trace in raw pixel frames.
“You're learning about physics. You're learning about object permanence. You're learning about gravity, you're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world.”Cristóbal Valenzuela · Everything will be Simulated · ### 00:02:30 — Scaling Next-Frame Prediction into Physical Understanding
Public benchmarks never measure a team's real task, so builders must engineer bespoke evaluation loops—synthesizing inputs from known answers, replaying historical outages, backtesting dated web snapshots, and grading trajectories with behavior rubrics—or redesign the agent until it is evaluable.
Eugene Cheah, Aaron Linsky, John Feminella +8 more
As frontier intelligence commoditizes while aggregate token bills surge, durable economics come from attributing spend per workload, routing subtasks to fast specialist or open-weight models, tuning serving concurrency and KV caches, and owning a model-swappable stack rather than renting lock-in from frontier labs.
Eugene Cheah, Louis-François Bouchard, Ryan Ahern +7 more
As frontier models absorb prompt-level coding tricks through a user-trajectory data flywheel, the durable agent harness converges on two layers models cannot internalize: isolated create-run-delete VM sandboxes and outer-loop context engines that supply institutional intent, living codebase maps, and precomputed KV caches.
Aparna Dhinakaran, Ford Prior, Daksh Gupta +6 more
Outside software engineering, enterprise agents remain near 5% production adoption not for lack of raw model intelligence, but because non-deterministic workers require institutional trust infrastructure—automatic registries, build-time policy DAGs, risk-tiered entitlements, and opinionated workflows—between reckless autonomy and paralyzing lockdown.
Because coding agents generate pull requests far faster than humans can read them—and reviewers accept wrong AI output four times out of five—shipping confidence must come from independent, multi-layered verification harnesses and runnable behavioral proofs rather than manual line-by-line diff inspection.
Software engineering is shifting from writing code locally to managing asynchronous cloud agent fleets and closing the trace-to-skill loop—centralizing transcripts and tool calls, grading runs post-merge, and letting meta-agents distill recurring failures and human corrections into versioned skills.
Belinda Neal, Aparna Dhinakaran, Ford Prior +5 more
In institutional investing and private markets, off-the-shelf web chatbots fail on stale or conflicting inputs; agents earn front-office trust only atop governed firm ontologies, validated golden sources, unified MCP connectors, and hybrid code-ML-LLM-human pipelines that write synthesis back into the vault.
Scaling laws extend beyond turn-based text to continuous video and action streams, turning pixels-to-actions policies and next-frame predictors into real-time world simulators that internalize physics, control robots, and replace static 3D engines with interactive visual interfaces.
No when deterministic checks are achievable: LLM-judge noise rivaled the agent's own error rate, motivating architectural redesign around ground-truth checks.
Fault line 02
Do spatiotemporal world models learn control from passive video prediction alone, or do they require explicit action labels?
Next-frame video prediction at scale internalizes geometry, gravity, occlusion, and object permanence directly from pixels to power interactive world simulators.
Explicit controller action labels transfer far better to real-time control because subtle high-frequency inputs like rudder adjustments cannot be inferred from video frames alone.
Fault line 03
Should enterprises own open, model-swappable foundations or scale atop frontier lab platforms?
Own open foundations: commoditizing token margins force frontier labs up into the enterprise agent layer and vertical markets to capture switching-cost stickiness.
Go a layer below frontier APIs now: pairing a frontier orchestrator with a post-trained 35B open specialist beat the frontier on exact-pass accuracy at lower latency.
Scale a shared foundation on frontier models: raw model capability and unit cost are no longer the bottleneck—organizational deployment from one-to-many is.
At mandatory mutation sign-off: financial agents operate with least-privilege identity and propose amendments that a human SME must approve before execution.
Only on genuinely ambiguous escalations: approving every action induces rubber-stamp fatigue, so deterministic fast paths and classifiers should handle routine calls.
At the behavioral proof gate: builders loop autonomously with reviewer agents, but a human product owner must watch each recorded feature video before merge.
Above line-by-line code reading: high-confidence automated merges rely on end-to-end, property-based, and visual verification rather than manual diff inspection.
Fault line 05
Will frontier models absorb the agent harness, or must builders keep engineering scaffolding and process around them?
Models absorb coding scaffolding via the data flywheel: trajectories generated through today's harness train tomorrow's weights until harness choice stops moving coding accuracy.
Opinionated process stays outside the weights: open-ended agents produce inconsistent enterprise work unless builders encode explicit domain workflow specs.
Every unmastered frontier re-inflates the harness: autonomous multi-agent knowledge workers immediately require coordinator hierarchies, new prompting, and guardrails.
Outer-loop context remains irreducible: cheap code shifts failure to product incoherence because source code alone never holds customer intent, ownership, or revert history.
Evidence ledger (176 claims)
Filter by theme below, or expand any section to inspect the full verbatim quotes linked to every talk story and transcript.
Theme 01 · 11 talks · 21 claims
Bootstrapping Domain Evals Without Public Benchmarks
Public benchmarks never measure a team's real task, so builders must engineer bespoke evaluation loops—synthesizing inputs from known answers, replaying historical outages, backtesting dated web snapshots, and grading trajectories with behavior rubrics—or redesign the agent until it is evaluable.
Across enterprise RAG, legal review, creative writing, code review, accounting, and quantitative investing, speakers converged on the same wall: off-the-shelf benchmarks tell you nothing about whether your agent works, and domain experts cannot hand-grade thousands of runs. Air Canada inverted dataset creation by generating questions from known source pages and populating legal templates with clauses engineered to breach or pass rules, even re-architecting an agent purely to make it deterministically evaluable. Greptile benchmarks prompt changes against historical outage-causing PRs; Bridgewater hand-labeled 200 investment queries in a week to lift precision@10 from 60% to 95%; Exa Snapshot backtests predictive agents against 400 billion dated web pages without lookahead leakage; Towards AI runs five drafts per prompt under a two-stage cross-family rubric; and Basis grades ground-truth-free production runs with behavior specs and trajectory forking. The sharp counter-position centers on the grader itself: Sonar and Air Canada both showed that uncalibrated LLM-as-judge verdicts flip across identical runs and add noise rivaling the agent's own error rate.
Also called: write the answer first · point-in-time backtesting · behavior specs · evaluating the evaluators · golden datasets
Distilling core workflow opinions into 'behavior specs' creates a ground-truth-free rubric that grades any production trajectory whenever a relevant trigger arises, scaling supervision signal far faster than static outcome evals.
“And you can say, in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant?”
When a multi-hour, thousand-call trajectory violates a behavior spec, builders can iterate on context fixes rapidly either by forking state right before the bad decision or by cheaply simulating the production prompt and skills in a coding subagent.
“when working with coding agents, ask the coding agent to make a subagent that pretends to be your production agent. And you can tell it to not cheat, and give it the system prompt and the skills, and you render everything perfectly.”
Benchmarking against recent production code as golden data revealed that once rich domain context is synthesized, smaller Sonnet-class models generate legacy code as well as frontier models at much lower cost.
“we ran benchmarks against like using our existing code recently written as the golden data and found out that Fable or Sonnet five wrote it just as good as, if not better than Fable.”
AI writing slop persists as models scale not merely through overused vocabulary, repetition, and hallucination, but fundamentally because generated prose collapses into a generic, voiceless average rather than matching a specific author's tone.
“Endlessly, the worst to me is that it usually has the wrong tone, or at least it has no tone and like everyone's tone at the same time, so it just doesn't really talk back to you.”
Generating five drafts per prompt across ten published and held-out scripts captures intra-model variance—which widens as model quality drops and masks lucky one-in-five outlier drafts—without making benchmark runs prohibitively expensive.
“But yeah, it's we found that five drafts seems to be the good middle ground in terms of cost for us to execute and to measure this disparity.”
Structuring 14,000-word judge prompts as a two-stage rubric—classifying prose into one of six qualitative buckets before scoring across nine metrics—outperforms binary or direct percentage grading on subjective voice matching.
“Then, the judge will categorize between six categories first, and after having found the category, it will give a rating. We found that this is the best way to measure for this more qualitative task.”
Because LLM judges exhibit strong family-wide self-preference—especially OpenAI models favoring all OpenAI outputs—creative-writing evaluations require cross-family judges spot-checked against over a thousand human pairwise comparisons.
“And the problem with AI judges is that they have references. Some models prefer themselves, all of those for themselves, and particularly open AI.”
Because model upgrades act like changing runtime dependencies that can silently alter output quality, firms should provide shared self-serve evaluation infrastructure—analogous to unit testing rather than centralized sign-off queues—where teams supply only agent-specific assertions.
“we wouldn't ship software without unit tests. So agents need evaluations for the same reason. And we want to provide this stuff at the level of the firm that's going to be the same for everybody.”
Ingestion design choices—such as chunking, raw image versus OCR input, translation versus structured extraction, and cross-referencing internal notes—act as tunable hyperparameters over an evolving search space.
“There's actually a lot of knobs here, and I almost think about those knobs — it's like hyperparameters in a traditional machine learning system, right? I think about each of these knobs as something for me to tune and optimize against.”
Because real steps combine multiple runners and tune model size, thinking effort, harness, skills, and hooks, workflows must be compartmentalized into evaluable units with a clear north-star metric.
“So the way to think about breaking down these problems is, what can I actually compartmentalize in a way that we can understand and evaluate?”
Hand-labeling roughly 200 representative investment queries in under a week established a baseline of 60% precision at ten and enabled systematic agentic search iterations that pushed precision to roughly 95%.
“The key metric we were looking at is precision at ten, which is of the top ten results, how many are the right documents, and we were hitting about sixty percent.”
Because automated PR and production gates are only useful if trusted, AI engineers must curate golden datasets and meta-evaluate LLM judges until they align with human judgment before deploying them offline or online.
“We run evals, verify them against the human output, and then try to align to make sure that the judge actually agrees with the human and they're in agreement before we actually ship out a new evaluator.”
Testing predictive models on live web search leaks post-event outcomes, whereas querying snapshots at fixed horizons before an event evaluates what signals were genuinely discoverable without lookahead bias.
“So I'm gonna go like one day before the announcement, seven days before the announcement, a month before the announcement, and see whether I could or could not predict it, right?”
Across traditional ML feature enrichment, hybrid models adding agentic adverse-media or equity scans, and end-to-end prediction agents, point-in-time snapshots let teams empirically prove historical lift over baseline processes.
“You can actually go back and back test that against your existing processes and say, hey, it was better.”
Without historical web snapshots and holdout sets, iterating on expressive semantic search features requires waiting months in live production to tell whether an agentic signal generalizes or overfits.
“You might be waiting for months to try and see how and why your agent set up actually is predictive or is overfitting in different ways.”
Because catastrophic bugs occur in roughly one in a thousand pull requests and live review output cannot prove bug absence, prompt changes must be benchmarked against a catalog of historical outage-causing PRs.
“the validation against really any reviewer can tell you about the presence of bugs, but unfortunately can't tell you about the absence of bugs.”
When an agent's original workflow design resists deterministic evaluation, teams should re-architect the system around evaluable intermediate steps rather than shipping an unmeasurable design.
“The original design we could not evaluate it. We switched to something that we can actually evaluate because of that.”
Generating synthetic questions from known source pages and checking retrieval in code accelerated iteration from one experiment per week to one per business day while eliminating brittle pipeline rules.
“So the impact of that is we went from one iteration week to business day, and we start to see the accuracy or the rating from the experts going up and up, and actually, removing a lot of complexity.”
Populating existing legal contract templates with synthetic clause arrays engineered to either violate or satisfy known rules creates ground-truth breach labels by construction and runs evaluations in five minutes.
“instead of trying to fill them, we generate an array for each of those blanks, which — some covers break some wood. Some covers doesn't. We know that. So we know what actually should be tagged as a breach or not.”
In enterprise Q&A over roughly 10,000 telecom vendor documents, retrieving the right source page produces the right answer 90% of the time, reducing evaluation primarily to deterministic page-retrieval verification.
“And for those that actually tried it, if you found the right page or the right documents, ninety percent of the time, you have actually the right answer.”
Right-sizing high-volume process workloads starts with building task-specific eval suites and searching open-weight models for the cheapest passing baseline before prompt-tuning or distilling down to 7B–27B domain models.
“You just need to make sure you set up your test cases, your e-dos, and your examples, and then subsequently just do the search process from there.”
LLMs are unreliable standalone verification judges because their verdicts flip 7–17% of the time across identical runs and change 7–34% of the time when challenged with a simple 'are you sure?' prompt.
“depending on the model and the situation, between seven and seventeen percent of verdicts change across what they call identical replications.”
LLM-as-judge scoring introduced variance comparable to the contract agent's own error rate, making calibrating a judge harness nearly as hard as building the underlying agent.
“Like we got so much noise compared to the noise with the with the agents that we're doing. Solving this problem is almost as hard as solving the whiteboard.”
As frontier intelligence commoditizes while aggregate token bills surge, durable economics come from attributing spend per workload, routing subtasks to fast specialist or open-weight models, tuning serving concurrency and KV caches, and owning a model-swappable stack rather than renting lock-in from frontier labs.
Practitioners attacked runaway inference costs across every layer below default frontier API calls. Featherless showed that splitting API keys by department, reserving superlinear low-latency serving (3x cost per 2x speed) for interactive paths, and leveraging higher tokens-per-second to entice developers onto smaller worker models deliver immediate ROI. Runpod's BenchPod tunes vLLM nightly builds, batch-token concurrency, and 4xH200 DeepSeek deployments live without 30-minute cold starts; Greptile routes 20 review subtasks across small and frontier models; Exa cuts trajectory tokens 95% via dynamic snippet highlights; Towards AI merges ten cheap GLM/Flash drafts with one high-thinking pass; and Ramp pairs frontier orchestrators with post-trained 35B Qwen models, 5x-cheaper open-weight security swarms, and semantic spend taxonomies—even as Apify warned that heavily subsidized flat-rate plans distort build-versus-buy incentives. Strategically, Nous Research argued that shrinking token margins force frontier labs to seek lock-in at the enterprise agent layer and compete vertically, making open, model-agnostic foundations essential. The counter-arguments cut both ways: naive turn-by-turn multi-model routing destroys prefix KV-cache hit rates, while falling frontier unit prices tempt teams to scale on frontier platforms rather than maintain custom post-training.
Also called: tokenomics · model right-sizing · owned vs. rented land · frontier orchestrator + specialist worker · disaggregated vLLM serving
Even when benchmark studies show harness choice no longer changes coding accuracy, harnesses drive major differences in task cost and speed, making token efficiency and model-mix routing durable differentiators alongside cloud runtimes and scoped secret permissions.
“If you think back to those two slides I showed, where the choice of harness did not make a significant difference in eval scores, both of those studies actually found that there was a significant difference in costs for tasks.”
Across 120 evaluated models, GLM 5.3 Flash offers competitive low-friction drafting at 50x lower cost than top-ranked Opus 5.5 extra high, while Opus 5.5 extra high matches Max Thinking quality at 2.5x lower cost.
“It's 50 times cheaper than the best one, which is Opus 5.5 extra high, which is two point five times cheaper than Max Thinking, but with same results.”
For production writing at scale, generating ten drafts with a cheap model like GLM or Flash and merging the strongest passages with a single high-thinking Opus pass beats a single frontier generation on both cost and quality.
“ten drafts of one specific thing with a cheaper model like GLM or Flash, and then have a smarter high thinking job like Office, or whatever you want, pick and merge the best parts of each of these drafts”
As practitioners switch readily among competing frontier models, falling per-token prices against flat serving infrastructure costs are compressing model inference margins despite surging total token spend.
“There was some charts going around Twitter this week showing the average cost of a token going down while the total spent on tokens surges, and the infrastructure costs associated with serving those tokens have stayed flat”
Frontier labs are moving up the stack to embed at the enterprise agent layer because owning workflows—where 70% of business output may soon be agentic—creates switching-cost stickiness and pricing leverage.
“If you're in front of the user, you have an opportunity for stickiness. Stickiness means margin. This is why the fight has largely shifted to the agent layer lately.”
Rather than selling commoditized API tokens at shrinking margins, frontier labs are partnering in sectors like finance to acquire domain context before solving lucrative vertical markets directly to recoup trillion-dollar investments.
“The best shot at earning it back is not to sell it for an ever-decreasing margin of others to solve these hard and lucrative problems — the best shot is to solve them yourselves.”
Frontier model capability advantages come primarily from data quality and scale rather than proprietary model architectures, allowing open builders to close capability gaps rapidly through synthetic model distillation.
“And the advantage in a model is very rarely in the architecture or some novel system that was created proprietary way for that model. In fact, it's usually in the quality of the data and the scale.”
Agents act as application operating systems combining deterministic code and data sources with non-deterministic model judgment, so enterprises should own an open, model-agnostic harness like Hermes Agent rather than rebuilding AI stacks.
“The next thing that you need to know about agents and about the new world is that agents are like operating systems, and they're like frameworks, and they're a lot more like those than they are like chatbots.”
Subsidized flat-rate LLM subscriptions—such as $100 monthly plans yielding $14,000 worth of tokens—destroy the build-versus-buy incentive for agents to pay external APIs when they can code bespoke tools from scratch at zero marginal cost.
“This is crazy because then the agents have no incentives whatsoever to, you know, to look for existing services because they can build anything from scratch because it costs nothing.”
While models, prompts, plugins, frameworks, and sandbox providers remain swappable, execution posture locks in system architecture because building a local POC couples every tool directly to the host process and filesystem.
“if you're guessing that you'll use first approach, where everything runs locally, then you write all your tools to use your host or app process environment, using your file system without separation and stuff like that.”
Because search cost compounds across an agent's downstream inference trajectory rather than a single tool call, returning dynamic snippet highlights across results delivers higher accuracy at 95% fewer context tokens than full-page text.
“we now do that across all of the search results, which means that you can actually get higher quality results at ninety-five percent fewer tokens than you would at full-page text.”
Naive sandboxed PR review agents burn $3 to $25 per pull request while flooding diffs with banal nitpicks that obscure catastrophic bugs for humans and push coding agents into writing overly defensive code.
“You just see lots and lots of comments and it's kind of annoying for you to just read through them, figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.”
Decomposing review into twenty specialized subtasks—reserving frontier models for the critical 20% like deep bug analysis while routing context reranking and flow checks to small models—slashes inference cost when guarded by evals.
“In practice, we have actually twenty different subtasks, and each of them has a very specific sense model to be used, and our small research team is constantly iterating exactly what models we need for the task.”
Per-request token speed scales superlinearly in hardware cost—roughly 3x cost per 2x tokens-per-second—so low-latency serving belongs only on interactive paths while background and CI/CD jobs run batched.
“It's not even close, like a rule of thumb that I sometimes say, right: for every two x of speed, it's three x of cost”
Issuing separate API keys per department and use case is the highest-ROI first step in cost optimization because 30–40% of organizations share a single key that hides workload economics.
“Like you need to start splitting up your keys. Start splitting up by departments. Start splitting up by major use cases so that you know where all your token spends are”
Developers rarely switch from frontier planners to cheaper worker models to save company budget, so serving the cheaper model at higher tokens-per-second is the incentive that actually drives adoption.
“People want to get their work done fast. If the model is smart enough to get it done, and it's cheaper, they will just use it because it's faster, not because it's cheaper.”
Pairing a frontier orchestrator with a post-trained 35B Qwen specialist model for financial question-answering surpassed the frontier model on exact-pass accuracy while serving Ramp Sheets at much lower latency.
“we partnered with the company Prime Insights to post train a Quan 3.5, 35 billion model, and we actually got above Bitcoin six, which was the frontier back then, exact pass accuracy”
In Ramp's 10,000-agent MapReduce pentesting swarm—which uncovered seven high-severity vulnerabilities including a multi-component email script injection—open-weight models up to 5x cheaper retain most of frontier detection accuracy when scaled across parallel runs.
“you can actually use open weight models that are up to five times cheaper compared to the frontier reasoning, and because you're set, setting it up as a swarm, the detection rate is not five times lower”
Ramp's token spend management product applies semantic taxonomy classification to API traffic so enterprises can attribute LLM spend to accounting categories and R&D tax credits rather than raw per-user or per-key totals.
“we created a taxonomy that contextually analyzes and semantically analyzes the LLM usage, so you can actually understand how LLMs are being used outside of just attributing it to a team or a person inside of your company.”
Ramp Labs' Portal projects latent post-training updates from an older specialist model onto newer model versions or families, amortizing post-training investment across base-model upgrades.
“Notably, one of the really interesting ones is Portal, where we figured out a way of transferring learnings from one post trained to another”
Restarting container images on every vLLM flag change stalls tuning behind 30-minute weight downloads and GPU cold starts, motivating BenchPod's live YAML reload and multi-node self-healing watchdog.
“You have to wait for cold starts to happen. It's all this loading, and you're not moving as fast. You're waiting like 30 minutes like for Kimi K3 to like download the weights, loading the GPU.”
Under concurrent Kimi benchmark loads, raising vLLM's max batched tokens leaves time-to-first-token largely unchanged while substantially cutting inter-token latency and end-to-end completion time on long agentic jobs.
“But an interesting thing that I did notice of increasing that is like the inter-token latency actually decreases quite a bit when you increase that batch tokens”
Teams deploying newly released open models frequently trigger serving errors, hallucinations, and KV-cache exhaustion by running stale vLLM releases instead of nightly builds and over-allocating context windows without sizing for concurrency.
“A lot of times, I see people are using the wrong v11 version. They might they're probably not using the nightly version. They have the wrong size for the contacts window.”
Right-sizing self-hosted inference means matching model scale to workload complexity—serving DeepSeek on four H200 GPUs handles routine production traffic cost-effectively, reserving frontier models like Kimi for long-horizon tasks.
“And just because it's the frontier model, it doesn't make sense. Like I said, like guy runs deepseek on a 48200s and does a really good job. Kimmy's more for like long tasks that I'm trying to solve something like very detailed.”
While multi-node vLLM pods suffice for roughly 50 internal users, scaling self-hosted inference to millions of users requires disaggregated prefill-decode serving over Ray clusters paired with dedicated KV-caching layers like LMCache.
“But once you start serving to, you know, millions of users, that's where I think, I mean, desired serving, and also like your caching mechanism is important.”
Model intelligence is no longer the primary constraint in financial services, shifting the strategic bottleneck from proving individual agent capabilities to building reusable infrastructure that scales deployments from one-by-one builds to many.
“And our opportunity here is scale — is not the model, our opportunity here is scale, is building the foundation for agents that can go one to many.”
Turn-by-turn auto-routing across many models invalidates prefix KV caches on multi-turn agentic workloads, so routing should be task-based by department label or restricted to two to three models.
“but if you put in thirty models and it keeps changing your model every turn, your low cash heat rate will start to wreck your optimization.”
Although total LLM usage at Ramp grew 123% and aggregate feature token spend keeps rising, unit economics continue falling as faster, cheaper frontier models ship.
“The use of LLMs has increased 123 percent, while at the same time, a lot of our top features are growing in token spend, but on a unit economic basis, they're actually becoming cheaper”
As frontier models absorb prompt-level coding tricks through a user-trajectory data flywheel, the durable agent harness converges on two layers models cannot internalize: isolated create-run-delete VM sandboxes and outer-loop context engines that supply institutional intent, living codebase maps, and precomputed KV caches.
OpenHands framed the central paradox of 2026 harness design: coding scaffolding makes harder tasks tractable, user trajectories feed model post-training, and models absorb those habits until harness choice barely moves coding benchmark scores—motivating self-cleaning guardrail shims that watcher bots delete once newer models stop triggering them. What models cannot absorb defines the lasting architecture. First is the execution boundary: Daytona showed that every competing framework reduces to create-run-delete VM primitives and is actively deleting local executors because embedded runtimes still expose host tools and a single allowed `python -c` command compromises the host; Arize and Greptile isolate every run in disposable sandboxes while persisting memory externally. Second is outer-loop context: Linear showed that five-cent intelligence shifts failure from coding capacity to product incoherence unless triage agents assemble customer value, logs, and prior decisions upfront; Greptile replaces grep with a living company wiki; Sonar Vortex feeds structural maps in-loop to cut token spend 36%; Ramp compresses 20–30x month-end context into offline KV-cache cartridges; GEICO routes edge-case HTML worksheets to domain owners; and Basis treats every working-window token as runtime training data. Yet as OpenHands noted in counter-point, entering unmastered domains like autonomous multi-agent colleagues immediately forces teams to re-add heavy prompting, coordinator hierarchies, and guardrails.
Also called: give the model a computer · the model ate the harness · outer-loop context engineering · living codebase wiki · KV-cache cartridges
In long-horizon agent harnesses, behavior emerges jointly from system prompts, progressive disclosures, tool descriptions, and model weights, so every token entering the working window must be engineered as runtime training data.
“really think of the context that agent is going to see as it's working as runtime training data. The word prompt, I think, trivializes it a bit, right?”
Coding harnesses obsolete their own context-layer scaffolding through a data flywheel in which scaffolding makes harder tasks tractable, user trajectories train frontier models natively, and harness choice stops changing coding benchmark scores.
“It turns out the harness is actually making itself obsolete through what I call the data flywheel. So in this process, the harness brings harder and harder coding tasks within reach.”
An agent harness divides architecturally into a foundational runtime layer—managing core loop execution and durable state persistence—and a context orchestration layer governing model routing, tools, memory, compaction, and behavioral guardrails.
“The runtime is kind of like the foundation of the harness. It controls where the core agent loop runs and how state is stored so that work can resume.”
Teams can ship deterministic guardrail shims for current model flaws without accumulating permanent codebase debt by logging every shim activation and having an automated watcher agent open a removal PR once newer models stop triggering it.
“So now we have a bot who watches this dashboard, and whenever a shim stops firing, that model or that agent will put up a PR and say this shim can be cleaned up.”
To iterate rapidly without Hyrum's Law lock-in or integration sprawl across Slack messages, CI status alerts, and timers, harnesses should hide internal mechanics from users and expose composable agent primitives like a unified event-subscription tool.
“Now, rather than having one separate integration, or you having separate integrations for each of these things, we have one framework and one tool that the model can use, that the agent can use to subscribe to an event.”
Supplying structural codebase maps and verification guide rails inside the agentic loop via Sonar Vortex reduces token spend by 36% and cuts commit-time issues by 92% on large, long-running tasks.
“And what we see when we provide agents in long-running, large, complex databases with long-running tasks, with this information is a 36 percent reduction in token cost because you're putting it in white claims.”
As intelligence that cost $100 three years ago drops to five cents and agents run unattended for hours or days, context can no longer be hand-fed interactively and must be orchestrated by outer-loop systems.
“I think one of the main reasons is that the context kind of moves out of that inner loop and largely into the outer loop.”
Cheap, abundant code shifts the core failure mode to product incoherence because agents reading source code alone miss user intent, prior design trade-offs, parallel org work, decision ownership, and recent revert risks.
“Implementation is not the bottleneck now. The failure mode here isn't an inability to build. The failure mode is like an inability to maintain coherence, to have a product that makes sense as a whole.”
Linear's triage agent deduplicates multi-channel intake and enriches issues via internal MCPs with customer value, request frequency, Datadog logs, stack traces, live bug reproductions, and prior related decisions before coding starts.
“It's been enriched with the customer information. Hey, this is a high value customer, this is the third time they've requested this in a month. Any stack traces, maybe if it's a bug, logs from DataDog that are associated.”
At each milestone, the war room generates interactive HTML worksheets and editable architecture diagrams that route sharp edge-case questions to domain owners and ingest human edits back into agent context.
“It was like, wow, that's actually a great question that we've never thought about identifying an edge case either in the business with technology.”
Embedded runtimes like V8 or QuickJS isolate generated glue code but still execute registered tools on the host machine, allowing a compromised agent to act maliciously through privileged database or email tools.
“these tools that your agent will call inside that runtime will actually run on your device outside that runtime.”
Host-level command allowlists and approval prompts fail to bound what a compromised model can do because permitting a single interpreter command like `python -c` grants unrestricted socket, filesystem, and code execution access.
“when you allow your cloud agent to run Python command with Python dash C, he can literally open any sockets, modify a file system, execute code, or do basically everything”
Across 2026 releases, agent frameworks such as CrewAI and AG2 are actively deleting legacy local code executors and forcing execution into isolated VM or process sandboxes that can be discarded upon compromise.
“And you can notice that they are not adding sandboxes, they already had them, but they are deleting all the alternatives”
Sandboxes have evolved from short-lived single-snippet executors into full agent runtimes and massively parallel fleets of thousands of concurrent environments powering reinforcement learning post-training, evaluations, and subagents.
“if they can run thousands of these executions environments concurrently, they can do really interesting stuff, especially regarding reinforcement learning, fine-tuning, post-training benchmarks, evals.”
Whether exposed as a tool, an external plugin, or a subclassed internal API, competing agent frameworks all reduce sandbox integration to the same three lifecycle primitives: create, execute, and delete.
“But under the hood, all of these ways speak the three same lines, so it's sandbox creation, running some code or task inside the sandbox, and sandbox deletion.”
Signal decouples ephemeral compute from durable state — tearing down sandboxes, credentials, and repo access after each run while checkpointing step state, detected issues, traces, and learned skills externally across model- and harness-agnostic runners.
“What stays is that if it found any issues, all of the output is actually saved off in some kind of memory, so that the next time an agent gets spun up, it doesn't have to start from ground zero.”
Replacing imprecise, token-heavy grep searches with a continuously maintained wiki that synthesizes codebase architecture, production telemetry, docs, and tickets gives review agents surgical tribal context.
“Now, the way that Greptile approaches this problem is we actually disintermediate all of these context sources with a living wiki. This wiki is essentially a sort of world model of how your company works.”
Offline KV-cache cartridges compress 20x to 30x more predictable context—such as ledgers and aging reports prepared overnight before month-end close—with lower engineering lift and broader generalizability than post-training.
“cartridges are really interesting family of techniques where you're able to compress a lot of context, up to 20 to 30 times more context, into the KB cache offline and serve it for your agent.”
Expanding beyond single-session coding into autonomous knowledge work requires shifting harness architecture to long-lived coordinator agents that own strategy, dispatch worker fleets, and reintroduce domain prompts and action-versus-asking guardrails.
“So Rockbot has a new kind of harness that operates not based off of one-off chats, but long-lived coordinator bots that orchestrate a fleet of worker agents.”
Outside software engineering, enterprise agents remain near 5% production adoption not for lack of raw model intelligence, but because non-deterministic workers require institutional trust infrastructure—automatic registries, build-time policy DAGs, risk-tiered entitlements, and opinionated workflows—between reckless autonomy and paralyzing lockdown.
Tenex quantified the conference's starkest macro reality: across roughly 100 large enterprises, general-purpose LLM access sits near 90% and coding agents near 50%, yet autonomous agents in non-engineering business workflows sit around 5%. Anthropic, Apollo, GEICO, Basis, and Apify all traced that diffusion gap to governance, process, and trust friction rather than model IQ—with Apify showing that even after eighteen months of agent payment standards, 30-day volume sits near $1M on x402 and $1,500 on MPP because static API keys, untrusted endpoints requiring marketplace middlemen, and deferred-settlement double-spend risks dominate the funnel. The architectural remedy across regulated firms was bounded autonomy: Two Sigma replaces months-long manual review queues and rubber-stamp approval fatigue with automatic framework registration, risk-scored scrutiny, registry-compiled runtime entitlements, and conservative classifiers that escalate only ambiguous calls; Coinbase separates build-time normative policy ('the original institutional system prompt') from mutating runtime state inside compliance DAGs; Basis encodes opinionated, auditable tax and accounting workflows rather than 'Move 37' creativity; and Anthropic scopes financial agents with least-privilege identities and deterministic hooks gated on advisor or SME sign-off. Qualifying this optimism, Daytona showed that host-level policy allowlists collapse if an interpreter command is permitted, while GEICO's one-day war rooms proved outer-loop enterprise inertia can be broken when decision-makers sit inside the loop.
Also called: the 5% enterprise diffusion gap · bounded autonomy · policy as original system prompt · automatic agent registry · opinionated workflows
Enterprise diffusion depends on predictable, repeatable execution calibrated to blast radius rather than 'Move 37' creative novelty, because organizations scale output through standardized workflows their teams can audit and trust.
“Nobody wants their agent to move thirty-seven their tax return. They want it done reliably at scale in a way that they can trust, and in a way that their people know how to work with.”
General-purpose coding-style agent products fail to raise enterprise-wide productivity because pushing prompt, memory, and workflow management onto end users creates both a user competency gap and an unstandardized process gap.
“The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user.”
Institutional banking and asset management remain early on the AI adoption curve because off-the-shelf research summarization delivers low value compared to automating complex middle- and back-office data workflows.
“In finance broadly, especially in call it the asset management, investment banking, the bank institution institutional world, I still think it's very early.”
Production financial agents enforce operational safety through least-privilege role identities, deterministic hook guardrails, and sandboxed write boundaries where agents can only propose booking amendments that a human SME must approve before execution.
“Once they approve, then that amendment will be made, and the change will be applied. Guardrails, deterministic guardrails, called hooks around your agent, so that it's never overreaching beyond what you'd like it to do within its confines.”
Model intelligence is no longer the primary constraint in financial services, shifting the strategic bottleneck from proving individual agent capabilities to building reusable infrastructure that scales deployments from one-by-one builds to many.
“And our opportunity here is scale — is not the model, our opportunity here is scale, is building the foundation for agents that can go one to many.”
Anthropic applied this shared-foundation pattern at the industry level for wealth management by combining twenty-five pre-built custody, CRM, and portfolio connectors with eight open-source workflow skills gated on mandatory advisor approval.
“We built a plugin that is available on GitHub that builds eight skills and workflows that an advisor can take out of the box, use today, and implement tomorrow. Layer four approval, the advisor proves everything that comes out of Claude.”
Although coding agents have collapsed inner-loop coding time and offloaded routine patch work, major enterprise architecture changes remain bottlenecked by the outer loop of cross-functional alignment, risk governance, and decision-making.
“So while we've seen this interloop speed up to use Jean Kim's concept, we find ourselves seemingly stuck within the bounds of this outer, which is to say, everything that happens other than the current moment.”
Neither total sandboxing nor written policy documents and prompt instructions provide workable enterprise safety, because air-gapped sandboxes prevent real work while prompt-level rules cannot guarantee runtime enforcement.
“And then, if you just have policy documents that you just look at, or that are part of your specification for the prompts, that's also not very forceful at runtime. You can't guarantee that will be part of the enforcement controls.”
Instead of blocking teams behind an onerous pre-registration gate that drives experimentation into ungoverned shadow IT, approved agent frameworks automatically record each agent's owner, product, declared tools, and endpoints in a central registry.
“It just happens when you use one of the firms approved agent building frameworks. It's automatically registered. All that information is recorded there for you, and if you want to do a manual registration, you can certainly do that.”
Applying uniform manual review to every agent stalls low-risk experiments and drowns reviewers in trivial requests, so governance queues must be risk-scored based on each purpose-built agent's declared tools and target systems.
“Again, that feels safe, but now we've got a trivial summarizer waiting two months for the same sign-offs as an agent-based captioning production system.”
Managed agent runtimes avoid blunt firm-wide allowlists by querying the agent registry at initialization time and compiling each agent's declared tool, host, and resource entitlements into enforceable runtime boundaries.
“when agents run in our managed environments, they look at the registry we talked about before to look up what policies apply to it at initialization time, and that policy gets translated into enforceable statements.”
Asking humans to approve every agent action induces rubber-stamp review fatigue, so production runtimes should handle clear rules via deterministic fast paths and filter routine calls with independent conservative classifiers, escalating only genuinely ambiguous actions.
“when explicit rules apply, we want deterministic fast paths. When explicit rules don't apply and there's some ambiguity, you want a classifier to do that work instead.”
Variable pay-per-use compute jobs cannot use naive deferred-settlement 'up-to' payment schemes because buyers can double-spend wallet funds before on-chain settlement, forcing providers toward upfront overcharge-and-refund workarounds or on-chain escrow with batch settlement.
“even though like it allowed you to charge random costs, there was not nothing preventing the buyer, the agent, to spend those money wise, you know, because the transaction was only like settled on blockchain later.”
Despite more than a dozen competing agentic payment protocols launching over eighteen months, actual 30-day on-chain settlement volume is negligible at roughly $1 million on Coinbase's x402 and $1,500 on Stripe's MPP.
“in the last 30 days, on this road too, there was just only like one million dollars. Nobody. And on MPP, only like one point five thousand dollars. I mean, it's basically nothing.”
Agentic payment protocols lose today not to card rails directly, but to pre-provisioned SaaS accounts and static API keys billed to a developer's credit card, which are far simpler when tool needs are known upfront.
“Like authentication are competing with Visa. It's they are competing with API keys because it's just the more convenient way.”
Because autonomous agents cannot verify whether an arbitrary external endpoint will deliver promised results or honor pricing, paid tool discovery relies on curated connector directories and marketplaces acting as trust intermediaries.
“same for ChatGPT, they have apps, and Apify is in marketplace as well, so basically there is just a middleman that tells you, okay, this is the service you can trust to know, and you're going to get results.”
Enterprise diffusion drops by 30 percentage points between provisioning an LLM (90%) and formally training employees to apply those AI surfaces in daily workflows (~60%).
“Under that is people actually getting trained in how to use the technology, how to actually use these surfaces in their work — that's like sixty percent.”
Autonomous agents running in non-engineering business functions sit at roughly 5% enterprise adoption, defining a 12-to-36-month diffusion frontier alongside open-source versus fine-tuned model selection and scaling AI engineer hiring.
“the final one where we are truly still day one is agents actually working in non-engineering workloads in an organization, that is around five percent right now.”
Roughly 50% of surveyed large enterprises have deployed coding agents inside software engineering and product workflows, making software delivery the leading production agent beachhead.
“Next up is coding agents, agents that are actually used within engineering and product workflows — that's around 50% of all the enterprises we've been in”
Across roughly a hundred large enterprises Tenex advises in financial services, healthcare, and industrials, foundational employee access to a general-purpose LLM has reached about 90% penetration.
“So ninety percent of the enterprises we work with have rolled out general-purpose LLM.”
In enterprise governance, agents act as a cross-cutting operational model spanning people, process, and technology simultaneously—collaborating with human staff, executing workflow steps directly, and operating within explicit process boundaries.
“So I see them as an operational model cutting across all three. They work with people, execute parts of the process themselves, and they use and are made up of technology.”
Regulated systems must architecturally separate build-time policy definition from the run-time production system, which mutates continuously through product changes while remaining bound by policy constraints.
“Everything is expected to still be roughly bound by policy, but ultimately, we separate this from the run state and the build state.”
Written enterprise and risk-compliance policy—expressed through normative must, may, and should rules—functions as the organization's foundational system prompt governing how operational decisions are made.
“Think of policy as the original institutional intent, or the original system prompt.”
One-day, thousand-dollar war room sprints over millions of lines of legacy claims code delivered concrete outcomes across three domains, including a rentals API extension saving 8,500 calls monthly and a salvage plan that caught a production bug.
“the agent used some interesting process to find and distill data to figure out that we can connect a couple dots and save ourselves eight and a half thousand calls per month by giving the customer up-to-date rental information.”
Host-level command allowlists and approval prompts fail to bound what a compromised model can do because permitting a single interpreter command like `python -c` grants unrestricted socket, filesystem, and code execution access.
“when you allow your cloud agent to run Python command with Python dash C, he can literally open any sockets, modify a file system, execute code, or do basically everything”
Because coding agents generate pull requests far faster than humans can read them—and reviewers accept wrong AI output four times out of five—shipping confidence must come from independent, multi-layered verification harnesses and runnable behavioral proofs rather than manual line-by-line diff inspection.
Software delivery talks confronted the review bottleneck head-on: Sonar reported a 2,450% explosion in AI-generated code yielding only 30% more releases because human reviewers succumb to 'cognitive surrender' (agreeing with wrong AI output 80% of the time) while multi-step agent accuracy decays exponentially across long horizons. Teams moving toward lights-on and dark software factories replaced manual line reading with executable proof loops. In the verification workshop, external MCP gates (`Ctx`) refuse agent self-reports until deterministic CLI and Playwright checks verify backend persistence, making verifier suites more durable than the application code itself. At Anthropic, merging 30,000 daily changes relies on automated first-pass reviews, edit hooks, and browser-driven screenshot and video evidence; Sonar pairs CPU static analysis with GPU generative checks and CVE scanning across every loop; Greptile inverts author and reviewer model families (+10% bug catch) and boots full-stack VM sandboxes for vision-model UI QA; Linear wires verification skills and product-history review agents into unattended bug fixes; GEICO enforces deterministic nine-step artifact gates; Untangle caps builder-reviewer loops at three iterations; and Coinbase runs parallel agent reviews inside gated DAGs. Against pure automation, Untangle insisted that a human product owner watching each feature video remains irreducible, and Featherless warned that unowned generation lobs 'slop grenades' onto peers.
Also called: who verifies the software factory · zero-trust verification · cognitive surrender · author-reviewer model inversion · executable proof commands
Software factories operate along an autonomy spectrum from lights-on pipelines—where humans apply taste and judgment across agent loops—to dark factories where agents scope, build, and verify continuously while humans set intent and handle exceptions.
“A lights on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person so apply your judgment and taste inside the picture”
With raw code generation largely solved, the highest-leverage engineering investment is building automated tests and benchmarks that let agents self-verify correctness and completeness for a two-to-threefold outcome gain.
“Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct.”
Scaling from roughly 2,500 to 30,000 daily merged code changes at Anthropic relies on automated first-pass Claude reviews, diff-triggered edit hooks, and agents exercising UIs in browsers to produce screenshot and video verification evidence.
“If I care about mobile or desktop experience, maybe it generates me screenshots or videos of that, but basically, you want to get to a place where the AI is giving you as much verification evidence as possible.”
Achieving high-confidence automated merges as unreviewed production code rises requires concrete behavioral checks—specifically end-to-end tests, property-based tests, and visual verification—rather than relying on manual line-by-line diff inspection.
“We lean a lot on end-to end tests, property tests, any visual verification tests—things that can help us just get that confidence to merge those changeset automatically.”
Enterprises should enforce zero-trust, multi-layered verification across the agentic loop, CI, and maintenance by combining deterministic static analysis for bugs and SQL injections, agentic review for logic and intent, and supply-chain scanning for CVEs and malware.
“We absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach, and verification should be done in every loop”
Explicitly labeling agent-authored pull requests preserves regulated auditability and triggers stricter automated verification constraints while keeping a named human engineer accountable for production outcomes.
“I personally think it's fine for the code to be written, labeled as being coming from an agent”
Human-in-the-loop review fails to convert a 2,450% surge in AI-written lines of code into proportional delivery (yielding only 30% more releases) because reviewers exhibit 'cognitive surrender,' accepting wrong AI output 80% of the time.
“they agree with AI when AI is right, 93 percent of the time, and four out of five times, they agree with it when the AI is wrong.”
Applying Toyota-style quality engineering to agentic software factories requires four multiplicative pillars—explicit intent, in-loop guide rails, independent verification, and auditable evidence—where missing any one pillar collapses overall confidence to zero.
“And we multiply these together because we've seen a number of situations where one of these elements is missing, and if it's zero, the confidence goes to zero.”
Pairing CPU-based deterministic static analysis with targeted GPU generative checks provides independent verification that left surveyed SonarQube users 81% more likely to see positive security impact and 76% on outage frequency.
“So we look at deterministic mixture of methods, and then you add in a generative mixture of methods, and it comes up with what we think is the gold standard of verification.”
Combining enriched intake context with agent verification skills and failure feedback loops makes unattended implementation reliable enough to auto-fix small triaged bugs locally or in the cloud.
“Between the base, the base context, being able to verify and being able to see failures, we see a massive amount of success in our team.”
Rather than checking code logic in isolation, a context-aware review agent joins PR diffs with original customer intent, session decisions, ownership, and historical reverts—catching rejected UI patterns, inverted label hierarchies, and DB transaction risks.
“It starts with that original issue and that base context that we created with the customers, the original intent, and then it goes and looks for product history, so things like previous decisions, reverts, regressions and incidents in the same area.”
Executing the nine-step modernization workflow across cloud code-dive agents and laptop-hosted enterprise MCPs relies on deterministic gate checks verifying that each step produced its required artifacts before advancing.
“And just an interesting detail, and this is something that the code monetization plugin does: is it does like a deterministic analysis of like, did the previous step create the artifacts that needed to include the file?”
Because model families exhibit distinct post-training blind spots and fail identically during generation and review, pairing an authoring agent with a different model family for review increases bug catch rates by 10%.
“And so you want to invert the model. If you use hard code to write the code, you want to use the GPT model to review it.”
Booting the full application stack inside configurable dependency-rich VM sandboxes with documented user-flow maps allows computer-use and vision models to catch layout and UI regressions that static diff inspection misses.
“And using this, you can now test your content and use vision models to say, oh, this button technically appears, but it's actually hidden on iPhones. It's not in the first fold on the iPhone Duo.”
Application-specific deterministic verifier CLIs become more durable assets than the application code itself because they allow teams to rewrite or port implementations while proving behavioral equivalence.
“So when you want that, if you have all those verifiers in place, then your software doesn't matter as much. You can write it again and prove that it works.”
Shallow UI-only assertions produce false-positive agent completions when optimistic DOM updates mask missing backend or storage persistence across page reloads.
“And yes, it did do something to the UI, but we didn't check the other half. And so it was never persisted, and we lost that email.”
Emitting explicit, state-detailed error messages from application and test tooling turns failed verification runs into closed feedback loops that agents can diagnose and self-heal autonomously.
“So if you make it so that your failure is explicit about why it failed, and give as much detail as possible about how it got into that state, then your AI can just self heal from that”
Defining tasks by an executable proof command rather than step-by-step procedural instructions improves frontier agent performance by constraining the target outcome while leaving exploration paths open.
“It's more about giving it the map, saying you must get here. You'll know you're here when this command returns to us.”
Pairing deterministic proof scripts with fast System 1 judge models evaluates semantic UI correctness without brittle string-matching contracts breaking whenever confirmation copy changes.
“So how would we fix that without adding a lot of tight coupling to everything that we're doing? The answer is the System One models, which we'll get into here.”
Centralizing verification authority behind an external MCP web service decouples workflow completion gates from agent self-reporting, marking tasks satisfied only after independent CLI and API checks pass.
“They go off and do that work. They come back through the MCP connection to Ctx and say, "This is what I've completed. Am I done? Please verify."”
Modeling gated approval flows as directed acyclic graphs (DAGs) lets agents execute previously cost-prohibitive independent reviews in parallel and aggregate their results without bypassing required compliance gates.
“Previously, some work used to cost too much or be too complex to coordinate in this way. Now we can run independent reviews alongside each other, bring results together, but keep the required checks in place.”
Builder agents run automated review loops capped at three iterations alongside child QA processes, escalating unresolved bugs to a manager agent and only then to the human.
“We typically stick to a three-review max, where we allow the builders to loop about three times with reviews.”
Unchecked AI generation creates downstream review bottlenecks when unverified code or decks shift verification onto peers, requiring strict human ownership of outputs and outcome-per-dollar metrics rather than token usage.
“to prevent the slop grenade from happening, enforce that people should own what they hand over or generate.”
Every builder agent records a video of the feature passing automated tests, making human product-owner inspection of that video on a laptop the irreducible quality gate before merge.
“Every builder records a video of the feature being used and passing tests.”
Software engineering is shifting from writing code locally to managing asynchronous cloud agent fleets and closing the trace-to-skill loop—centralizing transcripts and tool calls, grading runs post-merge, and letting meta-agents distill recurring failures and human corrections into versioned skills.
Across Warp, Arize, Linear, Untangle, GEICO, Anthropic, and Runpod, the unit of engineering work moved from crafting individual features to operating and improving the agent factory itself. Warp demonstrated 'factory engineering': shared cloud Docker sandboxes automatically centralize conversation transcripts, MCP/tool calls, and diffs, exposing silent agent workarounds (like brute-force port probing) to post-merge markdown judge rubrics and daily meta-agents (`Skill Doctor`) that open measurable skill PRs. Linear runs a daily post-merge distillation job that turns PR review corrections and reverts into `agents.md` updates; Arize designs agent-first telemetry to wake scheduled sandbox sweeps while tracing both production agents and the coding harnesses behind them; Anthropic treats SME-authored plain-text skills as living, versioned code co-iterated by business teams and engineers; GEICO coordinates legacy transformations through Markdown FleetKit protocols; Runpod exposes GPU benchmark controls as FastAPI endpoints for agents to drive directly; and Untangle replaces local development with communicating builder, reviewer, and merge-manager sessions—extending the paradigm to hiring by stack-ranking candidates on 45-minute recorded agent-operating trials, echoing the Day 2 keynote's call to pair human company curation with career-research agents. In tension with universal 'agent management,' Basis warned that pushing prompt and skill authorship onto non-technical enterprise users creates a severe competency gap, and Day 2's keynote cautioned against letting agents automate personal outreach like resume submissions.
Also called: your job is now agent manager · skill-improving skills · factory engineering · agent-first telemetry · FleetKit orchestration
Factory engineering treats the agent harness itself as an optimization target by using LLM-as-judge scoring agents to evaluate past runs against custom rubrics for efficiency, code quality, and test coverage.
“You can set up some scoring agents to score the quality of how those conversations went, so think of this like an LLM as a judge reviewing against rubrics that you define across metrics like efficiency, or code quality, or test coverage.”
Introspecting team-wide agent performance requires centralizing three artifacts per session—dialogue transcripts, external tool/MCP/filesystem calls, and resulting code diffs—which shared cloud Docker sandboxes capture automatically without syncing local laptop folders.
“you got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output, which is something that you may want to grade for quality purposes.”
Because unattended agents silently brute-force around broken environment instructions—such as probing local port numbers when test skills are stale—scoring tool-call efficiency over full traces uncovers harness bottlenecks hidden by passing PRs.
“these AIs are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it.”
Triggering markdown-defined judge rubrics upon PR merge produces structured pass/fail scores and failure rationales that a daily cron-scheduled self-improvement agent mines to propose targeted skill and codebase updates.
“So automatically, usually the trigger being a PR merged, kick in an agent to review that conversation across your grading criteria.”
A single-player batch audit skill (`Skill Doctor`) crawls local Claude, Codex, and Pi logs to produce a rubric report card and concrete skill edits, such as upfront port-discovery instructions and prescriptive changelog formatting.
“But this is a skill that you can run on your single-player setup, and it'll crawl through all of your agent conversations in Claude, Codex, Py, or whatever else and just do a one-time grading.”
Unlike opaque overnight memory or dreaming loops, continuous rubric dashboards let teams attribute measurable score inflections directly to merged skill and config fixes such as enabling missing subagent permissions.
“So the agent's constantly scoring setups and it's giving grades of like 50%, 60% on average, and you can attribute changes that you made in your agent setup to inflections in the score.”
A high-signal job-search pipeline combines human curation—first ranking conference speakers' companies by technical interest—with an agent that scrapes career pages to match open roles and extract published compensation ranges for updating rankings.
“The next step is to send your agent to those companies' careers pages, right, and find them open positions to match them to your experience, and collect the published compensation ranges.”
Hiring serves as a primary two-sided incentive for attending AI Engineer, extending even to conference speakers who are simultaneously recruiting talent and exploring new roles themselves.
“I spend the day talking to other speakers and participants, and what I've noticed is that one of the strongest incentives to come to this conference is hiring, to the point that even some speakers are hiring and also looking for work.”
A daily post-merge automation mines review corrections and reverts across merged PRs and distills actionable lessons into `agents.md` and documentation so human product judgment compounds across subsequent agent runs.
“It will find actionable lessons, places where these types of comments happened in the PRs, and changes were made, or reverse happened, extract learnings, and then inject them back into the codebase, either in the agents.md or in the related documentation”
Domain-specific workflows such as trade-break investigations should be captured from subject-matter experts as plain-text skills that are version-controlled, frequently iterated, and governed like living production code.
“this is the business context that an SME provides, and we think about skills as living, breathing, versioned code, essentially, even though they're just plain text. They're meant to be dynamic. They're meant to be updated frequently.”
Agent deployment reshapes the financial-services operating model into a tight loop where business SMEs directly correct live agents and propose changes that engineering validates against baseline evaluation suites before shipping as updated skills.
“Some observations that we've seen, especially in financial services, is that the business is now becoming closer to engineering, and the way that we're observing this is that the business will directly work the agent that engineering has built.”
FleetKit governs expensive multi-agent architecture transformations via structured Markdown rules that decompose goals, dispatch research agents, stage plans in a human approval queue, and only then dispatch builders.
“Takes a goal, breaks it down, dispatches leaders to do all the research and decide about the approach, puts everything into this like human loop cube.”
Out of roughly six billion people who have touched LLMs and 20 million professionals accelerating multi-week tasks into days, only about 5,000 frontier practitioners currently operate swarms of thousands of concurrent always-on agents.
“But really, those at the frontier labs are just super small percentages, like five thousand users are the ones who are running thousands and thousands of agents at once.”
AI engineering shifts from manually writing code, reviewing diffs, and checking dashboards to constructing automated build-eval-ship-monitor-improve systems where agents review non-deterministic execution paths and analyze telemetry.
“As an AI engineer, your job is no longer to just write the feature, write a bad feature. It's about how do you build the loop to write that feature.”
Designing telemetry for agents as primary consumers turns scheduled 3-to-6-hour trace sweeps, metric thresholds, and Slack user complaints into automated triggers that wake isolated sandbox runs behind a single traced proxy.
“how do you set up your telemetry data so that agents are the primary consumers of that data? And what that does is that observability data can become the trigger for actually running and triggering these always-on agents”
End-to-end observability must cover both production agents and the coding harnesses that build them (Claude Code, Cursor, Codex), using online evaluations over live traces to feed newly detected failures back into the improvement loop.
“And tracing everything now means not just tracing the agents, but also tracing the coding agents behind the agents that build this.”
A dedicated merge-manager agent session coordinates the wave of concurrent pull requests to prevent shared-state collisions such as conflicting database migration numbers.
“Like if all the agents are trying to grab these things, you constantly collide, right? So the merge manager just helps all that go away, but also is a little more intelligent about how the merge deals happen.”
Engineers who still code in local development environments bottleneck their teams because local workflows cannot orchestrate fleets of parallel, inter-communicating cloud agent sessions.
“I don't care whatever it is, but if you're running local dev, you are screwing up. You're holding yourself back. You're holding your team back. You're holding everything back, because you can't do this.”
Untangle eliminates resumes, pre-screens, and video interviews to avoid interpersonal bias, evaluating candidates instead by having models stack-rank unguided 45-minute screen recordings of them operating agents to ship a PR.
“You'll like their hobbies. You'll like the way they talk to you. You'll think they're interesting, and it's all noise. None of it is real, right? All that matters is can they operate an agent?”
The top five stack-ranked candidates complete paid 16-hour recorded trials shipping a real PR under NDA, with AI and human video review selecting finalists before any phone conversation happens.
“Then we basically pick the top five, and we invite them to a paid project. So we pay a fixed fee for sixteen hours at a time.”
Exposing GPU telemetry, error scans, and vLLM configuration reloads over backed-up FastAPI endpoints enables coding agents to run serving benchmarks and optimize flags programmatically without human UI intervention.
“It's all based fast API calls, so your agent easily just call all those things. You can actually just serve everything. Like you don't even have to use the UI.”
Job-search agents should stop at role and compensation research rather than automating resume submission, leaving candidates to start real in-person conversations grounded in speakers' talks.
“And please don't make your agents send out your resume automatically.”
General-purpose coding-style agent products fail to raise enterprise-wide productivity because pushing prompt, memory, and workflow management onto end users creates both a user competency gap and an unstandardized process gap.
“The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user.”
In institutional investing and private markets, off-the-shelf web chatbots fail on stale or conflicting inputs; agents earn front-office trust only atop governed firm ontologies, validated golden sources, unified MCP connectors, and hybrid code-ML-LLM-human pipelines that write synthesis back into the vault.
Every institutional finance talk located competitive advantage in governed data infrastructure and workflow design rather than raw model selection. Bridgewater contrasted an open-web chatbot answering a Korean won query from a single three-month-old note with Vault—which aligns 100M+ documents to a bitemporal ontology, serves 1,000+ QPS agent storms via turbopuffer cluster-local native filtering, migrates researcher email habits into saved collections, and writes human-agent workbench synthesis back into the corpus so knowledge compounds. At Apollo's Redding Ridge credit platform, engineers sat daily with portfolio managers to codify golden-source precedence across 20 conflicting systems, turning multi-hour weekly loan reconciliations into one-minute automated QA reports and cutting 200–400-name CLO portfolio construction from days in Excel to minutes for the first 80–90%. Coatue modeled the hedge fund as a composite state-machine function `(external_context, internal_context) -> (stock, size, timing)`, insisting that every pixel a human sees must be API-accessible while assigning each step whichever runner—deterministic code, classical ML, LLM, or human—maximizes quality. Exa supplied the external temporal layer with 400 billion dated web snapshots built for dense exploratory agent queries, while Anthropic standardized multi-system enterprise access behind a single MCP connectivity layer.
Also called: hedge fund as a function · bitemporal firm ontology · golden data sources · compounding investor workbench · hybrid code-ML-LLM-human runners
Rowspace deploys agents to perform continuous data engineering—reconciling fragmented records across disparate enterprise systems—so portfolio managers and traders act on unified data rather than querying isolated chatbots.
“We essentially use agents to do the work of data engineering, right information or reconciled data from across a variety of systems to the portfolio managers, traders, and so forth can make decisions faster and more accurately.”
Weekly warehouse loan trade reconciliation—previously requiring two to three staff members hours of manual bank-statement checks—now runs automatically in a minute, shifting the team's role to daily QA.
“That report was then created and now repeatable in the span of a minute, and now all of that is automated.”
Combining indenture waterfall rules, compliance constraints, and real-time pricing cut 200-to-400-name mock CLO portfolio construction from two-to-three days in manual Excel to minutes for the first 80–90% of the build.
“that allows these PMS to kind of build something to call it eighty, ninety percent completion, and then kind of use their judgment and their thought to further refine that portfolio, that last 10 percent, to make it theirs.”
Because a single financial data point can appear across 20 conflicting systems with only one authoritative source, earning trading-desk trust required engineers sitting daily with PMs to encode golden-source precedence rules.
“Michael sits with our team on a daily basis, learning and understanding our information flow, because for one piece of data that might come from 20 different sources, and there's only one that's actually right.”
AI grounded in validated proprietary data creates a durable competitive advantage in private credit—rather than becoming a commodity like Excel—by giving managers real-time accuracy on available liquidity and pricing.
“So knowing that you have a strong process in place to go out and execute, and knowing your available liquidity, or you know where things are pricing and what the value is at any given point is crucial.”
Manually wiring individual agents to enterprise data systems creates a brittle integration matrix as use cases multiply, so organizations should standardize source-system access behind a single secure MCP connectivity layer.
“as you think about scaling up your agents into new specialized, various tasks, you'll want to have those agents work on one secure connectivity layer, which is MCP.”
Modeling a hedge fund as a function compressing external market data and internal firm context into `(stock, size, timing)` decision tuples frames portfolio construction as a composable engineering pipeline.
“So at a high level, you can think about a hedge fund as a function that takes in two parameters — context external and context internal, and spits out one, two, volt selection size timing. What stock? How much? When?”
Each sub-function should be assigned whichever runner—deterministic code, classical ML regressions, LLMs, or human judgment—maximizes absolute quality for that step rather than automating with LLMs by default.
“There is no glory in automating things for the sake of automating things. The goal is not to take as many things and make them LLMs as possible.”
Composing multi-step transformations across every firm data source with 10+ hyperparameters and 20+ values per node yields an effectively infinite graph whose core challenge is technical and organizational legibility.
“In each of those nodes, we just talked about, at least ten hyperparameters with at least 20 different valid values for them. So the search space is basically infinite.”
Agent capability is wasted unless every external market feed and internal firm record—positions, liquidity and risk constraints, and decades of trade decision notes—is exposed directly via API rather than UI copy-paste.
“Every single thing that a human can see in pixels on the screen, an agent in your building should be able to see via API.”
Bridgewater ingests over 100 million documents from 135 external sources plus internal memos and debate transcripts into Vault, enriching each document with ontology tags for countries, concepts, markets, and discussed time periods.
“It is carefully aligned to our internal ontology with enrichments like the countries, concepts, markets, and most importantly, time period discuss in the document.”
On a live Korean won query, an open-web chat agent relied on one stale three-month-old note out of 15 results, whereas connecting the same agent to Vault time-scoped the search, matched 52 candidates, and synthesized seven current sources.
“And so for us, this is the big lesson for building vault to kind of motivate the talk, which is that it takes great data and the right connectors to make AI powerful.”
Because production queries nearly always filter by date, source, or asset and agent storms exceed 100 to 1,000+ queries per second versus 100 per hour for humans, Bridgewater switched to turbopuffer's cluster-local native filtering to pair post-filtering speed with near pre-filtering recall.
“Their approach, which they call native filtering, looks only in the nearest clusters if it can matches, and then scores just those.”
Winning daily investor adoption required replicating familiar broker-email workflows inside Vault — converting distribution lists into saved searches and inbox folders into saved collections — before introducing cross-dataset queries, dynamic topic feeds, and CLI/Slack agent access.
“So that's exactly what we did: everything searchable, the subscriber lists became saved searches, and folders became saved collections. It's the same habits, just a new home.”
The proactive Investor Workbench pairs always-on topic cards and raw source evidence with deep-research agents like Pocket Analyst, automatically writing the resulting human-agent synthesis back into Vault so institutional knowledge compounds across research cycles.
“And when they dig deeper alongside the agent to look at the dynamic driving this question, the learnings are automatically captured — the results of their collaboration are automatically captured back in the model.”
Unlike humans who enter brief keyword queries and stop early, AI agents issue high volumes of detailed, exploratory searches per task, requiring search infrastructure purpose-built for agentic retrieval.
“They tend to describe what they want to search in great detail, execute many searches, and want to understand the space more fully before they pass, and that means that they do way more searches on a post basis.”
By archiving page states during continuous freshness crawls, Exa Snapshot exposes over 400 billion daily-indexed web snapshots for both point-in-time URL lookups and historical semantic search queries.
“So just recently, we released Exa Snapshot. It's 400 billion plus snapshots of the web, and you can search them by day.”
No speaker argued that general-purpose models or ungrounded web search could support institutional capital allocation without proprietary governed data; Apollo, Bridgewater, and Coatue unanimously restricted ungrounded chat to low-stakes background research.
Theme 08 · 2 talks · 11 claims
Spatiotemporal World Models and Real-Time Simulation
Scaling laws extend beyond turn-based text to continuous video and action streams, turning pixels-to-actions policies and next-frame predictors into real-time world simulators that internalize physics, control robots, and replace static 3D engines with interactive visual interfaces.
General Intuition and Runway pushed AI engineering past autoregressive text into continuous space and time. General Intuition argued that real-time agents must observe while acting, leveraging Medal's gameplay pipeline—scaling toward three billion videos a year and 1.5 trillion monthly controller actions—because high-APM game controllers share the exact action space of low-DOF robot teleoperation and plug neatly as reactive control layers beneath frontier LLM planners. Runway bet on compute scaling laws for next-frame video prediction, demonstrating that models trained at scale internalize geometry, gravity, occlusion, and object permanence directly from pixels—unlocking action-conditioned robotics simulators, engine-free World Interface Models with no Unity or Blender mesh underneath, interactive Continuum video portals, and visual scratchpads where coding agents simulate spatial designs alongside code. Their sharpest technical disagreement sits on supervision: General Intuition contended that explicit controller action labels transfer far better to real-time control than passive video-only latent inference because subtle inputs like aircraft rudder adjustments leave almost no trace in raw pixel frames.
Also called: everything will be simulated · pixels to actions · observing while acting · world interface models · Continuum portals
Predicting sequential frames at scale forces video models to internalize real-world geometry, gravity, occlusion, and object permanence directly from pixel data without hand-coded physics laws.
“You're learning about physics. You're learning about object permanence. You're learning about gravity, you're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world.”
Runway bets that compute scaling laws hold for next-frame video prediction just as they do for language models, yielding emergent visual realism that humans can no longer distinguish from real footage.
“I think we believe in scaling loss, not only applied for language models, but also for video, and this has been true.”
Conditioning real-time frame prediction on interventions at arbitrary steps turns offline video models into branching counterfactual world simulators capable of serving as reasoning brains for embodied robots.
“You take a frame, you predict next frame, then X, then X, then X, but then it can act at any point in time, in that prediction, which means I can simulate any possible divergence or action in that world.”
World interface models trained on UI clicks and user actions can manipulate objects and viewpoints inside a single 2D image in real time without any underlying 3D mesh or game engine.
“Now everything you just saw here has no physical 3D representation. There's no Unity. There's no Unreal. There's no Blender.”
Runway's Continuum research prototype treats a real-time world model as an operating system layer that turns structured web text into persistent, interactive video portals whose behavior adapts to clicks, queries, and window resizing.
“We call this a portal, and the portal is a real-time video. It's fully interactive and fully based on the content you're experiencing reading through.”
Connecting a coding agent to Continuum gives it an interactive visual scratchpad to simulate spatial designs alongside code, letting humans inspect and steer the agent's imagination directly rather than through text alone.
“So imagine you ask your coding agent to create a 3D model for your bike, and instead of the model trying to just think through that via code, the model can spin a version of Continuum to think through the problem visually.”
Because low-degree-of-freedom robots ship with standard game controllers that abstract away raw motor torque, gameplay controller action spaces map directly onto real-world robotic teleoperation.
“So what this means is that the action space for video games, often because of game controllers, and the action space for operations or the real world many cases are the same.”
Medal's gameplay recorder is scaling from one billion to three billion uploaded videos annually, generating over 1.5 trillion action-labeled inputs per month and matching frontier robotics training volume in half a day.
“Medal at the moment sees about a billion videos uploaded per year. This year, we're on track for three billion, which lead to over a trillion and a half actions taken per month.”
High-APM gameplay trajectories provide far denser simultaneous multi-key supervision than sparse human teleoperation recordings, bringing high-frequency control tasks like drone micro-adjustment within reach.
“You don't want to start with tell operations because the human moves take a lot less actions. It has a lot less information, so you want a model that can predict really high APM.”
In production architectures, real-time pixel-to-action models serve as a fast reactive control layer plugged beneath a frontier LLM orchestrator such as Gemini that handles long-horizon planning.
“we still expect you to use an orchestrator model, so you may use something like Gemini or Astro in order to do long-term planning and everything, and you'll be able to plug these into your workloads.”
Autoregressive, turn-based LLMs struggle with spatial-temporal control because real-time physical and virtual agents must continuously observe incoming sensory frames while simultaneously executing actions.
“Olaf can, unlike LLMs, which auto-regressively predict tokens, constantly, part of that behavior is that it's constantly observing as it is acting.”
Training directly on explicit controller action labels transfers better to action prediction than JEPA-style latent action inference from video alone because subtle inputs like aircraft rudder movements are visually unobservable in pixel frames.
“There's no way of being able to tell that from the video frames. It's very difficult, so our mode of having the action data really transfers better to action sequences than it does to world models and to generate the pixels.”
The closing keynote block bridged foundational pixels-to-actions world models, architectural trade-offs between building agents on owned versus rented infrastructure, and applied AI workflows for investment decision-making. — Closing Remarks(Alex Lieberman, ### 00:00:00 — Day One Talk Highlights)
Conference talks function primarily as shared intellectual surface area to spark high-bandwidth, in-person conversations among attendees rather than serving as standalone passive content. — Closing Remarks(Alex Lieberman, ### 00:00:18 — Speaker Applause & Final Housekeeping)
Thin talks (3): 2026-10-13-closing-remarks (173 words); 2026-10-14-trusting-agents-in-policy-driven-workflows (503 words); 2026-10-14-welcome-back-featured-keynotes (569 words). A thin talk cannot carry a theme alone.
Unassigned: 2026-10-13-closing-remarks — keynote-block welcomes and closes that recap or introduce other talks and argue nothing a second talk argues; their atoms are listed as singletons.
Coverage: 31 of the conference's scheduled sessions were recorded; themes count only these talks, so an argument made mostly in unrecorded sessions is invisible here.
ASR: transcripts are Qwen3-ASR output, cleaned but not corrected. Quotes keep ASR garble verbatim; known garbles are recorded in talk-insights coverage gaps, not fixed here.
Counting: themes count talks, not mentions. A talk may sit in several themes; a claim may support one theme and qualify another.
Claim ids: every id was re-derived on 2026-11-14 when all claims were re-extracted (176 verbatim atoms across 31 re-cleaned transcripts); ids cited by earlier versions of this report no longer resolve.
