Public benchmarks never measure a team's real task, so builders must engineer bespoke evaluation loops—synthesizing inputs from known answers, replaying historical outages, backtesting dated web snapshots, and grading trajectories with behavior rubrics—or redesign the agent until it is evaluable.
Public benchmarks never measure a team's real task, so builders must engineer bespoke evaluation loops—synthesizing inputs from known answers, replaying historical outages, backtesting dated web snapshots, and grading trajectories with behavior rubrics—or redesign the agent until it is evaluable.
Across enterprise RAG, legal review, creative writing, code review, accounting, and quantitative investing, speakers converged on the same wall: off-the-shelf benchmarks tell you nothing about whether your agent works, and domain experts cannot hand-grade thousands of runs. Air Canada inverted dataset creation by generating questions from known source pages and populating legal templates with clauses engineered to breach or pass rules, even re-architecting an agent purely to make it deterministically evaluable. Greptile benchmarks prompt changes against historical outage-causing PRs; Bridgewater hand-labeled 200 investment queries in a week to lift precision@10 from 60% to 95%; Exa Snapshot backtests predictive agents against 400 billion dated web pages without lookahead leakage; Towards AI runs five drafts per prompt under a two-stage cross-family rubric; and Basis grades ground-truth-free production runs with behavior specs and trajectory forking. The sharp counter-position centers on the grader itself: Sonar and Air Canada both showed that uncalibrated LLM-as-judge verdicts flip across identical runs and add noise rivaling the agent's own error rate.
“The original design we could not evaluate it. We switched to something that we can actually evaluate because of that.”Nathanael Weill · Write the Answer First: Generating Eval Datasets for Agents With No Benchmark · 00:17:13
In Inside the Fable War Room (00:15:44), Ford Prior argued that benchmarking against recent production code as golden data revealed that once rich domain context is synthesized, smaller Sonnet-class models generate legacy code as well as frontier models at much lower cost. In How to build an AI code review agent: from the creators of Greptile (00:05:41), Daksh Gupta argued that because catastrophic bugs occur in roughly one in a thousand pull requests and live review output cannot prove bug absence, prompt changes must be benchmarked against a catalog of historical outage-causing PRs.
In Write the Answer First: Generating Eval Datasets for Agents With No Benchmark (00:05:31), Nathanael Weill argued that generating synthetic questions from known source pages and checking retrieval in code accelerated iteration from one experiment per week to one per business day while eliminating brittle pipeline rules. In Write the Answer First: Generating Eval Datasets for Agents With No Benchmark (00:13:36), Nathanael Weill argued that populating existing legal contract templates with synthetic clause arrays engineered to either violate or satisfy known rules creates ground-truth breach labels by construction and runs evaluations in five minutes.
In Write the Answer First: Generating Eval Datasets for Agents With No Benchmark (00:01:29), Nathanael Weill argued that in enterprise Q&A over roughly 10,000 telecom vendor documents, retrieving the right source page produces the right answer 90% of the time, reducing evaluation primarily to deterministic page-retrieval verification.
“we ran benchmarks against like using our existing code recently written as the golden data and found out that Fable or Sonnet five wrote it just as good as, if not better than Fable.”Ford Prior · Inside the Fable War Room · 00:15:44
“the validation against really any reviewer can tell you about the presence of bugs, but unfortunately can't tell you about the absence of bugs.”Daksh Gupta · How to build an AI code review agent: from the creators of Greptile · 00:05:41
“So the impact of that is we went from one iteration week to business day, and we start to see the accuracy or the rating from the experts going up and up, and actually, removing a lot of complexity.”Nathanael Weill · Write the Answer First: Generating Eval Datasets for Agents With No Benchmark · 00:05:31
In A Hedge Fund as a Function (00:04:53), Frank Long argued that ingestion design choices—such as chunking, raw image versus OCR input, translation versus structured extraction, and cross-referencing internal notes—act as tunable hyperparameters over an evolving search space. In A Hedge Fund as a Function (00:09:20), Frank Long argued that because real steps combine multiple runners and tune model size, thinking effort, harness, skills, and hooks, workflows must be compartmentalized into evaluable units with a clear north-star metric.
In Vault: Making Unstructured Data Work for Investing (00:06:23), Aaron Linsky, Sam Green argued that hand-labeling roughly 200 representative investment queries in under a week established a baseline of 60% precision at ten and enabled systematic agentic search iterations that pushed precision to roughly 95%. In Search the Web as It Was: An Introduction to Exa Snapshot (00:07:50), Ryan Ahern argued that testing predictive models on live web search leaks post-event outcomes, whereas querying snapshots at fixed horizons before an event evaluates what signals were genuinely discoverable without lookahead bias.
In Search the Web as It Was: An Introduction to Exa Snapshot (00:11:18), Ryan Ahern argued that across traditional ML feature enrichment, hybrid models adding agentic adverse-media or equity scans, and end-to-end prediction agents, point-in-time snapshots let teams empirically prove historical lift over baseline processes. In Search the Web as It Was: An Introduction to Exa Snapshot (00:15:36), Ryan Ahern argued that without historical web snapshots and holdout sets, iterating on expressive semantic search features requires waiting months in live production to tell whether an agentic signal generalizes or overfits.
“There's actually a lot of knobs here, and I almost think about those knobs — it's like hyperparameters in a traditional machine learning system, right? I think about each of these knobs as something for me to tune and optimize against.”Frank Long · A Hedge Fund as a Function · 00:04:53
“So the way to think about breaking down these problems is, what can I actually compartmentalize in a way that we can understand and evaluate?”Frank Long · A Hedge Fund as a Function · 00:09:20
“The key metric we were looking at is precision at ten, which is of the top ten results, how many are the right documents, and we were hitting about sixty percent.”Aaron Linsky · Vault: Making Unstructured Data Work for Investing · 00:06:23
In Process Supervision for Agents That Do Long-Horizon Work (00:12:01), Mitchell Troyanovsky argued that distilling core workflow opinions into 'behavior specs' creates a ground-truth-free rubric that grades any production trajectory whenever a relevant trigger arises, scaling supervision signal far faster than static outcome evals. In Process Supervision for Agents That Do Long-Horizon Work (00:13:40), Mitchell Troyanovsky argued that when a multi-hour, thousand-call trajectory violates a behavior spec, builders can iterate on context fixes rapidly either by forking state right before the bad decision or by cheaply simulating the production prompt and skills in a coding subagent.
In Can Any Model Write Like Me? Building a Benchmark for Creative Writing (00:00:00), Louis-François Bouchard argued that aI writing slop persists as models scale not merely through overused vocabulary, repetition, and hallucination, but fundamentally because generated prose collapses into a generic, voiceless average rather than matching a specific author's tone. In Can Any Model Write Like Me? Building a Benchmark for Creative Writing (00:05:34), Louis-François Bouchard argued that generating five drafts per prompt across ten published and held-out scripts captures intra-model variance—which widens as model quality drops and masks lucky one-in-five outlier drafts—without making benchmark runs prohibitively expensive.
In Can Any Model Write Like Me? Building a Benchmark for Creative Writing (00:08:55), Louis-François Bouchard argued that structuring 14,000-word judge prompts as a two-stage rubric—classifying prose into one of six qualitative buckets before scoring across nine metrics—outperforms binary or direct percentage grading on subjective voice matching. In Can Any Model Write Like Me? Building a Benchmark for Creative Writing (00:08:55), Louis-François Bouchard argued that because LLM judges exhibit strong family-wide self-preference—especially OpenAI models favoring all OpenAI outputs—creative-writing evaluations require cross-family judges spot-checked against over a thousand human pairwise comparisons.
In Balancing Acceleration and Safety for AI Agents (00:11:58), John Feminella argued that because model upgrades act like changing runtime dependencies that can silently alter output quality, firms should provide shared self-serve evaluation infrastructure—analogous to unit testing rather than centralized sign-off queues—where teams supply only agent-specific assertions. In What does an AI engineer do? (00:09:45), Aparna Dhinakaran argued that because automated PR and production gates are only useful if trusted, AI engineers must curate golden datasets and meta-evaluate LLM judges until they align with human judgment before deploying them offline or online.
In The AI Bill Is Too Damn High – Tokenomics 101 (00:20:26), Eugene Cheah argued that right-sizing high-volume process workloads starts with building task-specific eval suites and searching open-weight models for the cheapest passing baseline before prompt-tuning or distilling down to 7B–27B domain models.
“And you can say, in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant?”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · 00:12:01
“when working with coding agents, ask the coding agent to make a subagent that pretends to be your production agent. And you can tell it to not cheat, and give it the system prompt and the skills, and you render everything perfectly.”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · 00:13:40
“Endlessly, the worst to me is that it usually has the wrong tone, or at least it has no tone and like everyone's tone at the same time, so it just doesn't really talk back to you.”Louis-François Bouchard · Can Any Model Write Like Me? Building a Benchmark for Creative Writing · 00:00:00
No operational pattern at AI Engineer NYC went unqualified. Across the corpus, 2 counter-claims pushed back on naive versions of this shift—highlighting where judges introduce variance, where caches break, or where human judgment stays irreducible.
“depending on the model and the situation, between seven and seventeen percent of verdicts change across what they call identical replications.”Tariq Shaukat · The Machine that Changed the World, vAI · 00:08:01
“Like we got so much noise compared to the noise with the with the agents that we're doing. Solving this problem is almost as hard as solving the whiteboard.”Nathanael Weill · Write the Answer First: Generating Eval Datasets for Agents With No Benchmark · 00:16:18
Can LLM-as-judge rubrics be trusted to grade production agent output?
Reducing enterprise AI inference spend at scale requires splitting API keys by department and workload, matching model size and per-request tokens-per-second to human latency sensitivity, sizing dedicated GPU capacity to base load with cloud spillover, and measuring real business outcomes rather than raw token volume.
An investment firm can be engineered as a composite state-machine function from API-accessible external and internal context to `(stock, size, timing)` decisions, decomposed into compartmentalized sub-steps whose execution runners and pipeline configurations are tuned like hyperparameters against evaluable north-star metrics.
Rather than locking agents in useless air-gapped sandboxes or relying on fragile prompt policies and rubber-stamp human approvals, Two Sigma enables autonomous enterprise agents safely through five composable runtime controls: automatic framework registration, risk-tiered review, registry-enforced tool entitlements, shared evaluation infrastructure, and classifier-routed approvals.
Because AI writing slop persists as models scale, Towards AI built ToneBench—evaluating 120 models across ten long-prompt tasks, five drafts per prompt, two-stage categorical rubrics, and cross-family LLM judges—showing that multi-draft cheap generation merged by a reasoning model beats expensive one-shot prose.
Building a production-grade AI code review agent requires moving beyond naive diff prompting to combine historical incident eval datasets, cross-family author-reviewer model inversion, a living codebase wiki instead of raw grep, dependency-rich sandbox UI testing, and per-subtask model routing.
Because coding agents accelerated inner-loop implementation while leaving enterprise legacy modernization bottlenecked on cross-functional alignment and domain context, GEICO built a multiplayer AI-native war room steered by Markdown FleetKit protocols, human-in-the-loop worksheets, and deterministic nine-step gates to ship legacy claims APIs in a single day.
Because AI agents execute dense, multi-step exploratory web queries whose token costs compound across trajectories, Exa pairs 95%-token-saving dynamic snippet highlights with Exa Snapshot's 400-billion-page dated web archive so financial and predictive systems can backtest web-derived signals without lookahead leakage or unvalidated overfitting.
Bridgewater's Vault demonstrates that enterprise investment AI only compounds when internal and external unstructured data is enriched against a bitemporal firm ontology, served through native-filtered search built for both human precision and agent-storm scale, and embedded into existing researcher habits with automatic synthesis write-back.
The AI engineer's role has shifted from manually building and reviewing individual features to architecting closed-loop agent factories — where agent-first telemetry triggers isolated, durable sandbox runs, LLM judges are meta-evaluated against human judgment, and both production and coding agents are traced end-to-end.
When custom enterprise agents lack public benchmarks and continuous domain-expert access, teams should invert evaluation by generating synthetic inputs around known answers—source pages, business rules, or clause templates—and re-engineer agent architectures around deterministic checks rather than noisy LLM judges.
Enterprise agents achieve production adoption not through open-ended autonomy or outcome-only evals, but by encoding opinionated workflows as behavior specs that grade ground-truth-free trajectories and enable targeted debugging via trajectory forking and subagent simulation.
Because abundant AI-generated code overwhelms human reviewers, compounds multi-step errors, and defeats inconsistent LLM judges, enterprise software factories must engineer shipping confidence multiplicatively through clear intent, in-loop guide rails, hybrid deterministic-generative verification, and auditable evidence.