Software engineering is shifting from writing code locally to managing asynchronous cloud agent fleets and closing the trace-to-skill loop—centralizing transcripts and tool calls, grading runs post-merge, and letting meta-agents distill recurring failures and human corrections into versioned skills.
Software engineering is shifting from writing code locally to managing asynchronous cloud agent fleets and closing the trace-to-skill loop—centralizing transcripts and tool calls, grading runs post-merge, and letting meta-agents distill recurring failures and human corrections into versioned skills.
Across Warp, Arize, Linear, Untangle, GEICO, Anthropic, and Runpod, the unit of engineering work moved from crafting individual features to operating and improving the agent factory itself. Warp demonstrated 'factory engineering': shared cloud Docker sandboxes automatically centralize conversation transcripts, MCP/tool calls, and diffs, exposing silent agent workarounds (like brute-force port probing) to post-merge markdown judge rubrics and daily meta-agents (`Skill Doctor`) that open measurable skill PRs. Linear runs a daily post-merge distillation job that turns PR review corrections and reverts into `agents.md` updates; Arize designs agent-first telemetry to wake scheduled sandbox sweeps while tracing both production agents and the coding harnesses behind them; Anthropic treats SME-authored plain-text skills as living, versioned code co-iterated by business teams and engineers; GEICO coordinates legacy transformations through Markdown FleetKit protocols; Runpod exposes GPU benchmark controls as FastAPI endpoints for agents to drive directly; and Untangle replaces local development with communicating builder, reviewer, and merge-manager sessions—extending the paradigm to hiring by stack-ranking candidates on 45-minute recorded agent-operating trials, echoing the Day 2 keynote's call to pair human company curation with career-research agents. In tension with universal 'agent management,' Basis warned that pushing prompt and skill authorship onto non-technical enterprise users creates a severe competency gap, and Day 2's keynote cautioned against letting agents automate personal outreach like resume submissions.
“Takes a goal, breaks it down, dispatches leaders to do all the research and decide about the approach, puts everything into this like human loop cube.”Ford Prior · Inside the Fable War Room · 00:08:10
In Your agent needs a skill to improve its skills (00:01:44), Ben Holmes argued that factory engineering treats the agent harness itself as an optimization target by using LLM-as-judge scoring agents to evaluate past runs against custom rubrics for efficiency, code quality, and test coverage. In Your agent needs a skill to improve its skills (00:03:12), Ben Holmes argued that introspecting team-wide agent performance requires centralizing three artifacts per session—dialogue transcripts, external tool/MCP/filesystem calls, and resulting code diffs—which shared cloud Docker sandboxes capture automatically without syncing local laptop folders.
In Your agent needs a skill to improve its skills (00:06:06), Ben Holmes argued that because unattended agents silently brute-force around broken environment instructions—such as probing local port numbers when test skills are stale—scoring tool-call efficiency over full traces uncovers harness bottlenecks hidden by passing PRs. In Your agent needs a skill to improve its skills (00:08:55), Ben Holmes argued that triggering markdown-defined judge rubrics upon PR merge produces structured pass/fail scores and failure rationales that a daily cron-scheduled self-improvement agent mines to propose targeted skill and codebase updates.
In Your agent needs a skill to improve its skills (00:13:43), Ben Holmes argued that a single-player batch audit skill (`Skill Doctor`) crawls local Claude, Codex, and Pi logs to produce a rubric report card and concrete skill edits, such as upfront port-discovery instructions and prescriptive changelog formatting. In Your agent needs a skill to improve its skills (00:16:10), Ben Holmes argued that unlike opaque overnight memory or dreaming loops, continuous rubric dashboards let teams attribute measurable score inflections directly to merged skill and config fixes such as enabling missing subagent permissions.
In Your Coding Agent Has a Context Problem (00:18:47), Tom Moor argued that a daily post-merge automation mines review corrections and reverts across merged PRs and distills actionable lessons into `agents.md` and documentation so human product judgment compounds across subsequent agent runs. In Anthropic: Accelerating frontier AI in Financial Services. (00:07:57), Belinda Neal, Keon Shahab argued that domain-specific workflows such as trade-break investigations should be captured from subject-matter experts as plain-text skills that are version-controlled, frequently iterated, and governed like living production code.
In Anthropic: Accelerating frontier AI in Financial Services. (00:12:25), Belinda Neal, Keon Shahab argued that agent deployment reshapes the financial-services operating model into a tight loop where business SMEs directly correct live agents and propose changes that engineering validates against baseline evaluation suites before shipping as updated skills.
“You can set up some scoring agents to score the quality of how those conversations went, so think of this like an LLM as a judge reviewing against rubrics that you define across metrics like efficiency, or code quality, or test coverage.”Ben Holmes · Your agent needs a skill to improve its skills · 00:01:44
“you got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output, which is something that you may want to grade for quality purposes.”Ben Holmes · Your agent needs a skill to improve its skills · 00:03:12
“these AIs are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it.”Ben Holmes · Your agent needs a skill to improve its skills · 00:06:06
In What does an AI engineer do? (00:00:00), Aparna Dhinakaran argued that out of roughly six billion people who have touched LLMs and 20 million professionals accelerating multi-week tasks into days, only about 5,000 frontier practitioners currently operate swarms of thousands of concurrent always-on agents. In What does an AI engineer do? (00:01:43), Aparna Dhinakaran argued that aI engineering shifts from manually writing code, reviewing diffs, and checking dashboards to constructing automated build-eval-ship-monitor-improve systems where agents review non-deterministic execution paths and analyze telemetry.
In What does an AI engineer do? (00:04:10), Aparna Dhinakaran argued that designing telemetry for agents as primary consumers turns scheduled 3-to-6-hour trace sweeps, metric thresholds, and Slack user complaints into automated triggers that wake isolated sandbox runs behind a single traced proxy. In What does an AI engineer do? (00:12:10), Aparna Dhinakaran argued that end-to-end observability must cover both production agents and the coding harnesses that build them (Claude Code, Cursor, Codex), using online evaluations over live traces to feed newly detected failures back into the improvement loop.
In From Model to Metrics: Serving and Benchmarking Any LLM Live with vLLM and Runpod (00:06:31), Chris Vela argued that exposing GPU telemetry, error scans, and vLLM configuration reloads over backed-up FastAPI endpoints enables coding agents to run serving benchmarks and optimize flags programmatically without human UI intervention.
“But really, those at the frontier labs are just super small percentages, like five thousand users are the ones who are running thousands and thousands of agents at once.”Aparna Dhinakaran · What does an AI engineer do? · 00:00:00
“As an AI engineer, your job is no longer to just write the feature, write a bad feature. It's about how do you build the loop to write that feature.”Aparna Dhinakaran · What does an AI engineer do? · 00:01:43
“how do you set up your telemetry data so that agents are the primary consumers of that data? And what that does is that observability data can become the trigger for actually running and triggering these always-on agents”Aparna Dhinakaran · What does an AI engineer do? · 00:04:10
In Welcome Back - Featured Keynotes (00:02:12), Ksenia Se argued that a high-signal job-search pipeline combines human curation—first ranking conference speakers' companies by technical interest—with an agent that scrapes career pages to match open roles and extract published compensation ranges for updating rankings. In Welcome Back - Featured Keynotes (00:02:12), Ksenia Se argued that hiring serves as a primary two-sided incentive for attending AI Engineer, extending even to conference speakers who are simultaneously recruiting talent and exploring new roles themselves.
In Your job is now "agent manager" (00:03:38), Ryan Carson argued that a dedicated merge-manager agent session coordinates the wave of concurrent pull requests to prevent shared-state collisions such as conflicting database migration numbers. In Your job is now "agent manager" (00:03:38), Ryan Carson argued that engineers who still code in local development environments bottleneck their teams because local workflows cannot orchestrate fleets of parallel, inter-communicating cloud agent sessions.
In Your job is now "agent manager" (00:06:23), Ryan Carson argued that untangle eliminates resumes, pre-screens, and video interviews to avoid interpersonal bias, evaluating candidates instead by having models stack-rank unguided 45-minute screen recordings of them operating agents to ship a PR. In Your job is now "agent manager" (00:08:39), Ryan Carson argued that the top five stack-ranked candidates complete paid 16-hour recorded trials shipping a real PR under NDA, with AI and human video review selecting finalists before any phone conversation happens.
“The next step is to send your agent to those companies' careers pages, right, and find them open positions to match them to your experience, and collect the published compensation ranges.”Ksenia Se · Welcome Back - Featured Keynotes · 00:02:12
“I spend the day talking to other speakers and participants, and what I've noticed is that one of the strongest incentives to come to this conference is hiring, to the point that even some speakers are hiring and also looking for work.”Ksenia Se · Welcome Back - Featured Keynotes · 00:02:12
“Like if all the agents are trying to grab these things, you constantly collide, right? So the merge manager just helps all that go away, but also is a little more intelligent about how the merge deals happen.”Ryan Carson · Your job is now "agent manager" · 00:03:38
No operational pattern at AI Engineer NYC went unqualified. Across the corpus, 2 counter-claims pushed back on naive versions of this shift—highlighting where judges introduce variance, where caches break, or where human judgment stays irreducible.
“And please don't make your agents send out your resume automatically.”Ksenia Se · Welcome Back - Featured Keynotes · 00:02:12
“The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user.”Mitchell Troyanovsky · Process Supervision for Agents That Do Long-Horizon Work · 00:03:42
Can LLM-as-judge rubrics be trusted to grade production agent output?
Because frontier model capability, cost efficiency, and prompt-injection safety now outpace enterprise diffusion, financial institutions must shift from bespoke one-off agent builds to a reusable five-layer architecture combining MCP data connectivity, SME-authored versioned skills, least-privilege identity, human-approved sandboxed execution, and continuous regression evals.
Rather than deploying open-weight LLMs with default vLLM recipes and oversized context windows, Runpod's BenchPod wraps Dockerized vLLM builds with live YAML reconfiguration, self-healing watchdogs, and FastAPI benchmark endpoints so teams and agents can empirically tune throughput and latency before scaling out.
Because coding agents accelerated inner-loop implementation while leaving enterprise legacy modernization bottlenecked on cross-functional alignment and domain context, GEICO built a multiplayer AI-native war room steered by Markdown FleetKit protocols, human-in-the-loop worksheets, and deterministic nine-step gates to ship legacy claims APIs in a single day.
The AI engineer's role has shifted from manually building and reviewing individual features to architecting closed-loop agent factories — where agent-first telemetry triggers isolated, durable sandbox runs, LLM judges are meta-evaluated against human judgment, and both production and coding agents are traced end-to-end.
Because two-sided hiring is one of the strongest incentives driving conference attendance, attendees should pair human judgment—ranking speakers' companies by technical interest—with a research agent that matches open roles and published compensation ranges to inform live conversations rather than auto-submitting resumes.
Agents can systematically improve their own skills and harness configuration when teams centralize full session traces—transcripts, tool calls, and diffs—grade them post-merge with markdown LLM-as-judge rubrics, and run scheduled meta-agents that turn recurring failures into measurable skill PRs.
As inference costs plummet and coding agents run unattended in the outer loop, engineering's primary failure mode shifts from implementation capacity to product incoherence, requiring automated pipelines that thread customer intent, product history, and ownership through triage, implementation, review, and daily feedback distillation.
Software engineers should abandon local coding to manage communicating fleets of cloud builder, reviewer, child-QA, manager, and merge-manager agents—keeping humans in the loop via mandatory feature-demo videos and hiring strictly through recorded agent-operating trials.
Enterprise agents achieve production adoption not through open-ended autonomy or outcome-only evals, but by encoding opinionated workflows as behavior specs that grade ground-truth-free trajectories and enable targeted debugging via trajectory forking and subagent simulation.