Schedule

When money is involved, the numbers just HAVE to be right. For an agent to be truly useful, it needs a harness designed to earn our trust. In this workshop, we’ll share the design principles we use to build trustworthy agents for data and financial analysis. Drawing on our experience building agents at work and using them to support financial decisions at home, we’ll walk through a harness built for this workshop and use a small example to put the principles into practice. Bring a laptop and your LLM of choice. You’ll leave with a working harness you can extend and a blueprint for applying the same principles to build your own harness, wherever trust matters most.

Build a wealth management concierge in ChatGPT Work using financial data plugins, client information, and custom skills. We’ll create and customize the workflow step by step, then test it on client questions and advisor briefs.

Dynamic workflows in Claude Managed Agents, and the controls that keep them in bounds.

You shipped an AI agent. It works great on your test queries. But how do you know it works on your users' queries? How do you know a prompt change that fixes one thing didn't break three others? And when a new model drops next quarter, how do you know whether to upgrade? If your answer to any of these is "I run it a few times and check," this workshop is for you. In this hands-on session, you'll build a complete evaluation pipeline for an AI agent from scratch. Starting with a financial analysis chatbot built on the Claude Agent SDK, you'll instrument it with tracing, run it against diverse test cases, and then do the thing most tutorials skip: actually read your data. You'll categorize failures by root cause, define success criteria, and use that analysis to decide what to measure before writing a single eval. Then you'll build three types of evaluators: fast deterministic code checks, built-in LLM-as-a-judge evals like faithfulness, and a custom rubric you design yourself. You'll validate your judges against your own human labels, because an eval you haven't tested is just a fancy way of being wrong at scale. The workshop doesn't stop at scoring. You'll save failure cases as reusable datasets, improve your agent's prompts based on what the evals told you, and run controlled experiments to prove the improvement with numbers instead of gut feel. Along the way, you'll learn practical frameworks: the impact hierarchy for prioritizing fixes, the Swiss Cheese model for layering defenses, and the eval-iterate cycle that turns a prototype into a production system. You'll leave with a working notebook, a free Phoenix Cloud account with your traces and eval results, and a repeatable process you can apply to any AI application the day you get home.

Getting out-of-control AI bills in your org? With limited ROI? As your organization scales its AI usage to billions and trillions of tokens a month. We discuss the trade-offs we've seen at scale in production with our clients to optimize the AI "cost per task". From AI model routing and caching to fine-tuning industry-specific models (eg., AT&T OTEL line of models). We discuss the common roadmap and trade-offs enterprises can use to scale AI resources.

Building an AI demo is easy. Knowing whether it actually works — and keeping it working in production — is the hard part. Most teams ship agents on vibes: they try a few prompts, the output looks good, and they push to production with no real way to measure quality or catch regressions. This hands-on workshop walks through the full lifecycle of shipping a real AI agent, using a working financial-analyst agent built on the Claude Agent SDK as the running example. You'll instrument it with tracing, do structured error analysis on its actual outputs, and build a layered evaluation suite — from cheap deterministic code checks to LLM-as-a-judge evaluators with custom rubrics. We'll cover the parts most tutorials skip: why agents fail in ways single LLM calls don't, the eval anti-patterns that quietly mislead you, and how to know whether you can even trust your judge (meta-evaluation). Finally, we'll close the loop: turning eval results into datasets and experiments, running evals online against production traffic, wiring them to monitors and alerts, and feeding failure explanations back to a coding agent to actually fix the underlying problems. You'll leave with a runnable notebook and a repeatable, evaluation-driven workflow you can apply to your own agents the next day.

Agent = Model + Harness. The model emits tokens; state, memory, retrieval, tool dispatch, identity, retries, and budgets. All these concepts are harness code — and that is where agent quality is won or lost. In this hands-on workshop, you build that harness from primitives against Oracle AI Database and a realistic AML dataset: Meridian Bank, 25 branches, ~1,200 transactions over 90 days, and transactions flagged by Anti Money Laundering (AML) rules. We will go through long-term memory and schema scanning with OAMP, semantic and hybrid RRF retrieval with ONNX in-database embeddings, vector-indexed toolbox and skillboxes, and a bounded agent_turn loop. After this workshop, you will know everything required to build your own agent harness from scratch, staying faithful to best agent harness and AI practices. You leave with a readable ~100-line loop you can operate, and a mental model that you can improve for your financial AI agents.

Harbor is an open-source framework for evaluating agents on realistic tasks with real data in isolated, sandboxed environments. This talk gives a conceptual overview of how Harbor works: tasks defined as natural-language instructions, agents run in fresh Docker environments, verifiers that grade what the agent leaves behind, and rewards that close the loop so you can change a prompt, tool, skill, or whole architecture and get immediate feedback. We’ll see how Harbor integrates with Phoenix, an open-source AI observability, experimentation, and evaluation platform, and discuss concrete examples of how the Phoenix team is using Harbor to iterate more quickly and with greater confidence. You'll leave with a working mental model of Harbor and a sense of how to apply it to your own agents.

Across 700,000 followers and 200M+ views across Twitter, Youtube, TikTok, Instagram and more, we built a six-stage system—ideate, script, record, edit, repurpose, publish—that automates logistical work while keeping creative judgment human. We’ll unpack the harness behind it: agent skills learned from past work and performance data, local taste guides, controlled generation, approval gates, and a feedback loop that improves after every post. We’ll show the custom tools that turned recurring friction into software, and the places we deliberately refused to automate. You’ll leave with an architecture for AI-assisted creative work that scales output without flattening voice.

Gitar is Sonar’s AI code verification agent, complementing SonarQube with an understanding of each change’s context and intent. In this hands-on workshop, you’ll install Gitar as a GitHub App and use it across three pull request scenarios that show how verification goes beyond code review. Gitar verifies every change against your project's context and rules, catches problems, generates working fixes, and iterates until your CI pipeline passes. Code review is one part of what it does. You'll start with Gitar's out-of-the-box verification, then move into project-specific configuration that teaches Gitar your team's conventions, and finish with CI failure diagnosis and automated fixes. Everything happens in the browser, so you won't need to set up any local tools or environments, just GitHub and Gitar. You'll leave with a working Gitar install, three verified PRs, and a practical understanding of verification instructions, repository rules, and CI failure analysis.

Delegating work takes more than a capable model and a good prompt. It takes a system that can check results, catch unsupported claims, and decide when to retry or ask for help. In this hands-on, 60-minute workshop, you’ll build a verification and repair loop around an AI agent. We’ll use Playwright to check browser behavior and Jev to evaluate whether collected evidence supports specific acceptance criteria. Your code will combine those signals to accept work, request a bounded repair, or escalate to a human. Then we’ll try to fool it. Using deliberately broken outputs and misleading evidence, you’ll test not just the agent, but the verifier itself, and explore the trade-off between catching failures and interrupting people unnecessarily. Drawing on real projects involving delegated work and agent supervision, this workshop focuses on patterns you can apply with your own stack. You’ll leave with a working verification loop and a practical method for deciding what to automate, what to check, and where human judgment still belongs.

An agent that improves itself is a loop. Most teams have the traces, maybe some evals, and none of the rest. This workshop builds the whole loop. We start from traces. Arize's Signal agent reviews them and files what it finds the way a reviewer would. From there a failure becomes a dataset, the finding becomes an eval, and that eval becomes the regression test guarding the fix. The pieces are tracing, Signal, datasets, evaluators and experiments. The workshop is about how they connect. What you'll learn: • How to turn a production failures into a permanent tests • How to write an eval from a problem the system found instead of guessing at criteria up front • How to validate fixes without regressing something else • Where to add humans to the loop

In this workshop, we will demonstrate how to use OpenProse, an open-source agent framework, to build complex, event-driven multi-agent workflows. We will walk through an anonymized financial regulatory workflow developed for one of the largest banks in the US, showing how OpenProse can encode institutional knowledge in-house and run agent workflows reliably and repeatably, with auditing and guardrails built in.

See how Exa helps leading labs and firms leverage web search to develop new predictive models. Gathering the right data to train a predictive model at scale is hard. You need to give the model enough data to make predictions without accidentally giving it access to up-to-date information about the events it’s trying to predict. On top of that, if your model has any access to search, it’s hard to prevent or detect cheating. Ideally, the agent should simply be able to search the web as it was on any given date in the past. Now it can. Exa Snapshot is a research preview built on more than 400 billion webpage snapshots from the last two decades. In this session, Ryan Ahern (Forward Deployed Engineer at Exa) will show you how Exa Snapshot can be used to backtest prediction agents against yesterday’s web. You’ll build your own prompt to predict the future, with prizes for the most prescient prognosticators.

Building an AI agent demo is easy. Getting it into production is hard: you need to know what it actually does, catch it when it goes wrong, and keep improving it. In regulated industries you also have to prove all of that. In this hands-on workshop, Annabell Schäfer and Doneyli De Jesus from Langfuse show how to close that loop, using a banking support agent, Northwind Bank, as the example. You'll see how to: • Trace every agent interaction end to end • Use guardrails to catch and block risky requests, and record each block • Evaluate answers for quality and compliance using datasets as part of your CI/CD • Spot your agent's top failure modes on a dashboard and estimate the business impact The session suits both AI leaders and engineers. Bring a laptop if you want to follow along: a seed script loads the demo into your own Langfuse project.

What insights can we find in billions of pages of Insurance documents? Traditional Insurance models have been based on well structured data for underwriting, claims, fraud and risk. With more agentic models and the ability to utilize unstructured data, the ability to build better predictive models is improving across a number of insurance domains. This talk will focus on three use cases and how AI and Data transformation go hand in hand.

In this workshop, we'll share several simple ways to get the most out of fronter models, including harness design, prompting, and settin g up evaluations for successful hillclimbing.

In this hands-on workshop with Keycard and Temporal, you’ll learn to give AI agents their own identity, control how they act on a user’s behalf, and recover interrupted workflows. Working with an expense agent, you’ll practice enforcing permissions beyond the prompt and acquiring task-scoped, runtime credentials during execution without storing them in workflow history. You’ll leave with patterns for building secure, durable agents that act independently or on a user’s behalf.

Model intelligence and persistence have reached a level where we believe applied AI in the enterprise is now about context and connectors. Out of the box frontier intelligence in the investment domain is fundamentally limited by access to information. Bridgewater's investment philosophy is fundamental and systematic. We work to uncover truths about how economies operate, building up that understanding on one of the world's most comprehensive bitemporal structured data stores. But our investors don't work from structured data alone. They consume research, news, transcripts, and internal writing every day. An agent tasked with investment problems needs access to the same information our best human investors have, so we built the Vault: a centralized, high-quality repository of our best external and internal unstructured data, serving humans and agents alike. We'll walk through why we chose to solve this problem ourselves, tracing the Vault from our first instinct, which was to buy it, through expanding it across all of our third-party research and into our internal knowledge, building it up from a point solution into a platform. The Vault already powers PAT, our first generation of analyst agents, used daily by hundreds of Bridgewater investors, and is now enabling our second-generation Researcher.

Frontier AI is moving faster than financial services is adopting it, and that gap is the opportunity. In this keynote, Belinda Neal and Keon Shahab of Anthropic set out the two frontiers every firm now has to reach: the frontier of intelligence, what the best model can do today, and the frontier of adoption, how much of the firm's real work runs on it. They show how to go from a handful of agents to many by building connectors, skills and controls once, and what it looks like when a whole industry does the same, with Claude for Financial Advisors as a worked example. You will leave with three questions to take back to your own firm.

As AI agents grow in utility and power, they also grow in potential risk. For some, this creates a balancing act: too little safety means highly regulated firms are exposed to undue risk; too little innovation means agents cost more to run than they earn. How do we balance the desire for safe agentic AI usage with the goal of using agents to carry out innovative research and engineering? In this talk, I’ll cover some of the key technical approaches we use and have found highly effective at Two Sigma, and along the way hopefully convince you that there doesn’t need to be a tradeoff between these two properties.

There's so much debate online about "should we read the code?" Some people are yelling, "Everybody should build everything with a dark factory!" Others cry out, "The slop will destroy us all!" What this whole conversation is missing is principles for understanding how we would even make these decisions. Come hear the author of Code Simplicity say what he thinks is true now with AI coding, and see if there's some way to reliably find a path through all these opposing viewpoints.

Coding agents have effectively eliminated the need for manually writing code, and the current frontier is how to eliminate code review. But where are we going if we look past that? What happens when agents master devops, architecture, scaling, and security? What is an engineer if agents can do all the building? In this look into the next few years, Aparna Dhinakaran argues that the core skill that remains is the definition of “good”: knowing what your users want, and what it looks like when it’s “done”, are the two things agents don’t know and aren’t going to get to. At the same time, more and more AI is going to be involved in software, so software is going to get steadily less deterministic. That presents a dual challenge: you have to get good at defining success, while simultaneously success gets harder to measure. How to tackle those challenges is the subject of this talk.

Building production-grade foundation models in finance requires more than scaling architectures. It demands rethinking how systems are designed, trained, and validated in a regulated, high-stakes environment. In this talk, we share lessons from developing large-scale transaction foundation models trained on billions of banking transactions. We focus on the full build lifecycle: representing heterogenous transaction data, designing tokenization, and sequence strategies, curating high-quality datasets at scale, and constructing robust training and evaluation pipelines. We highlight key decisions and trade-offs required to move from experimentation to production. Finally, we show how adaptive fine-tuning and multi-task learning enable these models to support critical applications such as fraud detection and transaction decisioning.

Getting an enterprise AI agent through a Tier-1 bank’s security review and into an air-gapped VPC typically takes 9 to 12 months, period. Standard architectures built for SaaS envs immediately break when forced into strict data sovereignty, non-deterministic risk environments, and zero-leakage requirements. I lead the team that's done this inside 5 different Fortune 100 banks and discovered which problems are universal and which were just one bank's weird legacy system. A few things I've seen from this work: 1) An agent that writes and runs its own exploit test to prove a CVE is actually closed. 2) An LLM judge that leaks the exact PII it was supposed to protect via its own explanation (checked 11 observability platforms and only one attempts a fix). 3) And why the database most AI observability platforms run on has no managed data resilience story at all. I'll share the architectural and leadership blueprint behind all of this: 1. Accelerating Compliance Velocity: How to automate Docker package scanning and CVE remediation pipelines so security reviews don't become a burden for eng teams 2. Zero-Leakage Architecture: Design patterns for running agent quality and governance loops without exposing PII or violating data sovereignty rules 3. High-Availability SLAs & DR: Building zero-data-loss, multi-region failover setups that pass rigorous audits

Private equity firms hold controlling stakes in companies across nearly every sector - cargo airlines, education publishers, insurance brokers, industrial manufacturers. That makes PE one of the largest and least discussed deployment surfaces for AI: one operating team, dozens of companies, and a hard requirement that results show up in the P&L before the company is sold. This talk covers how that actually works. Drawing on portfolio operations work at a large PE firm, I'll walk through: Diligence: how AI risk and opportunity get assessed before a deal closes, and how AI assumptions now enter the underwriting model itself Deployment: what generalizes across a portfolio and what doesn't, with case studies in air cargo network decisioning and AI-native education products Build vs. partner: which capabilities a firm builds centrally as reusable IP, and where it stays model-agnostic across vendors Measurement: how AI work gets tied to operating margin, and what happens to projects that can't show it Portfolio companies rarely look like a research lab: thin engineering teams, fragmented data, and a three-to-five-year clock. This is a report on shipping AI under those constraints.

Every firm can buy the same frontier models. What separates one firm's agents from another's is what they know about the business, what they remember, and whether they improve. In financial services that is also where entitlements, audit and model risk sit, which is why so many agents stall between pilot and production. Take one request: an analyst asks an agent for the firm's exposure to a counterparty. The model can write the query. What it cannot know is that the counterparty is a dozen legal entities, that two desks calculate exposure differently, that this analyst is cleared to see only one of them, and that the agent was corrected on this exact request last week. A better model fixes none of that. Fixing it is the job of what we call the intelligence plane: a slice through the agent stack covering application, data, ontology and semantics, memory, and the governance that wraps them. I will follow that request through the plane one layer at a time, closing each gap as we go, then show how to build your own on Oracle. If that analyst asked your agent tomorrow, how many of those four things would it know?

Four months ago, out of frustration with slop, I set out to answer one question: which model writes a YouTube script that sounds like me. Not best in general, not best at writing. Which one passes as my voice, on my channel, by my standards, and isn't cringe to read out loud. So I vibe coded my own benchmark to answer it. Benchmarking 148 models took about two weeks, two 20x subscriptions on Claude and Codex, and $80 in API credits. That was never the bottleneck. The four months went into learning to build a proper benchmark. Deciding what a score means when the thing you are measuring is taste. Writing rubrics that quantify my subjectivity. Working out whether Elo was even the right tool. Finding out the same drafts average 74 out of 100 with one judge and 88 with another. Writing 10 scripts with two human editors and withholding half, so I could tell whether a model is contaminated. Most of the benchmark uses automated judging, but human judging is crucial for analyzing taste. So I added human ratings, sat down, and rated the first few hundred pairwise comparisons myself. My evaluation was broken, and only doing the work by hand revealed it. I fixed it, rated again, and I am releasing the benchmark publicly to get help rating these models. I will show the architecture, the numbers, what it cost, and the failure modes I would look for first in yours.

What is an AI-native company? At Arca, it means a company where you can steer the business by steering the agents. This talk gives you a look inside the agent runtime for Arca's long-horizon work: a single goal spanning weeks, crossing multiple human hands, and internal/external systems. We’ll look at our agent loop, our signals engine, tools for long-horizon work, and how we orchestrate human/AI collaboration to drive business outcomes.

What if LLMs could turn a company’s public data into an executable model of how the business works, instead of just another summary? We’re building an agent that takes 10-Ks, 10-Qs, earnings calls, and other public data and reconstructs a company like NVIDIA into a model of its key business drivers. You can run scenarios like “What happens if hyperscaler AI capex drops 20%?” and see the impact across the business. Then we time-travel the agent back to previous quarters, give it only the information available at the time, and test how well it predicts what happens next through AutoResearch. We’ll demo the system and share what we learned about turning unstructured data into an interactive business model you can actually run, test, and evaluate.

Join Amol (CEO of Nori Agentic), Cliff (CTO of Nori Agentic), and their cloud agent Jiro as they discuss how cloud agents work, why sec masters are important, and how they worked alongside a Fortune 200 asset management firm to automate their sec master maintenance.

How do you migrate hundreds of thousands of daily financial transactions off a decades-old mainframe and onto a cloud-native platform — without losing a single dollar, or a single night's sleep? At Vanguard, our Cash Program team took on exactly that challenge: rebuilding the ledger system underpinning the firm's cash offerings as a resilient, active-active microservices ecosystem, migrated while the legacy system stayed live in production. The key to pulling this off was running three tracks in parallel: producer migration, gold copy replication, and consumer migration. This let producers and consumers move independently, each on their own timeline, while data stayed consistent across both systems throughout. We'll walk through each track in turn. On the producer side, we'll cover how we onboarded dozens of data sources — APIs, direct service calls, and raw table inserts — onto the new platform. For gold copy, we'll dig into how CDC-based replication jobs kept legacy and cloud data in sync, and the tradeoffs and near-misses that came with running two systems at once. On the consumer side, we'll explain how we used GraphQL to decouple cash data from security transfer data, and why that mattered for downstream teams. Along the way, we'll touch on the architecture that made this possible — including why we built idempotency directly into the database layer instead of as a bolt-on service — and how AI tools helped us maintain two codebases through a multi-year migration. You'll leave with practical takeaways for modernizing high-stakes financial infrastructure without missing a transaction.

Not long ago, nobody knew what a good evaluation strategy for a RAG system looked like. Each team invented its own, and it took months. Then RAGAS arrived. Faithfulness, context relevance, and answer relevance became a playbook you could look up, and the same evaluation now takes a week. What's the equivalent for a custom agent, like one that reviews contracts, triages invoices, or runs your internal workflows? Nothing. There's no benchmark, no labels, and no playbook, so every team is back to inventing its own and losing the same months. This talk shares the techniques I used to generate eval datasets for an agent no benchmark covers, along with how to tell when a generated dataset will help you and when it will quietly mislead you. It won't be RAGAS for agents, but it will be enough to get started on Monday.

“Is this tool call allowed?” It’s never that simple. Allowing or denying individual API calls or MCP tools isn’t enough. You might give an agent access to read-only Notion MCP tools, for example, but still have no way to limit it to a specific database on a specific page. Intent-based auth adds the missing context: what is this agent trying to accomplish, and does this action serve that mission? We'll explore how AAuth, a new protocol designed for agents, allows you to track an agent's mission, how intent is enforced, and why intent based auth is key to unlocking agentic commerce.

In finance, the data that moves a decision rarely lives in a feed or a database. It lives in the investor's head: the position they carry, the risk they're worried about, the mandate they answer to. That's as true for an asset manager or a hedge fund PM as it is for a trader. LLMs are only as good as the context you give them, and that context is tacit, different for every person, and changes by the hour, so a generic tool hands everyone the same mediocre answer on stale data. This talk reframes the real problem in enterprise AI for finance as context engineering: not a bigger model, but capturing each user's evolving mental model and spending every token on what only they care about, without their data ever leaving the firm. I'll walk through how we think about per-user context, memory, and grounding for agents on live markets, and the security posture that lets any of this clear a bank. And I'll leave you with the open question I think decides who wins enterprise AI in finance: if the real edge lives in people's heads, how much of it can a machine actually capture?

Quant teams rely on point-in-time data for prices and fundamentals, but there has never been a point-in-time view of the whole web. At least until now. Exa Snapshot lets you search and retrieve the web as it existed on any date, using more than 400 billion page snapshots from the past 20 years. We'll walk through three uses: backtesting signals drawn from the web, keeping answer leakage out of agent evals, and reconstructing what was public at a specific moment for underwriting and KYB/KYC. You'll leave knowing how to try it with `snapshotAsOf` and how to talk to us about full coverage.

Key themes: • What are the key shifts in organizations that get AI/Agentic PDLC right? • How to manage skills/agents proliferation, without slowing innovation? • How to ensure and drive quality in agentic PDLC? • With coding activity compressed, where do we see bottlenecks form up in the PDLC? • What metrics/outcomes can organizations use to measure effectiveness of agentic PDLC?

A financial agent can pass every eval and still produce a decision no one can defend. The citation may point to the wrong document. The reasoning may omit the key business rule. The benchmark may treat last year's policy as today's ground truth. A high aggregate score can hide the failure that matters most. Defensibility requires four distinct controls: authorization, execution control, evidence provenance, and evaluation integrity. Using a synthetic audit decision trace, I'll work through each control: limiting what an agent can do, preserving what it did so contested steps can be reproduced, binding its material claims to the correct sources, and verifying whether its benchmark can be trusted. I'll also show where the agent should stop and a human should decide. You'll leave with a practical assurance pattern for financial agents whose decisions may be challenged months or years later. When that challenge comes, "the model said so" is not a defensible answer.

If you've ever tried rolling out an AI coding agent in a highly regulated and risky environment, you know the friction. Balancing productivity gains of agentic AI with the strict security, observability, and compliance constraints of a bank bring big challenges. So, we built our own. In this session, we'll explore how Monzo built Agent Chip, a sandboxed in-house agent that now safely authors ~10% of all merged PRs and runs over 1,800 tasks daily. Beyond the technical architecture of our MCP gateway and Agent Orchestration System, we'll focus on the foundations that make it possible to scaling an agent to hundreds of technical and non-technical staff. You will learn how we drove widespread adoption by integrating directly into Slack, how we balanced model non-determinism with the rigid guardrails required to keep our data secure, and the secret sauce of LLMs* that powers it all.

To be useful, an AI agent must be able to search *all* the data its user can access—and none they can’t. This is straightforward when the corpus is small. On millions of documents with overlapping and frequently changing permission scopes, it becomes a serious systems problem that prevents many enterprise agents from reaching prod. Bridgewater had to solve this problem while building the Vault, a centralized, high-quality repository containing millions of unstructured documents, including confidential data governed by user-level permissions. The Vault underpins several agents, including PAT, Bridgewater’s AI Analyst. PAT is already in production and is used daily by hundreds of Bridgewater investors. We'll trace why permission models that work at small scale, such as prefiltering on allowlists, don’t work as the corpus grows in size and complexity. Then we’ll show how Bridgewater and turbopuffer enforce complex row-level permissions at the retrieval layer rather than the app layer. *this is a co-presentation between Bridgewater and turbopuffer*

"Inside the Fable War Room" is the story of an enterprise experiment we ran in collaboration with Anthropic to aggressively forward-deploy their latest Fable model into some of our most complex distributed legacy systems. Rather than handing engineers another coding assistant, we put a small cross-functional team in a room, surrounded them with a network of on-call technical, product, and business experts, and armed Fable with a clear modernization strategy, a serious compute budget, access to the relevant tools and context, and a carefully-curated playbook. This playbook consisted of rules, skills, insider system archaeology and validation tips, instructions for Fable to continuously route questions back to the humans who could answer them in the form of editable Figma diagrams and interactive worksheets. We packaged the approach as an adaptable plugin methodology that other teams at GEICO can tailor to their own systems and architecture. On it's first day, this this approach saved the company $275k/year.

Your coding agent scores 74%. The release decision is due today. Nobody can say what that number has earned the right to decide. That ambiguity matters everywhere, but especially in regulated industries. Agents can modify transaction systems, access controls, reporting pipelines, and compliance-sensitive infrastructure. When software can affect money, customers, or regulatory obligations, “the benchmark looked good” is not a defensible release argument. Trust has to be supported by evidence that can survive challenge. An eval score compresses a model, task set, execution environment, evaluator, grader, and one sample from a variable process into a single number. Financial institutions have spent decades developing disciplines for decisions built on imperfect models: define the intended use, validate performance, expose limitations, independently challenge assumptions, and monitor what happens in production. In AI, we often just print the result to one decimal place and put it on a leaderboard. In this talk, I’ll put one convincing score on trial. The audience will choose between two release candidates, then reconsider that decision as I reveal what the aggregate concealed: a +5.4-point win in my own repository-level eval that shrank to +0.35 after repeated runs; a widely cited coding benchmark whose sponsor stopped reporting it after uncovering flawed tasks and contamination; and model graders that agreed with one another while missing the defects the release gate was supposed to catch. We’ll rebuild the decision around five checks: - What decision is this score allowed to change? - How large is the run-to-run variance? - Does the task distribution match the real workload? - Which plausible failures can the evaluator actually detect? - What are the grader’s true-positive and false-positive rates at the chosen threshold? You’ll leave with a practical evidence standard for deciding when an agent has earned greater autonomy, and when a precise score is disguising uncertainty that still belongs with a human decision-maker.

The next great AI companies may already exist, but they may not look like ones yet. Many businesses in mission-critical industries have entrenched customer relationships and strong market presence, but they lack the know-how to integrate AI and drive outcomes. M&A of traditional services businesses is one strategy for closing that gap. But how do you decide which industries to focus on? What makes a good AGI-proof business? And how do you deploy AI into those companies with measurable financial impact? This talk highlights learnings from bringing AI into the real world, what works and what doesn't, and how to evaluate opportunities for AI transformation. We present a different way to think about M&A: not as a financial engineering tactic, but as the go-to-market strategy for AI deployment.

The payment rails for agents are now readily available, but almost nobody uses them. The problem is not simply that people do not trust agents with money. It is that the basic one-shot payment flow handles only the easiest kind of purchase. x402’s `exact` scheme and MPP’s `charge` intent require a fixed price before execution begins, which only covers simple API calls. Newer primitives like x402’s `upto` and `batch-settlement` schemes and MPP’s `session` intent support variable or incremental charging, but they do not by themselves solve the full lifecycle of a job that runs for six hours and costs whatever it costs. I run a marketplace of 50,000+ community-built web data and automation tools, where all three purchase types occur in production. Fixed-price calls were easy to add. Metered usage took months to implement and still requires our own metering and orchestration layer. We still have not solved long-running jobs end to end, and I don’t think anyone has. I’ll explain where each protocol broke down in practice, what our transaction data reveals about where agent buyers actually stall, and the three things a complete agent-purchase stack still needs beyond the payment specs: a standard lifecycle for long-running jobs, auditable metering and delivery, and post-settlement refunds and recourse.

Your next user may be an agent discovering your API, creating an account, and taking action on someone’s behalf. Drawing on lessons from building for agents from registration to revocation, we’ll explore how onboarding, identity, and authorization change when agents become your users.

We analyzed 1,368 documented AI failures across healthcare, financial services, law enforcement, professional services, and consumer technology. The clearest lesson was not simply that AI models fail. AI risk lives in the system, not just the model. In our analysis, 92% of incidents implicated multiple weaknesses simultaneously. The typical documented failure crossed several parts of the deployed system: technical behavior, data, workflows, human oversight, controls, exposure, and downstream consequence. That creates a fundamental problem for how we evaluate increasingly agentic AI systems. Most evals still ask some version of one question: Did the model or agent complete the task correctly? But task success is not system success. An agent can reach the expected answer after retrieving unsupported evidence, selecting the wrong tool, passing invalid arguments, crossing an authority boundary, or taking an action that should never have been available to it. The output can say “pass” while the system tells a different story. Drawing on the research behind our forthcoming book, Evaluable AI: From Model Evals to System Evals, this talk introduces a different unit of evaluation: the deployed AI system. We use Risk-Based Evaluability Design (R-BED) to organize system evaluation around three connected questions: Technical Failure: Where can the system fail? Deployment Harm: Under what conditions can that failure reach people, processes, or consequential decisions? Business Impact: If it does, how broadly and severely can the consequence propagate, and what can be recovered? These questions cannot be collapsed into a single score or task-success metric. A technically weak system may remain contained. A seemingly small technical error may become material because authority, exposure, or controls allow it to propagate. And a system that appears to perform well may still be difficult to trust if its behavior cannot be reconstructed and challenged. This leads to the concept at the center of the talk: evaluability. Evaluability is the ability to determine, with evidence, whether a deployed AI system remains within acceptable bounds and continues to serve its intended purpose. For agentic systems, that means making the trajectory observable: the evidence retrieved, sources used, tool selected, arguments passed, state transitions made, authority exercised, human interventions triggered, and actions taken. Tanmay Sah and I will demonstrate this through a reproducible evaluation pipeline applied to benchmark agent trajectories. We will show where task-level success can conceal failures elsewhere in the trajectory, what can be evaluated directly from system evidence, where an evaluator or judge is required, and when the correct evaluation result is not pass or fail but insufficient evidence. We then turn those findings into an engineering discipline: BUILD: Define the system claim, operating boundary, evidence requirements, and permitted authority. BREAK: Stress the system, discover failure regions, and trace failures across evidence, execution, tools, workflows, and controls. FIX: Constrain, redesign, route, monitor, or intervene, then verify that the change actually worked. The key insight is that meaningful system evals cannot simply be bolted onto an AI application after it has been built. If the system does not expose the evidence needed to reconstruct what happened, important questions become impossible to answer. Evaluability has to be designed into the system. As AI systems become more autonomous and gain access to tools, workflows, data, and consequential actions, the distance between a technical error and a real-world consequence is shrinking. The next generation of evals therefore cannot stop at asking: “Did the model get the answer right?” They must also ask: Where can the system fail? Can that failure reach the real world? What happens if it does, and did we build enough evidence into the system to know?

For decades, financial institutions have managed open source software risk by accepting it. Legacy applications, strict change control, and the operational risk of touching business-critical systems made a growing CVE backlog the cost of doing business. That calculation no longer holds. Frontier models can now surface dormant vulnerabilities and chain weaknesses across the open source software supply chain faster than traditional security processes can investigate and patch them, while regulators, auditors, and customers demand stronger evidence of software integrity. Join us to explore how financial institutions can rethink how they govern and adopt open source as risks accelerate. We’ll discuss why reactive vulnerability management is breaking down, and how teams can reduce risk without disrupting business-critical systems, and the practices security and platform leaders can use to establish a more secure, governed approach to open source consumption.

RAG is about four lines of code, and it works fine until the conversation gets interesting. Then your agent forgets what you said two turns ago and starts repeating itself, while the token bill climbs. This talk is about the jump from retrieval to memory: why a read path stops being enough, and what you build in its place. We'll separate the kinds of memory worth keeping apart, then reassemble the prompt every turn instead of letting it grow forever. You'll leave knowing why the memory layer is the part of your stack nobody else can copy.

If you care about someone, you should build them a factory. I built one to help my colleagues publish what they know, and another to turn my family’s days into illustrated storybooks. Through real requests, revisions, and finished work, I’ll show how AI can improve narrative outcomes when human intention, craft, and judgment shape the process. Years of making something teach you what good looks like. That knowledge can become a tool other people use to tell stories you couldn’t tell for them. This talk is about building that capability, putting it in their hands, and finding your own contentment in what they make.

Every investment firm should have AI in their budget. However, far fewer can say what it returned, especially when it comes to a traditional, fundamental equity investing strategy. We are going to showcase the methods we've built to prove how agent workflows measurably contribute to performance when returns are made up of so many different factors. In a long/short hedge fund, the AI workflows we've developed sit directly in the investment process, rather than alongside it. Collaborating with our investors & researchers to develop tooling together has led to high-value products that get used daily, including those for new idea generation, overnight news synthesis, and earnings preview. These workflows greatly improve the research process, and we are now able to trace their impact back to the book. We'll discuss how every idea that a workflow produces is logged at the moment of generation, along with its source and reasoning, so it can be scored for utility. From there, we have built systems to track how many ideas reach our PM to become positions, and how those positions perform against given benchmarks. For softer AI-native signals, we grade against the traditional way that same metric would have been produced and measure both time & signal improvements. This has made great strides to solve one of the biggest challenges in AI implementation: understanding the ROI and alpha generated by every tool and decision made via these systems. We'll walk through this measurement and attribution process, both from a technical lens as well as a product framework. Ranging from the successes to pitfalls, this talk will have you energized and prepared to start measuring the impact of AI in your investment decisions.

Why Modal Is Built for AI Agents at Scale

The biggest mistake we're making with AI agents is treating them like software. Organizations have faced this problem before: how do you delegate real authority without giving up control? in this talk, we will explore the Agent Accountability Framework and the agent governance principles taken from a century of human governance learnings.

What's an eval? Why are they important? How does Arize help you do them? Head on over to the Arize AI booth to get the lightning version of our introduction to the art and science of evals.

Attackers are starting to use autonomous reasoning to find and exploit vulnerabilities faster, while pattern-based scanners miss flaws that depend on intent rather than syntax. Hunter Agent is Sonar's guided reasoning agent, built to find the logic flaws conventional scanners can't see. In this 15-minute live demo, watch it investigate a real application security scenario: connecting a SonarQube cloud project, running the security playbook, and tracing code paths and identity flows like a manual auditor. You'll see it catch broken access control, business logic flaws, and authentication or session vulnerabilities, then raise confirmed findings for your team to triage and fix. You'll leave with a clear picture of how this adds an auditable, repeatable layer of security analysis to your existing workflow, helping teams catch deep logic flaws before attackers do.

Distressed credit and restructuring is one of the most specialized areas of finance. The data is messy, definitions are unclear, and even extracting a single data point can require judgment. In bankruptcy, simple facts (i.e. cash on balance sheet, debt outstanding) can be disputed and depend on context or legal interpretation. This talk explores what it takes to build AI for a complex, regulated, and judgment-heavy domain. I’ll share my start up (Axim's) approach to "analytical provenance" and "DistressedBench," a new eval framework designed specifically around the reasoning tasks required in distressed investing.

Rayan Krishnan, CEO and co-founder of Vals AI, walks through what building industry-standard benchmarks has taught his team about the jagged frontier of AI in finance: the common failure modes, patterns of behavior, and why no single model leads on every task. No two workflows are identical, so independently evaluating models on your own work is essential for proving ROI, optimizing spend, and maintaining a competitive edge.

An agent calls a tool, the tool calls an API, and every request has a valid credential. We'll examine how that flow can still give the wrong user access to a resource. Using a small MCP application, I'll trace a request through a gateway and an SDK-protected service, showing where user identity and delegated permissions need to travel. We'll compare the integration costs of both patterns and show how to combine them without rebuilding authorization in every application. The demo will use Keycard for credential exchange and policy-controlled access, then run allowed and denied cases against the protected flow. Attendees will leave with a practical way to decide where to enforce authorization, which identity context to preserve, and how to test access boundaries independently of the model's judgment.

A human financial analyst would never put a number in front of a client that they cannot verify. Kepler has built an AI analyst that follows the same rule. It can only cite a figure that it has retrieved from a verifiable source. That only works if search retrieves the right number. Vinoo will share how Kepler searches 100M+ SEC filings, earnings transcripts, and investor presentations across 14,000 public companies in one large shared namespace — alongside each client's private documents living in their own namespace — to return financial analysis generated from numbers that are never hallucinated. You'll learn how to avoid analytical hallucinations by building an AI system that retrieves source material from very large indexes with high recall and low latency, a verification model that never allows a model to invent a number, and a security model that fully isolates privileged financial data.

Delta is a futuristic agentic development GUI from the creators of the high-performance Zed code editor. This session demonstrates using Delta to work on a real-world problem, while explaining how Delta's CRDT-backed persistence system keeps a perfect history of not only code changes and conversation, but how the two relate. You'll also learn how it works asynchronously (or in realtime) with multiple human collaborators across multiple agents, and see a live demonstration of how this all comes together in practice on a real project.

Your harness matters as much as your model when it comes to institutional buy-side research. In our benchmarks, moving one model to the right harness improved its score by more than 60%, as much as switching models entirely. Off-the-shelf agents fail because they miss hidden accounting traps, trust stale summary tables, and burn their token budget in degenerate search loops. We’re launching Mandate-Bench, a benchmark built by LinqAlpha with Google DeepMind’s Kaggle Benchmarks team to test real investment origination: the work an analyst does before a portfolio manager ever sizes a position. We evaluate agents on licensed point-in-time market data across 1,300 US stocks and nearly 5,000 SEC filings, where only 15.8% of frontier model runs meet the bar of an institutional portfolio manager. In this talk, we will walk through our benchmark findings and break down lessons learned from shipping agents in production to real hedge fund clients with billions in AUM.

Jack Woodruff ran one of the largest books at Citadel and went on to found Candlestick Capital. Today he's the buyer every finance AI company wants to win, and in this fireside chat with Vinoo Ganesh, CEO of Kepler and former Head of Business Engineering at Citadel, he'll speak candidly about what AI is changing in fundamental investing and what it isn't. Jack will share where AI is saving real time in his process, what he'd never hand over to a model, and why the judgment call still belongs to the investor. They'll also take on the questions most people only ask in private. Will the biggest funds win everything? If everyone has the same models, where does alpha come from? And when does a number have to be right, not just plausible?

AI agents have moved from experiment to essential infrastructure. KPMG estimates agentic AI will drive $3 trillion in annual productivity gains, and a June 2026 industry survey found 57% of teams already have agents in production, but adoption has outpaced trust. The same survey found 32% cite quality as their top barrier, and Gartner projects more than 40% of agentic AI projects will be canceled by 2027 due to cost, unclear ROI, or inadequate risk controls. That gap between "we built a demo" and "it survives production" is where most teams get stuck. Agents crash mid-task, lose progress, and fail silently in ways traditional monitoring never catches, and the more autonomous the agent, the more expensive that failure gets. This talk is a status report were we'll talk about what's actually going wrong when agents hit production: the crashes, the silent failures, the debugging sessions nobody signed up for, and what the teams who've figured it out are doing differently. No single stack or vendor is the answer here; the point is the underlying shift in how you have to think about state, failure, and visibility once an agent is running unattended. You'll walk away with a framework for asking the right questions about your own systems, wherever you're building.

Building a basic AI code review agent is easy. Building one that reliably catches meaningful bugs without overwhelming engineers with noisy comments is much harder. In this session, Daksh Gupta, co-founder and CEO of Greptile, will explain how to build a production-quality AI code reviewer. He’ll cover five key improvements: better prompting and evaluation, using different models to write and review code, giving agents deep codebase and product context, testing changes inside live development environments, and routing each task to the most effective model. Designed for teams building an internal reviewer or evaluating existing tools, this talk offers a practical look at the architecture, infrastructure, and tradeoffs behind AI code review at scale.

Optiver trades on thousands of instruments across every major exchange, and the code behind that has to be correct the first time. That's a demanding starting point for "AI-first." Over the past year we committed to it anyway: agents in every engineer's workflow, and a platform team whose job is to make the best way to build with AI also the easiest one. This talk covers what that took, with the parts that didn't work included. We'll walk through the two-track operating model we settled on: a supported golden path with the tooling and guardrails engineers can rely on, and an experimental funnel where anything can be tried, on the condition that it emits telemetry and either graduates or gets sunset. We'll show why we stopped treating model choice as a vendor decision and started treating it as a market: task-specific exams as price discovery, a gateway that routes to the cheapest certified model, and independence built in through multiple channels and self-hosted open weights. And we'll get into the unglamorous parts, such as where inference spend actually goes once a few thousand people use agents daily (long-context work, not runaway sessions), why context-compaction savings mostly don't materialize, and how to keep cost controls from getting in the way of legitimate heavy work. You'll leave with a concrete operating model, an evaluation approach that doubles as a routing signal, and a list of the assumptions we had to abandon.

Agents can now discover paid APIs and authorize machine-to-machine payments without a human in the loop. Protocols like x402 make this work over plain HTTP. But a payment that settles correctly is not the same as a payment that should have happened. I will walk through six attacks on an autonomous payment agent: price manipulation, duplicate payment, free shopping, payment-resource mismatch (the agent pays for one resource and receives another), malicious service discovery, and prompt injection into the payment decision. Two of them I will run live on a public testnet. In most, the transaction is technically valid, and still wrong: the agent overpays, pays twice, pays the wrong seller, or gets the wrong item. In one, the agent receives paid data with no payment at all. Then the controls that actually constrain the surface, every one of them deterministic, with no model call in the authorization path: spending limits, recipient allowlists, resource binding, nonce checks, retry limits, and pre-payment validation. Attendees leave with a threat model, a practical testing checklist, and a lab they can run against the official x402 SDK, on a testnet or a local simulation, without risking real funds. Key Takeaways • A working threat model for autonomous agent payments, with six demonstrated attack classes • Deterministic controls that do not require another model in the loop • A lab for testing your own x402 integration safely, on a testnet or a local simulation • Why “the payment succeeded” and “the payment was correct” are different assertions

Inside Revolut’s internal coding stack with Together AI. Revolut became a fully licensed U.K bank in March 2026, and serves more than 75 million customers. Its engineering org still moves at startup speed. Holding both at once is the problem every engineer in financial services recognizes. Revolut runs two kinds of models against its internal coding work: Language models that turn written descriptions into working code, and vision models that turn mockups into a working interface. That second capability is the one most teams in fintech still haven’t shipped. You’ll leave knowing which of the two to deploy first, what changes when your codebase is subject to audit, and whether mockup-to-interface is ready for the work your team actually does.

Choosing a vLLM version and serving config for a new model is usually guesswork and the wrong guess usually ships to production. In this live demo, we show vLLM BenchPod, Runpod's zero-setup template for serving and benchmarking any model on a single GPU pod. We boot a pod from a prebuilt image, serve a config, and watch vLLM come up with a live metrics dashboard. Then we run a real benchmark against the serving endpoint, swap configs, and compare results side by side, all with no repo checkout, no CI, and no infrastructure to manage. You'll leave with a working pattern for validating models and serving configs before they ever reach production.

Great financial agents unlock new ways to get work done across the web. Learn the core concepts behind browser-based agents, how to build them with Browserbase, and how companies like Ramp use them in production. You’ll leave with practical ideas for automating workflows across financial portals, from collecting receipts to retrieving documents.

Every enterprise is running agents that call tools, use MCP servers, and touch production systems, at volumes no human review loop can keep up with. That creates exposure: prompt injection, over-permissioned agents, MCP sprawl, and behavioral drift. At enterprise scale, defense in depth is the only architecture that holds. No single control catches everything. You need policies, runtime threat detection, and fine-grained access controls applied at every layer of the agentic stack, paired with full-session observability and behavioral baselines so you can find signals in the noise. You'll leave with a practical framework for securing AI agents without slowing adoption down.

Agent behavior emerges at runtime. So why are we still debugging only at the code level? A prompt edit, model swap, new skill, or tool change can fix exactly what you were looking at while quietly breaking something you weren't. In one example, a one-line policy fix raised an agent's action-safety score while dropping task completion from 0.89 to 0.72. The aggregate outcome looked fine. The regression was hiding underneath it. This session is a hands-on walkthrough of how to turn your agent observability into a continuous regression testing loop. We'll build a regression dataset from real production traces, run a candidate change against it as a full-agent experiment, diagnose competing metrics, and put the workflow into CI so every change is evaluated before it ships. We'll automate the full feedback loop from observe → evaluate → change → test → improve. Observability is becoming more than humans understanding agents. It's becoming the infrastructure agents need to understand and improve themselves.

Why do organizations with access to the same AI achieve vastly different results? This keynote explores the technical and organizational choices behind that gap - from choosing the right problems and redesigning workflows to building shared capabilities and feedback loops that turn AI into sustained competitive advantage.

As AI agents take on more of the software development lifecycle, engineering teams face a new challenge: moving faster while maintaining code quality, consistency and control. This session explores how quality and governance can evolve alongside AI adoption, from shared coding standards to independent review and human oversight. Through practical examples and a live demo of Qodo, attendees will see how AI powered workflows can help teams scale development with greater confidence in the code they ship.

Financial voice agents operate in an environment where a misunderstood date, amount, identity, or authorization can become an operational and reputational risk. Yet teams often select models using component leaderboards and optimize latency metrics that do not reflect what callers actually experience. This session examines voice agents as complete realtime systems rather than chatbots with speakers attached. Attendees will learn how speech recognition errors flow through an agent stack, why perceived latency differs from dashboard latency, how asynchronous tools preserve natural conversations, and how turn detection affects both user experience and task accuracy. Using production examples and an open evaluation methodology, Jesse will show how to test complete conversations against real outcomes, including correct actions, privacy safeguards, efficient tool use, and natural turn-taking. Attendees will leave with a practical checklist for building and evaluating reliable voice agents in financial services.

Building an agent is easy. Deciding where its code runs is the part that follows you to production. This talk comes from a year of building Daytona integrations across the agent ecosystem: LangChain, CrewAI, Google ADK, Mastra, Pydantic AI, the OpenAI Agents SDK, and a long tail behind them. Integration after integration, the same pattern kept surfacing. Frameworks that agree on nothing, not even the language, not the mental model, not who controls the flow, all converge on one identical primitive: give the model a computer. We'll trace why. Every framework ends up facing the same question: where does the model's code actually run? Only three answers exist, and two of them stop holding once real users show up. That's why the past year went the way it did, with frameworks deleting their own code interpreters rather than defending defaults they couldn't. And that's why the computer keeps growing: first a tool call, then the place the agent lives, then a thousand machines at once. You'll leave with one rule: pick the boundary before the framework. Frameworks are opinions. The boundary is a commitment.

Arize started building its in-product agent on GPT-3.5 with a vision and no clear path. Two years and one full architectural rewrite later, it plans multi-step work, manages memory well past the context window, debugs itself from its own traces, and carries fixes through to pull requests. This talk is about the decisions along the way, especially the ones that got reversed. It covers why planning had to be built into the harness instead of left to the model, how the memory design changed once sessions outgrew the context window, what it took to use the agent to debug the agent without fooling ourselves, and what still needs a human in the loop after all of it. Every example comes from a real incident or a real design argument. Attendees building or maintaining a long-running agent leave with a list of decisions they'll face and a record of how one team made them.

Frameworks for Thinking about Applying AI to Investing

Pim de Witte is the co-founder and CEO of General Intuition, the frontier lab for acting in space and time. He previously founded Medal, the world’s largest and fastest-growing platform for gaming clips, where millions of gamers share their best moments. Before gaming and AI, he worked with Médecins Sans Frontières and ran the largest Runescape private server at 14. General Intuition recently raised $650M+ from Khosla and General Catalyst.

Modern businesses require open agents.

Ramp bets on the Bitter Lesson: models will keep getting dramatically better. But finance teams need capabilities today. This talk covers how Ramp Labs finds the gaps the frontier hasn't closed yet and closes them by replicating research quickly.

Software's future runs on agentic factories — and the same lessons that transformed manufacturing (prevent defects at the source, push quality control to the people doing the work, target the bottleneck) apply directly to building them. In a world of code abundance, the real constraint isn't model capability — it's verification: can you trust the code AI agents produce enough to put your name on it? Tariq will show how enterprises can build software factories that ship fast and stand up to scrutiny, centered on three interlocking loops: an agentic loop that catches issues as code is generated, a CI verification loop that catches what slips through before shipping, and a maintenance loop that fixes issues already in production. Attendees leave with a practical framework for designing verification-powered agentic pipelines of their own.

Institutional IT has run on one mantra for decades: People, Process, Technology. Agents now execute work that once required a human hand, and policy-controlled organizations need a model that accounts for them explicitly. Using workflow examples and basic graph theory, we’ll show how to map a clear, auditable process while giving agents room to loop and go deep within defined boundaries. We’ll also explore how software quality assurance changes when outputs are probabilistic. Attendees will leave with a practical approach to deciding where agents belong, when humans should stay in control, and how to use measured quality to guide greater autonomy. The goal is to gain the efficiency of agents while preserving the controls, visibility, and accountability that policy-driven work requires.

What does it take to turn advances in AI into meaningful progress for financial services? Joseph Kim, Head of ChatGPT for Financial Services, explores why and how the product was built, and the research and product direction guiding its evolution.

With the recent agent breakout news, AI is changing both sides of the security equation. It is: giving attackers new tools while creating new risks inside the systems we build. This talk examines recent real-world incidents, the attack patterns emerging around AI-enabled systems, and practical solution patterns for defending against them. From prompt injection and data leakage to agentic AI and supply-chain risks, we’ll focus on what’s happening now, what actually works, and how security architecture needs to evolve.

Off-the-shelf models are finally good at basic analyst tasks, whether it's small data cuts in Excel, point edits in PowerPoint, or first-draft memos. But ask anyone on Wall Street, and they'll tell you there's still a wide gap between what Claude or Codex can do and the leverage a real junior analyst gives you. This talk is an anatomy of that gap, drawn from production experience building agents for finance workflows: (1) multiplayer agents — deal work happens in a thread between an associate, a VP, and an MD, so the agent has to sit in the group chat and on the email chain, know who said what, answer the right person, and turn each correction into firm memory; (2) Office rebuilt for agents: finance runs on Excel, Word, and PowerPoint, but long-horizon agents need native engines rather than the tools Microsoft exposes, and (3) context at scale — a deal doesn't fit in a context window: a 15,000-file data room, dozens of parallel workstreams, and years of firm conventions all have to reach the right agent at the right moment. We'll close with a mapping of which analyst workflows are fully delegation-ready, and which are still copilot automations that require humans in the loop.

Model upgrades are often sold as more capability per dollar. In production, the harder question is what quietly changed underneath: attention sparsity, 4-bit quantization, speculative decoding, or how we prompt and template the model itself. This talk is a practical tour of those changes and the tests they require. Using a recent model release as a concrete case study, we’ll examine how blockwise 4-bit attention and MoE quantization change scale selection and rounding; how speculative decoding changes serving assumptions; and how reasoning controls can alter the path from API request to prompt construction. Attendees will leave with a good understanding of the hidden costs of change that affect latency, output quality, reproducibility, observability, and rollback for teams shipping AI in latency-sensitive and high-stakes environments.

As Untangle's solo founder, Ryan Carson runs 10 to 15 AI agents around the clock, and together they ship about 40 pull requests a day. The skill that makes this work is management. In this talk, Ryan shows how he runs agents like an org chart: priority folders for every thread, a handwritten weekly priorities sheet that keeps him focused, a "Watchdog" agent that reports the top issues at every customer, and a review step where an agent records a video before anything ships. He'll explain why companies were built as pyramids because humans can't manage a thousand direct reports, and what changes now that agents can. He'll also cover why he hires people by watching them manage agents instead of interviewing them, and why the one thing agents still can't do is tell you what to build. That still means getting out of your chair and talking to customers.

Agents are becoming the primary consumers of financial and developer knowledge, and the stakes are higher in fintech than almost anywhere else. When Coinbase Developer Platform rebuilt its documentation, agents quickly grew to 59% of all traffic, with over 30,000 MCP tool calls and 145 distinct AI tools pulling from its docs to build onchain. The old question was whether a developer could understand the docs. The new question is whether an agent can reason over financial infrastructure correctly, safely, and fast enough to move real money. Hahnbee Lee, co-founder and CTO of Mintlify, and Lincoln Murr, Head of Applied AI at Coinbase, will share what it took to rebuild CDP's documentation as knowledge infrastructure built for agents, and what that means for the future of trust and speed in financial systems.

Upstart runs voice agents across hundreds of thousands of customer calls a month in one of the least forgiving environments for AI to get wrong: regulated consumer lending, where a bad answer isn't just a bad experience, it's a compliance problem. This talk is the real path Upstart's AI/LLM engineering team took to make that manageable: starting from ops teams manually reviewing 100% of calls, building LLM-as-judge metrics that had to earn trust by matching human graders before they were allowed to run unsupervised, and the automated eval loops they're building toward now. Shiv will walk through what broke along the way and the actual steps they took; Brooke Hopkins will cover the evaluation-infrastructure mechanics underneath, drawn from applying the same closed-loop simulation discipline she built at Waymo to conversational agents. The throughline: autonomous, self-improving agents don't start with a better model, they start with evals a regulated business can actually stand behind an auditor.

Investors just rated Lambda investment grade and are about to price Nscale at IPO. SemiAnalysis shows the operator spread that compute indexes and futures still miss, how lenders already price it, and what a contract should settle on.

The next model release isn’t going to fix your agent. This panel gets into what actually separates agents that demo well from agents that work: context, tooling, workflows, and the engineering around the model.

More context should make an AI agent better. In our accounting agent, it often did the opposite. At Ramp, we built a benchmark of real-world accounting workflows to test frontier models, tools, skills, memory, and the agent harness end to end. What we found was that some of the most intuitive improvements actually made the system worse. In this talk, I’ll walk through what we learned building and benchmarking a production finance agent: how we used skill ablation to identify context that was hurting performance, reduced one frequently loaded skill from 14k to 5k characters without sacrificing quality, and learned the hard way that agent memory needs to preserve reusable process, not stale facts from the last task. The broader lesson is that production agents are systems problems, not prompt problems. Better models help, but tool design, context management, telemetry, evals, latency, and cost all interact. I’ll share the metrics and development loop we use to decide whether an agent change actually makes the product better. Attendees will leave with a practical framework for evaluating agent-harness changes against quality, reliability, latency, and cost rather than optimizing any one of them in isolation.

Knowledge work on enterprise data looks solvable: point a capable model at contracts and spend, let it reason through the sourcing decision. The approaches vary - direct retrieval, retriever-augmented systems, agentic SQL access with governed infrastructure - but they're all trying to solve the same problem. When the data is ambiguous, each one fails differently. This workshop compares how different architectures handle enterprise knowledge work when parallel agents operate without visibility into each other's outputs. You'll see what breaks when agents work on vendor contracts and spend data: where sub-agents independently claim the same savings, where citations resolve cleanly but describe the wrong thing, where reconciliation works beautifully and proves the agent solved the wrong question. We'll examine working implementations: where a better data layer genuinely helps, where it masks problems, and one case where cleaning the warehouse made the answer worse, not better. Each approach trades off differently between false confidence and surfacing uncertainty. Leave with a toolkit for your own systems: which architecture fits which problem, what signals tell you it's failing, how to set confidence bounds for a real business decision, and the structural patterns that actually catch failure instead of hiding it.

Tok/s is the GPU industry's vanity metric. It measures how fast a system generates tokens that may or may not solve your problem, at a concurrency of one, under conditions nobody deploys in. If you're making infrastructure decisions based on tok/s, you're flying blind. n this talk, we'll uncover and quantify how models, the software stack beneath and the infrastructure supporting it all contribute to the right token economics. We will present aistack, an open benchmarking initiative by imec that measures what actually happens when real AI workloads hit real hardware. What 48 concurrent coding agent sessions - across various open models - do to your GPUs - across various hardware configuration. We provide insights on the tipping points for each model to switch from API-costing, to leasing GPUs, to owning your own infrastructure. We will show this in detail through a thorough analysis of what software coding currently buys you across models and hardware. We ran 64 SWE-Bench Pro tasks across everything from a DGX Spark on a desk to an 8×B300 node, swept concurrency from 1 to 128 across multiple frontier models, and tested five different coding harnesses holding model and hardware constant. Some findings were uncomfortable: a single H200 comfortably serves 32 concurrent devs, but an 8×B300 running a frontier-matching model starts crawling at 8. Renting GPUs saves 35× over API for one model and costs more than the API for another. vLLM's throughput collapsed at high concurrency in ways no datasheet warned about. And the harness, the part most benchmarks ignore, turned out to move the needle more than the model in some configurations. But this talk goes beyond the data. It's about why nobody had done this measurement before and what it takes to build the neutral ground between chip designers who plan in 3-year silicon cycles, model builders who ship weekly, and the infrastructure operators caught in between. aistack exists because those three groups are optimizing against different metrics, running their own benchmarks, and publishing numbers that don't compose. We're building the place where they look at the same evidence. This talk shows what that looks like in practice, what we've found so far, and what's coming next.

San Francisco Compute built the first real market for compute. But a compute market only works if it's physically settled. This determines what can be standardized, what a price even means, and which market designs survive contact with reality. Evan Conrad will walk through where this belief came from, what it rules out, and what it implies for anyone buying, selling, or betting on compute.

You'll leave with a data-backed argument for making accessibility a default in every AI-assisted development workflow. Not just because we build products for everyone, and everyone deserves to use them — but because it's cheaper too.

AI agents are becoming economic actors, but today’s financial rails weren’t built for them. Give an agent a bank account today and it inherits every constraint designed for humans: cards that decline sub-dollar charges, transfers that take days to settle, and permission models that assume a person is always the one clicking “approve.” This talk explores what it takes to build a financial account for agents from the ground up, laying the foundation for AIFi rather than forcing agents into infrastructure that was never designed for them. Lincoln Murr will unpack what changes when an agent needs to hold assets, transact autonomously, and move money at machine speed. He’ll cover how stablecoins enable machine-native payments, how to design accounts with human-defined controls, and what Coinbase is learning from x402, the open payment protocol that has already processed $53M+ across 193M+ payments for AI agents, apps, and APIs. With agentic commerce projected to reach $3–5T by 2030, this is a look at the infrastructure that has to exist for that future to become real.

CircleCI analyzed 28 million CI workflows this year and found that throughput jumped 59%, the largest increase it has ever recorded. But nearly three in ten attempts to merge into production now fail. Writing code has stopped being the constraint; the new challenge is validating it. In this onstage conversation, Manish Kapur, VP Product Marketing and DevRel at Sonar, sits down with Addy Osmani, Member of Technical Staff, Anthropic to look at the current landscape from both ends: the people building coding agents, and the teams who have to decide how to build trust in what those agents produce. Manish will preview new Sonar research, never shown publicly, based on a survey of more than 1,000 developers. The two will dig into the questions it raises: What percentage of AI-authored changes reach production with no separate human review at all? How has the split between writing and reviewing shifted since agents arrived? And are the productivity gains still climbing, or have they quietly plateaued? You'll leave with a clear read on which verification practices separate the teams shipping fast from the teams shipping blind, and what it takes to keep review and testing from becoming the bottleneck as agents take on more of the work.

We show how our CFPs built organizational skills to make planning more efficient and effective. Our planners write and maintain the agent skills they use daily, and none of them write code. We started by using AI as a chat box to polish client messages. The real shift came when work began happening in the background: I can be on a call with one client while a full cash flow or insurance analysis is drafted for another, then review, adjust, and send recommendations afterward. A planner’s bottleneck was never typing speed, but being in only one place at a time. That requires an agent that works where the work lives and knows how we do planning, not how planning works in general. That’s a skill. To build one: pick a repeatable workflow, walk the agent through one real case while narrating your judgment, package it with the questions it must ask and the limits it must respect, test it on five to ten real cases, then ship it with an owner and a reviewer. Six months in, we have 24 skills (17 in planning), and adoption spread to sales, product, and marketing without a mandate. I’ll cover what changed when subject matter experts built for their own niches, how we peer-review skills like we review plans, and our open question: how to check quality at scale when the output is judgment.

Compute is the core input to AI, yet it still trades the way many commodities did before exchanges existed: through private quotes, brokers and long bilateral contracts. Affaan Mustafa & Alejandro Soumah (Compute Trading Leads at Jump Trading) will step back from individual prices to look at the forces that set them. On the supply side, that means power and data-center buildout, along with the lenders and depreciation assumptions that decide what new hardware really costs. On the demand side, it means the way buyers are shifting between generations, contract lengths and providers. He will then turn to policy, from export controls and sovereign demand to the first regulated compute futures, and close with what a more mature compute market could mean for anyone who builds on it.

Many critical monetary, information, and communication transactions still end with human verification. To move money, recover an account, or pass KYC, you often have to jump on a call and show your face to prove you’re a person. That check is the backstop a lot of security rests on, and, as I argue in this talk, a potential control mechanism for AI agents. This summer’s Hugging Face incident—where AI agents broke into a third party’s production infrastructure, despite being tested in a closed, internal evaluation sandbox—was a reminder that agents reach for whatever capability is available to finish a task. Whether you read it as intent or missing controls, now that human-like avatars can join a video call as a convincing person, could agents use one to get past the human on the other end? In this talk, I measure when today’s agents actually do this, what tips them over, and whether safety alignment is enough to stop them. Besides examining the traces of agents deciding to impersonate a human, I will discuss control, alignment, and detection, with a particular emphasis on organizations that already employ human verification.

Jediah Katz discusses how we built Cursor and Grok Bot, and where we think the agent harness is going.

There is a massive paradox in software engineering right now. AI coding agents have ascended to become an incredibly important and powerful part of development, due to substantial improvements in model capability over the past year. If you follow the spirited debates over benchmark performance and general vibes that attend every new model release, you might think that improvements to foundation models are the most important factor driving our industry today. However, if you looked at real-world production data from software development workflows, you’d find that after controlling for differences between individual developers, model choice explains almost none of what ships. What matters most is the individual’s abilities and judgment, and the environment (the skills, harness, and context) they’ve assembled for themselves. I’ve been studying developer productivity using data from the Jellyfish platform, which combines AI agent telemetry with software engineering workflow data across 1,000+ companies and hundreds of thousands of developers. We see in the data that premium model mix doesn’t correlate with meaningful gains. What does matter is experience and scaffolding. Senior developers are continuing to out-ship junior developers, and this gap grows as token spend increases. And investments in skills and context compound: developers and teams building reusable skills ship measurably more each week, with returns that grow alongside their platform investments. The foundation models will keep improving, and without much help from most of us. This talk is about the part we all control, which matters even more: the developer practices and workflow tooling you need to cultivate top operators on your own team.

AI systems used in high-stakes environments cannot be evaluated solely by asking whether a model produces the right answer. A system can perform well on standard benchmarks while failing in unexpected, adversarial, or consequential situations. In this talk, I will explore how engineers can design evaluation systems that better reflect the realities of production AI. I will cover how to identify meaningful failure modes, evaluate failures based on their severity and impact, combine automated evaluations with human review, and use red teaming to uncover behaviors that conventional tests may miss. The session will also examine how evaluation needs to evolve as models, prompts, tools, and surrounding systems change. I'll bring these ideas to life with a first look at eSlams.com, a new platform and open-source framework where AI models play games to explore strategic reasoning, adaptability, and decision-making under uncertainty. The goal is to provide a practical framework for building evaluation processes that help teams understand not just whether an AI system works, but where it can fail, how those failures should be prioritized, and whether the system is ready for high-stakes use.

AI in coding has gone from autocomplete to agents attempting full Rust rewrites of browsers and compilers. Meanwhile, the zeitgeist in finance is still limited to spreadsheet edits and deck formatting. Why aren't AI agents taking positions? We built an internal proof-of-concept at Samaya, and here's what we learned. We first created agentic harness that can rewind time and block any information from the future. Then, instead of predicting stock prices, which can be noisy, asked the agents to predict the headline metrics of an earnings print. We'll share what worked, what didn't, and how close we think this actually gets us to an agent that can make a real bet on the future.

Coding agents produce 741% more code, and only 30% more software ships as a result. The gap is human review: people top out around 400 lines an hour and agents are not slowing down. This talk is for engineering leaders trying to decide what to do about that. It covers what happens to code review when the reviewer is the bottleneck, which review work is moving to agents and automated verifiers and which is not, what the change does to hiring and to the junior pipeline, and where the "product engineer" role is landing as the line between writing code and shipping software dissolves. It draws on published data from the largest code hosts and on how Arize restructured its own engineering and operations work around agents that generate more traces and eval results than anyone could read. Leaders leave with a framework for deciding what their people should read, what they should delegate, and how to measure whether the new arrangement is working.

Modern finance runs on software that was built decades ago, understood by fewer people each year, and too critical to stop. The hardest part of modernization is not writing replacement code. It is reconstructing the hidden map of business rules, dependencies, data flows, and downstream effects that accumulated around it. Neeraj Deshmukh of Blitzy and Stephen Chin of Neo4j will show how autonomous AI agents and a dynamic knowledge graph can turn a legacy codebase into a living model of the system, then use that model to plan, generate, and validate a migration to a modern stack. Drawing on implementations at some of the largest financial firms in NYC, they will examine what makes this approach production-safe: system-level context, explicit guardrails, continuous validation, and human approval at the decisions that matter. The result is a practical pattern for compressing modernization from months to weeks or days while preserving the behavior financial institutions cannot afford to lose.

Traditional product experimentation assumes that we can define a feature, measure a response, and compare outcomes. AI products make that much harder. Outputs can vary, user responses can depend on context, and a metric improvement does not necessarily tell you why the experience improved. Drawing on experimentation across consumer technology and fintech, as well as the development of an AI-powered product experience at Bumble, I will examine what product teams need to consider when evaluating AI-powered experiences. We will look at common experimentation failure modes including metric selection, novelty effects, premature conclusions, and misleading behavioural signals, then connect them to the additional complexity introduced when the product experience itself is generated or personalised. I will share a practical approach for defining the product question first, selecting meaningful outcomes, interpreting experiment results, and deciding when an AI feature is ready to scale.

Interactive agents aren’t scaling. Every company is coping with runaway costs, lack of governance, and difficulty automating. This has led to the need for software factories: a closed-loop system for automating development, with primitives that provably improve throughput and efficiency over time. Right now, folks are trying to roll their own internal software factories from bare metal, solving the same problems 100 different ways. We need shared, flexible infrastructure that everyone can build upon. In this talk, we’ll share the core building blocks every company needs to build their own self-improving software factories based on our own experience building Warp Factories: 1. Cloud infrastructure to centralize agent conversations, skills, and security permissions 2. Configuration as code for humans and agents to manage that cloud infrastructure 3. Metrics for humans and agents to track efficiency and code quality, beyond “number of PRs shipped” 4. Self-improvement loops for agents to review past conversations, score the quality, and propose improvements to the skills driving the factory You’ll walk away with a clear understanding of how to improve coding agent ROI, and how to deploy a system of your own using Warp Factories. This session will include both slides and a live demonstration of: - How to run tasks through a software factory - How "self-improvement" works, watching agents review past conversations to suggest improvements to skills in a code repository This will include info about Warp Factories, but is meant to be useful to any team learning about software factories or building their own.

What's an eval? Why are they important? How does Arize help you do them? Head on over to the Arize AI booth to get the lightning version of our introduction to the art and science of evals.

Coding agents don't know your architecture, your team's standards, or the issues SonarQube already flagged in the files they're editing. So they guess, and those guesses come back as failed quality gates and PRs sent back for rework. The faster your agents run, the more of this piles up. The Agent Centric Development Cycle builds verification into every stage of agentic development, starting inside the agent's own session. This session shows how Sonar Vortex puts that into practice: it gives the agent your project's rules and a map of the codebase before it writes any code, then checks each change against the same SonarQube standards your CI runs. The agent fixes its own mistakes before they reach a PR. You'll see the full loop running and how little setup it takes.

100x differences exist between the most intelligence models, Opus 5.5 and GPT 6 Astra, and still very capable but cheaper models. This session focuses on navigating those decisions to optimize your use of AI for cost efficiency (and other considerations) with Artificial Analysis and custom benchmarks.

Built to Scale: Future-Proofing Your Technology Roadmap, In Conversation with John Stecher, CTO of Blackstone and Scott Worland, CTO of Norm Ai.

Moving an AI agent from prototype to production is difficult anywhere. In regulated environments, the harder problem is proving that it is reliable enough to put in front of real customers. This session looks at what that validation process requires in practice. We’ll explore why a single accuracy score is not enough to characterize agent quality, how human evaluation can expose ambiguity in what “good” even means, and what happens when the LLM-as-judge used to automate evaluation becomes another system that needs to be validated. We’ll also look at how evaluation moves from an offline exercise to production infrastructure: combining continuous scoring, regression gates, observability, and staged rollout to identify failures before they reach customers. The session shares practical lessons for teams deploying agents in high-stakes environments, including how to define multidimensional quality, evaluate the evaluators themselves, and build the evidence needed to move from prototype to production with confidence.

The industry is rushing to integrate agents into every surface of every product and service. To do so sustainably, keeping cost, quality, performance, and reliability in check, most organizations have turned to posttraining and deploying open-source models. Reinforcement learning is the most critical component of the posttraining stack. At Modal, we've needed to support our customers in scaling up agent RL across our multi-cloud, multi-region fleet. We'll present Stitch, our open-source framework for disaggregating RL rollouts across clouds, regions, and heterogenous GPU types, as well as a stable recipe for posttraining agentic models on off-policy rollout data.

Six months ago, every AI roadmap conversation was about headcount. That conversation is over. In a recent closed-door session with engineering leaders from some of the world's largest financial institutions, not one mentioned reducing headcount. They talked about security exposure and a code review bottleneck that's quietly eating the productivity AI was supposed to deliver. Developers are shipping more code than ever, but they aren't shipping it faster, because a human still has to review and merge every line, and reviewers are drowning. Meanwhile, security has become the boardroom-level risk AI created, and SRE and platform engineers, the people who can actually contain it, are the hardest hires on the market, and the hardest to keep. Drawing on direct conversations with Wall Street's own engineering leadership, this talk makes the case that SRE and Platform Engineering aren't AI's back office anymore. They're the control plane that determines whether AI-generated code will ever be trustworthy enough to ship at scale.

A model hallucination should never become a production action. At Wise, that action could move money. A lot of money. We move hundreds of billions across borders every year, with billions moving almost instantly. How do you use AI at that scale and speed without giving a probabilistic model authority over customer funds? Our answer is to separate reasoning from authority. Models can propose, but only we can decide to act. In this talk, I will explain how we built the system that enforces that distinction. AI governance traditionally relies on human reviews and pull requests. Wise is taking a different approach. LOOP, our Lifecycle Onboarding and Oversight framework, turns human decisions into automated, versioned policies that the platform can enforce. Repeatable decisions become preventative controls, while people remain involved when judgement is required. I will show the technical architecture, its trust boundaries, where controls are enforced, and how this approach helps teams adopt AI quickly without creating an approval queue for every new use case.

A frontier model built a dilution model from a 10-K, and the output looked perfect. Asked where the share counts came from, it said “Honestly, I estimated those.” Anyone shipping AI at a bank or fund has a version of that story. There’s the chat assistant nobody trusts with a number, the retrieval system that read 20 files when the answer was in file 4,312, and the 40-step agent whose mistake at step 3 nobody caught. Kepler read 500 reviews of finance AI tools and found that bankers rated none of the best model’s work client-ready, with complaints falling into four buckets (accuracy, sources, context and coverage) that no model upgrade has moved. Vinoo Ganesh, CEO of Kepler and formerly of Palantir and Citadel, lays out what each one needs built underneath it. Retrieval and math run in code so the model never writes a number, provenance is kept from the first step, an ontology resolves the firm’s own entities and definitions, and every document is processed before anyone asks.

Join Adit Abraham, CEO and Co-Founder of Reducto, and Raj Bakhru, Global Chief Technology Officer of Datasite, for a candid conversation about enterprise AI transformation, build-versus-buy decisions, and scaling AI in the document-heavy world of M&A. Drawing on Datasite’s work with Reducto, they’ll discuss how to decide what to build in-house, how to evaluate document infrastructure, and what it takes to put it into production. Attendees will leave with practical lessons on evaluating accuracy, citations, latency, and security; weighing build-versus-buy tradeoffs; and focusing engineering effort on the capabilities that differentiate a product.

AI agents are getting database access faster than most teams are getting comfortable with it. MCP makes it trivially easy to hand an agent a live connection to your data - but that access needs to respect least privilege and your existing corporate identity and roles, not a parallel permission model invented just for AI. And the database itself has to hold the line too: row-level security and deep data security controls that enforce policy at the data tier, so a bad prompt or compromised session can’t walk out with more than it should. Here’s the part most people miss: the database can also be where the AI runs. With native vector search sitting alongside your transactional data, there’s no need to copy sensitive data into a separate vector store with its own security model to babysit. I’ll show what we’ve built at Oracle to make this real - vector search governed by the same access controls as everything else - and close with fintech and banking examples of agents doing real, sensitive work safely, because security and AI capability live in the same place.

Agents write the code. You skim the diff. You type “LGTM.” Is that engineering judgment, or could a fruit fly do your part? I gave a simulated fruit-fly brain a vote on code changes. It has zero code-review training. This is not a talk about why you should trust it. It’s a talk about what makes anyone’s approval worth trusting, including yours. Using this deliberately absurd reviewer as a starting point, we’ll explore what code review should become when humans aren’t writing every line. We’ll use targeted tests and verification loops to demand evidence from agents, not just assurances. Then we’ll tackle what passing tests can’t answer: why this change exists, why this approach was chosen over alternatives, and what a human still needs to understand before approving it. The goal isn’t less human judgment. It’s better use of it: reviewing intent, decisions, and evidence, with deeper code inspection where it matters.

We just found out our orchestration agent was wasting hundreds of tool calls this past month. The reason? One bad instruction in our message passing skill. Peeling back the curtain, there are dozens more examples you probably have in your own agent skills too. As you scale these agents to longer-running workloads, those gaps add up to real token cost. To solve it, you'll need a feedback loop for agentic self-improvement. I'll show you how to build one: a system that evaluates past agent conversations, finds failure points, and turns those findings into skill improvements you'd want to merge. We'll set up the evaluation loop from scratch, and show how to automate it as part of a software factory. You'll walk away with the skills you need (pun intended) to close the loop and let your agentic system improve itself.

We build an always-on quantitative AI coworker using Gemini and Managed Agents as our infrastructure. From authoring custom skills that parse institutional stock holdings and earnings deflections, to configuring event stream connectors and background triggers that wake the agent the second new financial documents drop. Living continuously inside your internal channels, your virtual teammate operates independently across market cycles: auditing portfolio CSVs in real time, retaining cross-conversation memory, and messaging trading desks when market conviction scores shift. You'll leave with a ready-to-deploy engineering blueprint for running self-directed financial agents under strict corporate access controls.

Financial institutions already know how to govern credit models. LLM apps change too fast for that playbook. In 18 minutes we’ll walk a practical pattern used with a top global bank: golden datasets, domain evaluators, a PASS/FAIL deployment gate, and reviewable evidence in Langfuse — so model risk gets traces and scores, not screenshots.

To do real work, an agent needs its own computer where it can research, write files, and execute code. Giving agents their own sandbox means you can stop babysitting your agents and let them run and verify their own work in a secure environment. This used to be difficult to set up or do on your own but now it is a single API call away. Managed agents give you a cloud sandbox, a file system, and a full agent loop in a single API call. Build an agent for your daily tasks or build one into your product that you can scale to thousands of customers. Let’s put it to the test by building a full team of financial agents in just a few minutes!

Every company is racing to get value out of agents. Few have built the infrastructure to do it safely. Regulated industries will be under increasing scrutiny. Only 44% of organisations running AI agents have policies in place to secure them. Does your enterprise have sufficient controls? This talk covers what you need to build.

This session presents a practical hill-climbing process for making agents more reliable and economical across browser, document, and spreadsheet workflows. We’ll draw on a competitor-pricing pilot with Domino’s, where nine agent versions saw manually assessed accuracy increase from 41% to 91%, while average cost per source fell from $2.11 to $1.16. We’ll examine the operational lessons behind those iterations: defining what “correct” means, diagnosing failures, refining playbooks, and checking that the final deliverable is actually usable. From there, we’ll work through a repeatable improvement loop: collect failures, build representative evaluation cases, change instructions or tools, and compare versions for improvements and regressions. We’ll also discuss how to account for retries, verification, and human review when assessing an agent’s economics. Attendees will leave with a framework for: - Evaluating business outcomes beyond task completion. - Turning recurring failures into playbook changes and regression tests. - Comparing reliability and cost across agent versions. - Prioritizing improvements when engineering and evaluation budgets are limited. The methods apply to financial and enterprise operations wherever agents must turn messy information into dependable work.

The systems of record lie by omission. Jira, the CRM and the monthly status deck are a sparse, lagging, self-reported projection of what is actually happening inside a company, and every AI product built on top of them inherits that projection. The real state is sitting in the exhaust. Hundreds of Slack channels, 500,000+ messages, 5,217 pasted screenshots and PDFs, two years of executive decks, voice notes dictated in the back of a cab after a client meeting. This is what it took to bootstrap a usable model of an enterprise out of that material, and what broke on the way. I am going to use the phrase "world model" deliberately and then defend it, because the term is doing real work. This is a state estimator rather than a knowledge graph. It holds beliefs about entities over time, updates them from noisy partial observations, tracks how stale each belief is, degrades when a source goes dark, and predicts. Which project is about to slip. Which client is next to trade a name. Which green cell turns red next month. That framing carries a negative claim I want to argue hard, which is that you cannot retrieve your way here. RAG over Slack answers what was said. A world model answers what is true now and what is about to go wrong. Those need different architectures.

Financial agents can create models, analyses, and investment memos, but only if they can find the right facts. In practice, evidence is scattered across filings, dense tables, scanned PDFs, spreadsheets, emails, and multiple revisions. Loading everything into the model’s context is expensive and often makes its answers worse. This talk shows how to separate searching from reasoning by giving the main agent a specialized evidence layer. We’ll explore how a search subagent decomposes questions, combines semantic and exact retrieval, checks sources and versions, and returns compact evidence for analysis. Using OfficeQA and realistic investment banking tasks from APEX-Agents, we’ll show how better retrieval produces more accurate answers with fewer tool calls and lower cost.

You may have heard a lot about graph engineering. The term has made the rounds with some overblown noise and hype, but it does point to a collection of useful design patterns that solve real AI engineering problems worth delving into. We cover the practical fundamentals in this session: what graph engineering is, its core benefits around performance, efficiency, and governance, and how to actually apply it in your day-to-day work. We bring a unique perspective as well, shaped by making AI work in regulated industries like financial services. Instead of treating graph engineering as part of the agent runtime, we approach it as standards around memory and context: creating APIs and artifacts portable to any agent framework. In other words, a knowledge layer. That means you can move faster while maintaining trust: keeping your existing agent infrastructure and a simpler stack, while taking on new, complex agentic workflows as technology changes and your products and services evolve. You will leave with worked examples, libraries, and reference architectures for putting graph engineering to work in your own projects.

Stop by and hang out with the PlanetScale team for an open Q&A on Postgres, sharding, and whatever else you want to dig into. We'll have plenty of merch to give away, so swing through before you head home!

This presentation provides a technical roadmap for building large-scale automatic evaluations for subjective tasks. Specific concepts that will be covered include setting up and coordinating human annotation efforts, building and evaluation LLM judges, using production monitoring to inform model evaluations, and reducing evaluation uncertainty via statistics. By combining these components and monitoring them properly, we can create a scalable evaluation framework that evolves and improves over time as our understanding of a domain deepens.

An agent loop needs a computer, but it shouldn't need to know whether that computer is an EC2 instance, a Kubernetes pod, a micro-VM, or a third-party sandbox. We built OpenCompute to separate the two - one small interface that owns the full compute lifecycle, with each provider implemented as a driver behind it. This talk covers why we built this and how an outbound-only connection lets it run inside locked-down customer owned infrastructure.

Every financial decision gets its own model: credit risk, fraud, forecasting, claims, pricing. Each one needs a feature pipeline, a tuning budget, and a retrain when the data shifts. In practice that puts model development weeks behind the decisions it feeds. A tabular foundation model predicts on raw structured data in a single forward pass, in under a second, with no feature engineering and no tuning. It beats gradient-boosted trees out of the box. Sauraj Gambhir, Co-founder and Co-CEO of Prior Labs, shows how teams at Teza, Creditplus and Taktile go from a raw table to best-in-class predictions in seconds, and how to do the same on your own trading, lending and risk data.

As enterprise AI adoption grows, organizations are asking different versions of the same questions: What is our AI usage costing us? Where are those costs coming from? Which applications and models are driving consumption? And how can we use this information to make better decisions? The challenge is that the answers must serve very different audiences. Finance teams need forecasting and cost allocation. Engineering leaders need actionable insights into models, applications, and architecture. Product teams need to understand adoption and usage patterns. Executives need a clear view of how AI resources align with organizational priorities. Creating separate solutions for each audience leads to fragmented data and competing definitions of success. In this session, we’ll share what we’ve learned from enterprise stakeholder conversations and from building a tokenomics product that supports these perspectives through a common foundation. We’ll show how AI usage data can be transformed into relevant insights for each audience while preserving a shared view of consumption, cost, and operational efficiency. We’ll also explore where enterprise tokenomics is headed. Richer telemetry and more automated instrumentation could make insights increasingly timely and actionable, while stronger connections between AI activity and business outcomes could help organizations develop a more complete understanding of AI return on investment. Attendees will leave with a practical view of tokenomics as more than usage reporting or cost control. It can become a shared decision-making layer—helping organizations understand how AI resources are being used today and make more informed choices about where to invest next. Takeaways Attendees will learn: • How a common tokenomics foundation can support finance, engineering, product, operations, and executive stakeholders • How shared data can produce audience-specific insights without creating disconnected tools • Which questions enterprise stakeholders need AI usage data to answer • How visibility into consumption, cost, and efficiency can support better planning and optimization • How richer telemetry, automated instrumentation, and business-outcome data could advance tokenomics toward ROI measurement

Give an agent 40 tools and it gets worse at using any of them. Give it a sandbox and a way to write code and it can compose whatever it needs. Recent research has bash-only agents beating typed tool catalogs by 21.8 to 24.5 points on the same benchmark while spending fewer tokens, and the industry is quietly moving in that direction. The shift is really one of abstraction. We used to obsess over prompts; now we optimize skills. We used to design individual tools; now we ship MCP servers and plugins. We've stopped handing agents primitive verbs and started handing them programming. This talk makes the case for code mode while staying realistic about what it costs, and closes with how our open-source observability stack exposes agent traces to coding agents through read-only SQL and code mode, and how we use our own evals and tooling to improve agent outcomes.

Rowspace customer co-presenting keynote

When an agent takes on hours of research, data extraction, tool use, and judgment, it may make hundreds of decisions before producing a final result. That result can be correct even when the steps behind it are unreliable. Evaluating long-horizon work requires evidence about both the result and the decisions that produced it. Basis builds AI agents for accountants and encounters this problem across complex accounting work. An agent may reach the correct tax conclusion while relying on pretraining or a blog instead of checking the tax code and regulations. An outcome eval marks the answer correct, even though the research would not meet the standard an accountant expects. Basis developed behavior specs to evaluate those decisions across the full trajectory. Each spec defines how an agent should behave when a specific situation occurs. A judge reviews the trajectory and determines whether that situation arose and whether the agent followed the expected behavior. Basis partnered with Braintrust to publish the approach as an open standard for teams building long-horizon agents. In this session, Mitch Troyanovsky will explain why outcome evals become expensive and slow to iterate on for long-horizon work, how Basis chooses the few behaviors worth measuring, and how behavior failures guide changes to prompts, tools, or runtime context. He will also discuss how teams can evaluate behavior across production traffic without turning the spec into an unmanageable list of every rule an agent might follow.

AI models have spent years learning to generate text, images, and video. The next frontier of AI development focuses on models that understand space, motion, and the physical dynamics of the real world. Cristóbal Valenzuela, Co-CEO and Co-Founder of Runway, will discuss world models mature enough to simulate real environments in real time, and what this progress opens up for robotics, digital interfaces, and broader scientific discovery.