From APIs to answers: let the agent cook

Why one-tool-per-API breaks in production

Rolling your own harness with one tool per SaaS integration handles toy prompts like fetching your latest GitHub issue on day one. Under real workloads, agents stall across three failure modes: picking between similar tool descriptions (like Jira versus Linear), looping on pagination and API rate limits, and tripping over cross-system IDs—searching a CRM by email while Stripe keys on customer ID:

“The agent starts picking the wrong tools sometimes. So it gets confused between tools that have similar descriptions.”00:01:32

At his earlier data-agent company, connecting agents straight to OLAP cubes like Snowflake and Databricks exposed the sharpest trap: without semantic guardrails, the agent inspects the very first table it finds and asserts a wrong answer with total certainty:

“You can either like look at the first table and stop. And declare that this is the ultimate source of truth, and give you the wrong answer pretty confidently.”00:01:32

Evidence: transcript 00:00:32, 00:01:32

Organize tools by use case—or let the model write code

Following Anthropic’s guidance on writing effective tools, push deterministic plumbing—pagination, retries, parameter mismatches, and cross-source joins—out of the model’s reasoning loop and into code. Replace separate CRM, database, GitHub, and Linear wrappers with semantic use-case tools like find_user() and find_pending_tasks():

“There are deterministic things that should still be in code, and your agent would benefit from something like that.”00:03:01

“The first key is not categorize the tools, but actually combine the tools into some sort of semantic meaning.”00:12:07

At Sentry, Bashir’s team pushes the idea further with programmatic tooling—letting the model write and execute code against APIs directly rather than maintaining dozens of hand-written wrappers (collapsing in Sentry’s setup toward two tools: write code and run):

“We don't even write our tools. We do something called programmatic tooling, which allows our model to kind of write tools.”00:03:01

Evidence: transcript 00:03:01, 00:12:07

Pre-optimizing the harness gives the agent tunnel vision

Early on, Bashir’s team worried the model would fumble table names and joins, so they built an elaborate pre-processing pipeline: classify intent, pre-fetch schemas, and steer the prompt before the model saw anything. Instead of helping, those heuristics blinded the agent and broke with every frontier model release:

“The first and obvious one was if this optimization kind of became a blind spot for the model. And it's also not very resilient to new models coming out.”00:05:18

Stripping out those “training wheels” and letting the agent explore with clean tools paid off immediately—mirroring Vercel deleting 80% of its tool code:

“The agent only got four times faster, but also used 30 percent. Fewer tokens across all of our kind of user queries”00:08:07

“That as models get better, you might find your harness kind of compensating for a model that you use. Six months ago, that's a common like pitfall”00:08:07

Evidence: transcript 00:05:18, 00:08:07

What deterministic guardrails to keep in the harness

Letting the model choose its investigation path does not mean putting security inside the prompt. Bashir draws a clean boundary between cognitive routing (delete) and hard product invariants (keep deterministic and thin):

“Things like handling permissions, execution limits. Stripping out personally identifiable information, etc. These things do make sense to do outside of standard models.”00:05:18

“On the other hand, you also want to have deterministic systems that are like libraries that kind of encapsulate. The absolute requirements that you have, for example, user A should not be able to. Kind of access into these tools. That stuff should be deterministic. It should be outright, but that layer should be super thin, as thin as possible”00:12:07

Evidence: transcript 00:05:18, 00:12:07

Cheap deterministic checks build user trust

When you stop micromanaging how the agent investigates, you replace step-by-step guidance with fast, cheap validation at the output boundary. In a data agent, that means BigQuery dry runs, dbt metric definition checks, and automated join/count/freshness checks; in a coding agent, it means linters and unit tests feeding exact errors back into the loop:

“You have some sort of cheap, deterministic, fast way to validate results.”00:09:45

“We're getting better results, which means the agent writes better SQL queries, writes better Python code. But also, you know, give confidence to the user.”00:09:45

“So in summary, these are the three things we learned: automate what you know. But the second one is to let the agent explore and find out things you don't know. And finally, try to allow your agent to check its work whenever possible.”00:11:40

• Combine overlapping APIs into semantic tools: own pagination, retries, and ID stitching in code, or use programmatic code-writing tools (00:03:01, 00:12:07).
• Delete tunnel-vision heuristics: stripping intent routers and schema pre-fetchers made Sentry’s agent 4× faster and 30% cheaper in tokens (00:05:18, 00:08:07).
• Keep a thin deterministic perimeter + cheap validators: enforce permissions, limits, and PII scrubbing outside the model, and validate outputs with dry runs, dbt checks, or tests (00:05:18, 00:09:45).

Evidence: transcript 00:09:45, 00:11:40, 00:12:07