Your immune system doesn't file a Jira ticket

Regular evals only grade what you already thought to test

Tonse starts with the standard observability baseline: instrument sessions, traces, and spans with OpenTelemetry, attach evaluators for hallucinations, tool use, and context adherence, and wire guardrails to steer bad actions before they hit users:

“And then, of course, what's the point of observing if you can't do anything about it?”00:02:08

Galileo’s trajectory from its 2023 ChainPoll work pushed toward compact, low-latency evaluation models rather than expensive LLM-as-judge prompts. Yet when Mohammad takes over, he points out the structural gap in every static eval suite—you only test what you anticipated, and searching individual trace IDs collapses when user inputs drift, models change, and adversaries probe production:

“And why just the regular eval loop is not enough? Regular evaluation is just not enough.”00:06:02

Evidence: transcript 00:02:08, 00:03:19, 00:04:28, 00:06:02

Your immune system doesn't file a Jira ticket

Instead of dumping alerts into another ticketing queue, the team modeled Signals on biological immunity. Your body does not wait for someone to write a regression test before reacting to a new pathogen—it detects the anomaly, treats the root cause, and remembers it afterward:

“Pathogen entering, it's not waiting for some test. Is trying to understand the problem, trying to respond in terms of getting to the root cause of it, and then he tries to remember. So that going forward, if you have a similar situation, it makes sure it prevents you from falling sick again.”00:07:27

Evidence: transcript 00:07:27

Clear diagnosis and prognosis across every session

Signals 1.0 surfaced unknown failures for enterprise buyers, who immediately asked for four things to turn discovery into healing: run continuously on a daily or weekly schedule; scan every session instead of small samples; name the exact failure category; and prescribe what to change:

“Look at all the sessions. And then, can you be a little more opinionated about them? Like, tell me. What kind of failure is it?”00:08:50

That feedback shipped as Signals 2.0—built on an opinionated taxonomy of 45 problem types across 8 families (arXiv:2611.04507) paired with actionable fixes across prompts, model selection, guardrails, and new evals:

“Just because an issue has emerged, it's not enough.”00:11:22

Evidence: transcript 00:08:50, 00:10:14, 00:10:56, 00:11:22

Closing the loop from trace to fix verification

Walking through an insurance workflow and a live payment-agent demo, Mohammad shows Signals flagging a systemic failure occurring in one out of ten sessions reaching a payment step—the agent transferred money to the wrong account, detected the mismatch only afterward, and could not reverse it:

“Agent used that account and send the money, and afterwards it checked. There was a problem, but it couldn't make the money back.”00:13:42

Diagnosis links straight into remediation (integrating with Codex in their demo to locate and apply the code change), followed by the step most teams skip—verifying in live telemetry that the failure rate stays down:

“Once you fix it, you want to verify that it's. Indeed, and as a trend, the number of people fix so we can. Keep going in production, and make sure that it doesn't recur.”00:12:42

Evidence: transcript 00:12:42, 00:13:42, 00:16:29

Every percentage point matters across millions of sessions

Customers often ask why they cannot paste trace dumps into a frontier LLM prompt to generate the same recommendations. At half a million sessions per day, token costs explode—and benchmarks against generic prompting show double-digit F1 gaps that tuned small models and structured pipelines close:

“The reason we talk about this accuracy is that for enterprise customers, even one percent data point accuracy. Makes a big difference, especially when you have like millions of sessions.”00:17:23

Because catching silent failures before customers encounter them determines whether enterprises trust autonomous multi-agent systems in production, buyers keep pushing the precision bar higher:

“One of the enterprises was like, hey, for me, 92% accuracy is not good enough. I wanted to go to like 95, 98.”00:19:48

• Grading is only half the loop: pre-written evals miss user drift, model changes, and adversarial edge cases that continuous full-session sweeps catch (00:06:02, 00:08:50).
• Opinionated taxonomy + remediation: 45 failure types in 8 families map directly to code fixes, prompt edits, guardrails, or evals, then verify recurrence drops (00:10:56, 00:12:42).
• Specialized detection wins at scale: across 500k+ daily sessions, dedicated small models beat ad-hoc LLM prompts on both token cost and F1 accuracy (00:17:23, 00:18:24, 00:19:48).

Evidence: transcript 00:17:23, 00:18:24, 00:19:48