Don't trust an unmeasured judge
The recorder caught only the final 28 seconds of this session (15:22:39–15:23:07), so the body of the talk is absent from the archive. What survived is the punchline: teams treat an LLM-as-judge as settled infrastructure right up until they actually test it:
“We entrusted our judge until we measured it. Measure yours.”00:00:00
Evidence: transcript 00:00:00
Run a second check in shadow mode
Auditing your evaluator does not require pausing production or swapping out your stack upfront. Run a second check alongside the primary judge in shadow mode and record where the two scores part ways:
“Run a second check. Run in shadow mode, seeing where it disagrees.”00:00:00
Evidence: transcript 00:00:00
Start small and read the disagreements
Skip designing an elaborate benchmark suite before looking at real diffs. Start with a small slice, pull up the cases where the two checks disagreed, and read them directly—because the failures hiding inside a trusted judge rarely look like what you expect:
“Then read the cases, see where the two disagree, and don't overthink it. Just start small.”00:00:00
“You may be surprised at what you find.”00:00:00
- Honest archive note: only the final 28 seconds (46 words in
transcript.md) were recorded; speaker and company attribution follow the folder title and session abstract (links). - Measure the grader: run a second check in shadow mode alongside your live LLM judge instead of treating judge scores as ground truth (
00:00:00). - Inspect diffs directly: start small and read the disagreement cases—surprises show up quickly once two checks are compared side by side (
00:00:00).
- Scott Roberts: Autocomplete for Actions & Diogo on TypeSafe & Jev — Decision-model context referenced in the session dossier.
- Asif Mohammad & Sudhir Tonse: Heal Your Agents — Moving beyond static LLM-as-judge grading to continuous production diagnostics.
Evidence: transcript 00:00:00