Wed, Nov 11 · 3:06 PM–3:19 PM · Continental 4–6 (Mainstage)

Your Judge Is Hallucinating Too

Brian Kennedy, ZoomInfo

Don't trust an LLM judge until you've measured it: run a second check in shadow mode and look where the two disagree.

← All talks · ← Prev talk · Next talk →5 of 7 built · Z4 Evidence Hub
Level 4 · Raw Evidence & DossierWant the visual narrative with stage slides?
Recording · 0 minTranscript · qwen3-asr-1.7bLinksPhotosNotesStory · draftTalk page
ThemesTheme 01: Own the verification loop →#Themes presentation →

Speaker

Brian Kennedy ZoomInfo
no X handle foundLinkedIn

My notes

None captured.

Photos

None filed.

Recording

My Voice Memos from the room. The public livestream: YouTube, around 6:45:53 (approximate).

2026-11-11_1522_brian-kennedy · 0 min · qwen3-asr-1.7b

Transcript

Machine transcript (qwen3-asr-1.7b). Garbled words are kept as heard.

Read the transcript

A 28-second fragment: the last lines of the talk only, recorded 15:22:39–15:23:07, after the slot's scheduled end (15:06–15:19); the talk ran over. Filed on the operator's attribution, not on a name in the audio: the folder title names the speaker, and the closing lines answer the abstract (a judge measured against a second check run in shadow mode). Neither transcript names the talk, the speaker or his company.

Transcribed locally with Qwen3-ASR 1.7B (timings per ~5 s segment), prompted with the speaker names and affiliations of overlapping sessions. Qwen's text ends at 00:00:18; the closing "Thank you." is only in the Voice Memos transcript. The Voice Memos transcript is kept beside it as transcript.txt.

00:00:00 — Closing

Then read the cases, see where the two disagree, and don't overthink it. Just start small. We entrusted our judge until we measured it. Measure yours. Run a second check. Run in shadow mode, seeing where it disagrees. You may be surprised at what you find.

Generated Nov 13, 2026 at 5:56 AM · bun scripts/render-pages.ts