Too good to be true, too bad to be useful

The puzzle

Two cults and a jagged frontier

He opens with his own odd position: he loves chatbots and coding agents, and still thinks the field has gone wrong.

“One of the few people who hates on ChatGPT, who made it.”00:00:37

Then the two cults. One reads the benchmarks, where models have beaten humans for two years:

“We've been announcing AGI three months away for two years now.”00:01:26

The other looks outside San Francisco, at a world that has barely changed. Both are looking at the same facts.

“It's a rare. Point of view to actually think AI is cool and transformative, and also like underwhelming the world right now.”00:01:26

That's the jagged frontier: models that can work on Millennium Prize problems but can't be trusted with bookkeeping. (The transcript garbles his example: “They can't even be used to touch a counting.”) He wants an Occam's razor explanation, and the rest of the talk is his.

Evidence: transcript 00:00:37, 00:01:26, 00:02:29

The spine

What explains the line?

His slide splits the field. Too good to be true: ChatGPT, copilots, coding agents, deep research, voice agents, customer service without decisions, and slop. Too bad to be useful: customer service, drive-thrus, accounting, QA, insurance, data entry, anything requiring reliability. His answer:

“Everything on the left side is intrinsically a human-in-the-loop task.”00:03:24

Deep research writes better than he could. But if the answer had to be correct, not well written, you'd pick him. Pleasing a person and doing the task are different goals:

“And my claim is actually for these tests pleasing a human. Is nothing to do with doing the task.”00:03:24

The cause is the method he co-created.

“I get the irony. That you know, I co-created RLHF, and I do believe RLHF is part of the problem.”00:04:15

“That is what the HF is in RLHF.”00:04:15

The human feedback is the human in the loop, built in. Unless a task is trivially verifiable, a person still checks the output afterwards. Never pick him over ChatGPT for a ChatGPT question; but for background work that has to be right on its own,

“You should not trust LMs at all.”00:04:15

His “WTF is going on in AI?” slide lists it: LLMs are too good to be true at assistance; 0% are optimized for automation (“RL??”); they're too bad to be useful at automation; his guess is that automation is worth more than assistance; then “???” and “Profit”.

From the notes: “where is the automation”, “why are some too good to be true and the others are too bad, what explains this line”, and “all ai today does require human in the loop, rlhf is the reason”. His answer: the left column is work a person checks anyway, so a model trained to please that person shines; the right column is work nobody checks in the moment, so it needs reliability the training never asked for. The reading: sort a client's AI ideas into his two columns. Anything on the right won't be fixed by a better chatbot; it needs a narrower model call trusted without a reviewer, or a person kept in the loop on purpose.

Evidence: transcript 00:03:24, 00:04:15

The cause

The bitterest lesson: doing the right task

He extends Sutton's bitter lesson. Compute beats clever algorithms; data beats compute; and underneath both:

“I would argue that the bitterest lesson is doing the right task.”00:06:05

In classical RL, the game supplies the task:

“So actually, when you're doing RL, you don't even need to think about the right task.”00:06:05

Language models were different. The field had to find next-token prediction, which mattered not for its maths but for what you could do with it afterwards: generate text. Then instruction following: before RLHF, the data for it didn't exist, and with simple algorithms a small model trained on it beat far larger pretrained ones. The transcript garbles the model sizes he gave, so they're left out.

From the notes: “when you are doing RLHF you don't even need to think about doing the right task”. He says it of RL on games, where the rules are the task. The catch is that RLHF chose a task too, pleasing a rater, and that's the one models got good at. The reading: the garbled sizes are most likely the InstructGPT result, where a much smaller instruction-tuned model was preferred to GPT-3.

Evidence: transcript 00:06:05, 00:08:04

The fix

Reliable intelligence

So TypeSafe picked a different task.

“You get what you optimize for.”00:08:57

“I often call it you get Einstein ninety five percent of the time, but Mr Bean five.”00:08:57

“If you can't tell which is which, how the fuck is it automated?”00:08:57

Instead of occasional brilliance that someone then has to check, aim for narrow, dependable smarts:

“What if you focused on extreme, extreme narrow focused reliability?”00:08:57

It doesn't show up in benchmarks, and he credits it, not speed, cost or interface, for how developers took to Jev. He isn't sure of the name himself:

“I'm looking for a better word than reliable.”00:08:57

From the notes: “reliable intelligence is the phrase he's using, but it's not really the right one, or what do we deeply mean by this”. He's asking too. What the talk means is intelligence you don't have to check: the Mr Bean five percent is the problem, because one unpredictable miss puts a reviewer on every answer. Possible better words: “dependable judgement”, or “unsupervised-grade”.

Evidence: transcript 00:08:57

What it unlocks

Smart software: a fourth logic gate

Not more chatbots. Software has barely changed in a decade except that it's cheaper to write, and the best AI has done inside it is

“Is having a chatbot to the side that sometimes interacts with the software, but cannot take expensive decisions for you.”00:10:25

“And I think the problem is we're focusing too much on coding agents.”00:10:25

The intelligence belongs in the program's inner loop. His slide draws it: traditional software is a flowchart of fixed rules, with one branch that needs a human; just-in-time software is a long chain of LLM steps ending in a question mark; smart software is the flowchart again, with small model decisions at a few nodes, and every branch pays.

“I think the narrative of SaaS apocalypse is obviously not going to happen.”00:10:25

His analogy is the blue LED: red and green made indicator lights; adding blue gave white light and displays. Software has been built from greater than, less than and equal.

“This is kind of conceptually adding a fourth semantic logic gate.”00:12:12

“Like, do we still need regexes with like crazy things, or can we just say like we want a Gmail email address or a real looking one?”00:12:12

A later slide, “Infinitely replicable lego blocks”, draws AND, OR and NOT beside three model gates that take data and a question: a yes/no with probabilities, a score, and a choice among options.

From the notes: “smart software”, “we need to build out a better area of how tools fit in with decisions”, and “SaaSapalooza, not SaaSpocalypse”. The three model gates are that map's missing piece: a yes/no, a score and a choice are the shapes a decision takes inside a program, next to tools (which act) and code (which follows rules). Jev is the decision model that kept coming up this week, alongside Cloudflare's Clef and Gemini Decisions, and in Scott Roberts' talk Jev was called inside a SQL statement in DuckDB.

Evidence: transcript 00:10:25, 00:12:12

The call

LLM Stockholm syndrome and software research

His “AIE secret” is that the field isn't pushing hard enough; the slide says “the field isn't jevving hard enough”.

“I think the world still has LLM Stockholm syndrome.”00:11:32

Most ideas just replace a step with an LLM instead of rethinking the software. His alternative:

“I call this software research.”00:14:32

Like the early web, it means building things because we can, and finding out what's possible.

“Should all of those websites existed? Probably not.”00:14:32

“The theme is going to be real software engineering, not just white coding.”00:14:32

(“White coding” is the transcript's vibe coding.) That means decomposition, stages that can be verified, real encapsulation, and state you design, not

“Instead of assuming everything is global state within a KV cache”00:14:32

The reading: the gains come from engineering around the model, not a bigger one. It's the same advice as Yegge's two layers and Cognition's “teach the agents how to test the codebase”. And it answers your Cognition note, “most teams are limited by the amount of ambition they have”: software research is a call for more of it.

Evidence: transcript 00:11:32, 00:14:32

The ideas

Ideas worth building

Agents beyond the KV cache. Designs that fight the cache get slow and expensive, so people abandon them.

“They're so inflexible in design because of the tyranny of the KB cache.”00:15:50

Why can't every call include all your past coding-agent history? The memory exists; what's missing is the smarts to search and summarize it. And why can't agents working at once share what they learn?

“Why do they need to rediscover the same patterns of the codebase?”00:15:50

Principal-agent problems. Not coding agents: anyone acting for someone else whose interests differ, such as elected officials acting for citizens. A fair society has to stress the letter of a rule, which breeds loopholes.

“And I think the root of that problem is we can't encode the spirit.”00:17:20

Real-time security, semantic types, new interfaces. Spotting a bad action is easier than doing it; what's missing is checks fast, cheap and reliable enough to run live.

“I would argue it's a lot easier to figure out they're doing a naughty thing. Than to actually do the naughty thing”00:18:11

He wants a type system that checks meaning, with a model in the linter, and interfaces that do what you mean.

From the notes: “read the tyranny of the kv cache”: it's his phrase in the talk, not a published piece. “What is a principal agent problem”: answered above. “What is the DWIM twitter demo”: DWIM is “do what I mean”, an old idea that failed on too many hard-coded rules; he ties it to “a great Twitter video” of a voice interface but doesn't name it, so it stays open. The reading: his memory idea is the opposite of Yegge's warning the same afternoon. Yegge prunes agent memory because it becomes a garbage dump; Diogo would keep all of it and get smart enough to search it. Both say the missing piece is judgement about what matters, which is a decision model's job.

Evidence: transcript 00:15:50, 00:17:20, 00:18:11

The close

The root of slop, and what couldn't we automate?

Because AI is unreliable, no one uses it for decisions that cost them anything. Pushing the cost onto someone else is another matter:

“But it's actually totally fine to push the costs into other people.”00:19:45

That's slop: an externalized cost, and an advantage for people who don't care. Then he turns playful: even “cursed software”, deliberately terrible interfaces, is worth making.

“Doing things because we can is what software engineering is about.”00:20:25

His closing question: every engineer knows what software can't do. What if those limits went away in six months or a year?

“What couldn't we automate?”00:21:07

From the notes: “slop is pushing the work to other people”. The same diagnosis as Lauren Tan's talk: unchecked output moves the checking onto whoever receives it. Diogo puts the cause in the model: make it reliable and there's less checking to push. The reading: it closes his opening question. Reliability draws the line between assistance and automation; move it, and the question stops being whether something can be automated and becomes how much work it is.

Evidence: transcript 00:19:45, 00:20:25, 00:21:07