When the bottleneck moves · Natalie Serrino

The wait

Waiting breaks flow

She opens with her own agent making a slide look better in Figma. It did good work, in about two minutes, after one request. Faster than she could have done it, but long enough that she'd moved on to something else.

The web went through this already. In 2000, someone could have looked at dial-up and said it was fast enough. The large companies of that era found otherwise: a hundred milliseconds of added delay had a measurable effect on the bottom line, and users didn't fully come back even after the delay was removed.

“Even very small delays cost the users.”00:02:57

The psychology is about flow. A tenth of a second feels instant; at one second you keep your train of thought; at ten seconds you pick up your phone or switch to another agent.

“From data from June, the median coding agent turn is 63 seconds.”00:02:57

From the notes: “i want to see how much faster these agents can be, like do a research on fast token inference”. Still open; her yardsticks (0.1 s, 1 s, 10 s against a 63-second turn) and Gimlet's posts on the links page are the starting point.

Evidence: transcript 00:01:02, 00:02:57

The trade-off

More tokens or faster tokens

The catch is compute scarcity. Infrastructure is asked for two things at once, and they pull against each other.

“More tokens and faster tokens. Unfortunately, these goals are fundamentally at odds because faster tokens require more energy to produce than slower tokens.”00:06:01

Her Token Mint game makes it concrete: you run a token factory, and the strategy for one fast token is completely different from the strategy for the most tokens in a minute. Plotting real runs from her colleagues gives a Pareto frontier.

“If we have faster tokens, we don't have as many. And if we have more tokens, they're not as fast.”00:06:33

Upgrades in the game push the frontier up and to the right. Gimlet's real-world upgrade is matching the chip to the job: splitting a model into pieces and running each on a different kind of accelerator, GPUs alongside SRAM-centric chips. Even with the network overhead, that gives much faster tokens for the same power. Her Snake demo shows a coding agent finishing on the split setup while the conventional one is still running.

From the notes: “what really is the pareto frontier? does that mean the curve of price/performance tradeoff?” Yes, in this sense: the curve of best trade-offs between how many tokens a data center makes and how fast each user gets them. Every point on it is one you can't improve on one axis without losing on the other. Better hardware matching doesn't move you along the curve; it moves the curve. The game is at gimletlabs.ai/token-mint-game (laptop only).

Evidence: transcript 00:06:01, 00:06:33, 00:08:06, 00:09:43

The shift

The bottleneck moves

People ask whether inference is even the bottleneck, or the tool calls are. On her slide-editing run, inference took about 80 percent of the time and tool calls 18 percent. Simulate 10x faster inference and the tool calls become 67 percent.

“That's why it's important that we have to constantly attack the bottlenecks of the application and get the low-hanging fruit where we can.”00:10:37

Nobody has optimized tool calls much, because today they aren't the bottleneck. Once inference is fast, they will be the reason everyone waits. This kind of optimization is verifiable (known inputs, known outputs), so agents can do it; Gimlet is building a self-optimization loop from data-center telemetry up to the model.

From the notes: “the bottleneck will change when we increase inference speed”. That's the line her slide makes: speeding up one part of a system exposes the next slowest part, so the work is systematic, not one fix.

Evidence: transcript 00:10:37, 00:11:38

The demo

Agents on the critical path

Her security demo is Dog Club, a site for dog lovers, with a real vulnerability in a Node dependency that lets an upload overwrite files it shouldn't. A security agent watches live traffic. It doesn't know this vulnerability, but it knows its job.

“And the defender is monitoring, but it's not in the critical path because it's not fast enough.”00:12:18

So it watches asynchronously. When a malicious “dog” is uploaded, the defender works out that the upload is doing something it shouldn't, but too late: the attacker has already taken over the site. Rerun the same upload with ultra-fast inference and the defender sits inline and blocks the request. We think of agents as asynchronous; fast enough, they can work online, in the request path.

From the notes: “she's saying that we could use inference on the critical path if things are really fast”, and the demo where the attacker turned the dog website into a cat website. The reading: speed changes the architecture, not just the wait. An agent that answers in milliseconds can be a gate; one that answers in a minute can only be a reviewer.

Evidence: transcript 00:12:18

Forward

What faster unlocks

Faster internet took us from webmail at dial-up speed to on-demand streaming. She asks the same of inference, which runs in the low hundreds of tokens per second today.

“What does it look like when we get to a million tokens per second?”00:15:11

Her list: security that runs online, telling an agent what to build and watching it appear, tap-complete for everything, many more agents, even robots. Requests that fail today will start working as latency drops, the same way they start working after a model release.

The reading: this pairs with the decision-model talks (Cloudflare and the Jev talks). They make a decision fast by making the model small; she makes it fast by making inference fast. Either way, the agent moves from reviewing afterwards to deciding inline. For a client: ask which checks you run after the fact only because the model was too slow to run them in the path.

Evidence: transcript 00:15:11