我现在（大部分时候）选模型主要看速度，而不是看智能。 | I'm (mostly) picking models on speed now, not intelligence

我现在（大部分时候）选模型主要看速度，而不是看智能。

I'm (mostly) picking models on speed now, not intelligence

📰 martinalderson.com 📅 2026-08-02 00:00 ✓ 已读

For the first time I can remember, I'm not choosing my daily driver models on raw intelligence. I'm choosing them on speed.

Have we reached an intelligence tipping point?

This is probably going to age like spoilt milk, but right now, models around the ~Opus 4.6 level seem to be 'smart enough' for most of my daily tasks - code, pulling together research, designing slide decks and doing analytical tasks against a plethora of databases.

While like most I was hyped to play around with Fable, ironically the US Gov shutdown gave everyone time to get used to Opus again. When Fable came back post-hype with additional guardrails, the first thing I noticed was just how slow it is^[1]. So slow, actually, that I switched back to Opus pretty quickly.

I've spent a lot of my career making software fast. It's remarkable how much better software feels to interact with when it's fast. In my experience (and many studies), you can take the most beautiful product, but if it's slow, you won't enjoy using it. Equally, you can take a very basic product that's super fast and it will feel brilliantly utilitarian. ^[2]

It's clear to me that when only a few, big, slow models cleared the aforementioned (and hypothetical) intelligence bar, it wasn't worth the trade off really to use a slower model. Whatever speed you gain you quickly lose in having to redo it because it was broken.

Is 100tok/s the new 100ms?

I've written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected.

The key fact I remember is that to humans, ~100ms feels 'instant', the gold standard. I reckon 100tok/s output on a model is about as fast as I can keep up with. After that, it comes in faster than I can (skim) read. This isn't an exact bar, because increasingly most of the model time is spent in reasoning, and not actually showing you output tokens. And also it massively varies on your output, prose gets output with far fewer tokens per character than code, so your mileage may (and will) vary.

But roughly, 100-200tok/s to me seems pretty damn fast. Below 50tok/s output feels increasingly slow. Ironically, going past 200tok/s seems almost unnerving - you can try a model out here at 10,000tok/s+ (!). I'm sure this feeling will edge upwards as we get used to it and push our agents to do more complicated work.

Given the plethora of new models that I think are ~clearing the aforementioned bar - such as GLM5.2 and DeepSeek V4 Flash GA - that are open weights and small(er), we now have a wide range of models and speed. If you look at the speed rankings of various providers for GLM5.2 on OpenRouter you can see the enormous range of serving speed - from less than 30tok/s at the bottom to 129tok/s at the top.

This is another huge plus to the open weights ecosystem. While there are great benefits in cost that are obvious, the fact that providers are also incentivised to compete on speed like this is really interesting.^[3]

But there are limitations

If you're familiar with Pareto's Principle and Amdahl's Law you'll know what's coming up.

Assuming "good enough" models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them.

Take an agent using a model processing at 50tok/s. Most of the time is spent waiting for inference to come back. Now run the same turn at 250tok/s and you'll see that increasingly you are bottlenecked on tool calls on your "local" machine and your decision making.

Stacked bar chart comparing one agent turn at 50 tok/s and 250 tok/s. At 50 tok/s the turn takes 65 seconds - 40s inference, 15s tool calls, 10s human review. At 250 tok/s it takes 33 seconds - 8s inference, with the same 15s of tool calls and 10s of review. A 5x faster model gives only a 2x faster agent.

Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn't move.

And even worse, making your local machine faster on these tool calls is sort of stalling out, because hardware costs have gone parabolic because of AI. Yet again another weird derivative effect of the AI market.

So I suspect (for now at least) there is a limit to how much demand there will be for speed, past a certain point. No doubt there'll be some examples where huge amounts of reasoning are useful (like mathematics research), and speeding that up is helpful. But I'd expect many agents to start getting bottlenecked on your local/internal hardware, database calls and other bits of latency.

The price war is coming

Interestingly OpenAI reduced the cost of their Luna variant by 80% just before the DeepSeek V4 Flash GA release, making it remarkably affordable for a frontier model. While I haven't had as much luck with getting great output out of it vs GLM5.2, I think it points towards an absolute bloodbath of pricing at this end of the market.

You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We're already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus.

OpenRouter provider table for GLM 5.2 showing discounted per-million-token prices and throughput for StreamLake, NovitaAI, Decart, DeepInfra, Baidu Qianfan, CoreWeave, AkashML, GMICloud and Inceptron

While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra.

As the next generation set of GPUs start being deployed over the next few months - Nvidia's Vera Rubin series and AMD's MI400s, amongst others - the new HBM4 memory in those chips will deliver a 2x+ speedup on output tokens from memory bandwidth alone, plus more on top from additional compute and interlink.

In 2027 it's very possible we'll have very good quality models, at reasonable prices running at 500tok/s+. Staring at "still thinking on xhigh effort" for most of your day may finally become a thing of the past.

What will be interesting to watch for - and I'm not sure where to bet - is if the vast 2-3T+ param models actually do perform dramatically better for everyday tasks. On one hand it feels like we've hit a sweet spot right now, on another having an order of magnitude more intelligence in the model may make that sweet spot look very, very primitive.

Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day. ↩︎
A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product. ↩︎
I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market. ↩︎

AI 总结尚未生成

For the first time I can remember, I'm not choosing my daily driver models on raw intelligence. I'm choosing them on speed.

Have we reached an intelligence tipping point?

Is 100tok/s the new 100ms?

I've written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected.

But there are limitations

If you're familiar with Pareto's Principle and Amdahl's Law you'll know what's coming up.

Assuming "good enough" models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them.

Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn't move.

The price war is coming

You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We're already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus.

While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra.

Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day. ↩︎
A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product. ↩︎
I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market. ↩︎

原文

I'm (mostly) picking models on speed now, not intelligence

For the first time I can remember, I'm not choosing my daily driver models on raw intelligence. I'm choosing them on speed.

Have we reached an intelligence tipping point?

Is 100tok/s the new 100ms?

I've written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected.

But there are limitations

If you're familiar with Pareto's Principle and Amdahl's Law you'll know what's coming up.

Assuming "good enough" models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them.

Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn't move.

The price war is coming

You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We're already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus.

While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra.

Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day. ↩︎
A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product. ↩︎
I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market. ↩︎

中文翻译

我现在（大部分时候）选模型主要看速度，而不是看智能。

在我记忆中，这是我第一次不是根据原始智能来选择日常使用的模型，而是根据速度来选择。

我们是否已经到达了智能的临界点？

这话可能很快就会过时，但就目前而言，大约在 Opus 4.6 级别的模型似乎已经"足够聪明"，足以应付我大多数日常工作——编码、整理研究资料、设计幻灯片，以及针对大量数据库进行分析任务。

虽然我和大多数人一样，对尝试 Fable 感到兴奋，但讽刺的是，美国政府停摆让所有人有时间重新习惯 Opus。当 Fable 在热度过后带着更多护栏回归时，我注意到的第一件事就是它有多慢^[1]。实际上慢到让我很快就换回了 Opus。

我的职业生涯中有很大一部分时间都花在让软件变得更快上。当软件快速时，交互体验的提升是惊人的。根据我的经验（以及许多研究），一个产品即使再精美，如果它很慢，你也不会喜欢使用它。同样，一个非常基础的产品如果速度极快，用起来也会感觉极其实用。^[2]

我很清楚，当初只有少数几个庞大而缓慢的模型能够越过上述（且假设性的）智能门槛时，使用更慢的模型确实不太划算。因为无论你获得了多少速度优势，都会因为模型出错需要重做而迅速损失掉。

100tok/s 是新的 100ms 吗？

我之前写过关于智能体像拨号上网一样的文章，那时前沿模型还以 30-60tok/s 的速度缓慢爬行。如今这个变化比我预想的要快得多。

我记得的一个关键事实是，对人类来说，约 100ms 的延迟感觉就像"瞬间"，这是黄金标准。我认为模型以 100tok/s 的速度输出时，差不多刚好是我能跟上的速度。再快的话，输出速度就会超过我（略读）的速度。这并非一个精确的标准，因为现在模型的大部分时间越来越多地花在推理上，而不是真正向你展示输出 token。而且这也极大地取决于你的输出内容——散文每字符消耗的 token 远比代码少，所以你的实际体验可能（而且一定会）有所不同。

但粗略来说，100-200tok/s 对我来说已经非常快了。低于 50tok/s 的输出会感觉越来越慢。讽刺的是，超过 200tok/s 又几乎让人有点不安——你可以在这里体验一下 10,000tok/s+ 的模型（！）。我相信随着我们逐渐适应，并让智能体承担更复杂的工作，这个心理阈值还会继续上移。

鉴于大量我认为刚刚达到上述门槛的新模型——比如 GLM5.2 和 DeepSeek V4 Flash GA——它们不仅开放权重，而且体量（更）小，我们现在拥有了丰富的模型和速度选择。如果你在 OpenRouter 上查看 GLM5.2 各家服务商的速度排名，就能看到巨大的服务速度差异——从最低端不到 30tok/s 到最高端的 129tok/s。

这是开放权重生态系统的又一大优势。虽然成本上的巨大好处显而易见，但服务商们因此也有动力在速度上进行这样的竞争，这真的很有意思。^[3]

但也有局限

如果你熟悉帕累托原则和阿姆达尔定律，你就会知道接下来要说什么了。

假设"足够好"的模型继续变得越来越快，那么速度优势将越来越多地被工具调用，以及监督它们的人类所消耗。

以使用 50tok/s 模型的智能体为例。大部分时间都花在等待推理结果返回上。现在以 250tok/s 运行同一个回合，你会发现瓶颈越来越多地出现在"本地"机器上的工具调用，以及你自己的决策上。

只是粗略的数字，但趋势是对的。模型速度提升 5 倍，整个回合只快了 2 倍，因为另外 25 秒没有变化。

更糟糕的是，让本地机器在这些工具调用上变得更快这件事，几乎已经停滞不前了，因为 AI 让硬件成本呈抛物线式飙升。这又是 AI 市场带来的一个奇怪的衍生效应。

所以我怀疑（至少就目前而言），对速度的需求在超过某个临界点之后是有限的。毫无疑问，在一些场景下大量推理非常有用（比如数学研究），加快推理速度也很有帮助。但我预计许多智能体会开始在本地/内部硬件、数据库调用以及其他延迟环节上遇到瓶颈。

价格战即将来临

有趣的是，OpenAI 在 DeepSeek V4 Flash GA 发布前将其 Luna 变体的价格降低了 80%，使其作为前沿模型变得非常亲民。虽然我在用它与 GLM5.2 对比时，并没有那么好运气获得出色的输出，但我认为这表明市场这一端将迎来一场惨烈的价格血战。

你可以在 OpenRouter 上看到 GLM5.2 正在发生这一切——层出不穷的折扣试图吸引客户。GLM5.2 的价格已经降到每百万 token 0.42/1.32 美元——只有 Opus 价格的 5%。

虽然最便宜的服务商速度较慢，但只需多一点价格，你就可以从 DeepInfra 获得 109tok/s 的速度。

随着下一代 GPU 在未来几个月开始部署——包括 Nvidia 的 Vera Rubin 系列和 AMD 的 MI400 系列等——这些芯片中的全新 HBM4 内存仅凭内存带宽就能带来 2 倍以上的输出 token 加速，再加上额外的算力和互联性能提升。

到 2027 年，我们很有可能拥有质量非常高、价格合理且能以 500tok/s+ 速度运行的模型。每天大部分时间盯着"仍在以极高强度思考"的日子可能终于要成为过去了。

接下来值得关注的是——而我也不知道该押注哪一边——那些庞大的 2-3T+ 参数模型在日常任务中是否真的会有显著更出色的表现。一方面，感觉我们现在已经达到了一个最佳平衡点；另一方面，模型中多出一个数量级的智能，可能会让这个最佳平衡点看起来非常、非常原始。

其次当然是无休止的护栏触发，而且往往在最糟糕的时刻发生——正当我深入一项困难任务，觉得需要 Fable 提供的额外"火力"时，不过那就是另一个故事了。 ↩︎
一个典型的例子是 Craigslist 或 Hacker News。虽然它们看起来过时了，但响应速度快得让你根本不会注意到这一点。同样，你的"标准"SPA 应用为了渲染一个首页要加载 30MB 的 React，用起来感觉像糖浆一样黏稠，大多数时候都是一种煎熬，尽管在设计上肯定投入了巨资。 ↩︎
我知道 OpenAI 和 Anthropic 早已提供模型的快速变体，但 API 定价高得令人咋舌。除非你拿着一张空白支票去刷基准测试，否则我没见过有人把它们用于日常运营。以合理价格提供快速又优质的模型，是市场上非常新的产物。 ↩︎