Kimi K3 Hits 842 Tokens/s on vLLM: What Is Claimed vs. What Is Verified

NVIDIA and the vLLM team say Moonshot's 2.8-trillion-parameter Kimi K3 now decodes at 842 tokens per second at batch size 1. The number is real and comes from a primary source, but it is a controlled benchmark. It assumes a synthetic speculative-decoding acceptance length of 6, so it is a ceiling for a tuned setup and not what a typical chat request will see today. This post lays out what vLLM actually measured, which comparisons are fair, and what to check before you quote the figure.

The source is the vLLM blog post Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly, published October 16, 2026 by Benjamin Chislett of NVIDIA. It builds on the day-0 support post, Kimi K3 Is Here, and the DSpark training write-up, How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72.

Quick reference: the claims at a glance

What is claimed

The vLLM post describes a unified decode path written with CuTe DSL, NVIDIA's Python-based kernel language, for Kimi K3 on GB300 GPUs. It fuses attention, mixture-of-experts (MoE) work and inter-GPU communication so that a single decode step launches fewer, larger kernels.

The post's headline results, quoted from the page:

• 842 tokens/s at batch 1 on 16 GB300 GPUs with DSpark, versus 488 for stock vLLM (1.73x).
• 1.5 to 2x higher decode throughput in several tested configurations without speculation.
• 1.8x higher P90 interactivity on the AgentX workload at concurrency 1: 403.2 versus 223.7 tokens/s/user.

Kimi K3 842 tokens per second on vLLM with DSpark versus 488 for stock vLLM at batch 1 Image: vLLM

What is verified, and what is not

"Verified" here means the claim is stated with its conditions in the primary source. We did not run these benchmarks, and no independent party has published a replication yet. Here is how each claim holds up on the evidence in the post itself.

The 842 figure is a controlled benchmark, not a measurement of real prompts

The post says the setup uses DSpark with seven draft tokens, a synthetic acceptance length of 6, an input length of 1 and an output length of 1k. In speculative decoding, a small draft model proposes several tokens and the big model verifies them in one pass. The acceptance length is the average number of tokens that survive verification per step. Fixing it at 6 removes the variable that depends on the prompt.

Real acceptance rates vary a lot. In the earlier DSpark training post, the team reported a macro average of 4.11 tokens per verification round across nine domains: 6.42 on math reasoning, 4.96 on HumanEval, 4.65 on translation and up to 5.31 on a 378K-token LongBench prompt. The July launch post gave about 4.73 for coding and other low-entropy tasks and about 2.61 for high-entropy tasks such as creative writing. An acceptance length of 6 is therefore at the high end of what the same team measured, reachable on math but not typical.

That does not make 842 dishonest. The post labels it as controlled. It does mean you should not read it as "Kimi K3 answers your question at 842 tokens per second".

The 842 and 370 numbers are not the same experiment

Earlier, vLLM reported 118 tok/s without speculation and 370 tok/s with DSpark at batch 1 on TP16, using the actual trained draft model and its measured acceptance. Someone comparing 370 to 842 could conclude that kernels alone gave a 2.3x gain. The October post gives a stock-vLLM figure of 488 tok/s under the synthetic AL=6 assumption, which shows how much of the jump is an assumption change rather than a speedup. The apples-to-apples gain from the new kernels under identical conditions is the 1.73x.

The non-speculative numbers are the more portable result

Without speculation, the post gives a full table of total decode throughput across batch sizes:

These do not depend on a draft model, so they are the safer basis for capacity planning. Gains shrink as batch size grows, which is expected: large batches are more compute-bound and less limited by kernel launch overhead and memory latency.

The AgentX 1.8x mixes hardware and software

AgentX, from SemiAnalysis' InferenceX project, replays coding-agent traffic: long contexts, multiple turns, tool pauses, parallel subagents and KV-cache reuse, with private content replaced by synthetic text. The post reports 403.2 tokens/s/user at P90 interactivity on 16 GB300 GPUs versus 223.7 from an earlier vLLM result on 8 GB300 GPUs. Because the GPU count doubled, the headline 1.8x is not a pure software gain. The post says so itself.

How the speedup is achieved

The post shows a kernel timeline for one layer: stock vLLM takes 130 microseconds across 23 kernels, the specialized path takes 69 microseconds across 13.

Kernel timeline showing stock vLLM at 23 kernels per layer versus 13 in the specialized Kimi K3 decode path Image: vLLM

The authors list four families of change:

1. Shape tuning. Small-matrix kernels for attention input projections at batch sizes 1 and 2, split-K kernels at larger batches, and a tuned decode kernel for Kimi Delta Attention (KDA).
2. Communication overlap. The attention output projection is fused with its all-reduce and the next residual computation. GPUs write partial results into NVLink-accessible buffers with a fence-less exchange.
3. MoE scheduling. Router scores and the latent input projection, which the baseline computed redundantly on every GPU, are split across ranks. The shared expert runs on a separate stream.
4. Weight prefetching. During communication phases, which do not saturate memory bandwidth, a side-stream kernel prefetches the next operation's weights.

The design only applies where all GPUs share one NVLink domain with multicast support, which is why it is scoped to GB300-class hardware at 8 or 16 GPUs and decode batches from 1 to 64.

The AI-agents angle

The most interesting part of the post is not the number. Chislett says the work was built with agent-assisted development: coding agents wrote complex GPU kernels, fused operators, overlapped computation with communication and tuned across batch sizes. The authors argue agent output now outpaces human review, so agents should also write tests, run evaluations, investigate failures and document when an optimization works, leaving humans to judge design and evidence.

They also report an honest failure. A correctness audit found a compiler reordering of activation reads ahead of a synchronization wait; the fix restricted that load type to static weights. Some numerical problems in attention and communication kernels showed up only in long-context and long-generation evaluations such as MRCR and AIME25, and did not reproduce in standalone kernel tests with mock inputs.

If you build with coding agents, that is the practical lesson: kernel unit tests passed while end-to-end evals caught the bugs. Pair both.

Why it matters for people who use Kimi K3

Single-stream speed is what makes an agent feel interactive. A coding agent that waits on dozens of sequential model calls spends most of its wall-clock time in decode at low batch. Faster batch-1 decode is the metric that matters there, which is why the post stresses AgentX-style traffic.

For most readers the practical effect today is indirect. You will not run 16 GB300 GPUs yourself. Hosting providers might, and per-user speed could rise. If you want to run K3 on your own machine, see our guide to running Kimi K3 locally and the Kimi K3 open-weights launch coverage. For the architecture behind the hybrid attention, read Raschka's Kimi K3 teardown.

Related work in the same post

The post says the same design pattern follows into other models:

• DeepSeek-V4.1-Flash gets an opt-in latency-specialized decode backend, supporting up to 64 concurrent requests with fallback for prefills and mixed batches (vLLM pull request 61868).
• Meta's Muse Spark Open: early results within two days exceeded 950 tokens/s per user at seven draft tokens and AL=6, again a synthetic-acceptance setup. See our Muse Spark Open DFlash 2 coverage.
• GLM 5.3 is promised a similar low-concurrency mechanism in a follow-up.

For the state-space-model side of the same optimization wave, see NVIDIA ReplaySSM, and for the speculative-decoding background, our explainers on DeepSeek's DSpark and tokens per second.

How to check the claim yourself

1. Read the vLLM pull request referenced in the post (vllm-project/vllm number 61867), which holds the full results and design.
2. Look for the acceptance length. If a speed claim does not state whether it is measured or synthetic, ask.
3. Compare like with like: same GPU count, same batch, same draft model, same input and output lengths.
4. Watch for independent runs through the Bring-Your-Own-GPU evaluation RFC (vLLM issue 61698) and InferenceX's AgentX methodology.
5. Treat prefill separately. The 842 figure excludes it, and long prompts can dominate time to first token.

What people are asking

Is this the fastest anyone has served a 2.8T model? The post does not claim a record. It claims a speedup over stock vLLM on the same hardware. Other engines have published their own Kimi K3 numbers, and different metrics (per user, per GPU, per megawatt) are not interchangeable.

Does it change quality? The authors say features like full-context serving, temperature sampling and structured output must not be opted out of, and that kernel tests are paired with end-to-end evals. No quality numbers for the optimized path are given beyond that.

Will it reach my API provider? The change is isolated to model code with a path back to the standard one, which makes adoption easier, but it needs the matching NVIDIA hardware.

Bottom line

Verified: a first-party post from the vLLM project and an NVIDIA engineer reports 842 tokens/s at batch 1 and a 1.73x speedup over stock vLLM under identical, stated conditions, plus 1.5 to 2x gains without speculation. Not verified: that real workloads reach 842 tokens/s, since acceptance length 6 is synthetic; that the AgentX gain is purely software; and any independent replication. Quote the 1.73x and the non-speculative table, and call 842 a controlled ceiling.

Figures reflect the vLLM post of October 16, 2026 and may change as the pull request evolves.

Related reading

• Run Kimi K3 locally
• Kimi K3 open weights: 2.8 trillion parameters
• Kimi K3 architecture teardown
• NVIDIA ReplaySSM
• Muse Spark Open with DFlash 2 on vLLM
• DeepSeek DSpark explained
• What is TPS?