The headline is true in the narrowest possible sense: FreeToken served the 753-billion-parameter GLM-5.2 on a single workstation GPU. But that GPU was not the whole computer. The FreeToken paper used one 96GB RTX PRO 6000 Blackwell, backed by 512GiB of DDR5 system memory, to serve a 433GB NVFP4 checkpoint at 14.9 tokens per second.
That correction is not a reason to dismiss the work. It is the reason the work is interesting. FreeToken's contribution is not squeezing 433GB into 96GB. It is treating the GPU, CPU, host RAM, PCIe link, and storage as one inference machine, then scheduling a sparse Mixture-of-Experts model around the measured bandwidth of each tier.
If you have been choosing between a Mac and a discrete GPU for local LLMs, or pricing a build after the recent RAM-cost shock, that distinction matters more than the viral parameter count.
TL;DR — what people are actually asking
| Question | Direct answer |
|---|---|
| Did 753B fit in 96GB VRAM? | No. The complete 433GB checkpoint lived primarily in 512GiB host RAM; the 96GB GPU held non-expert weights, runtime state, KV cache, and a changing cache of active experts. |
| Was it one GPU? | Yes: one RTX PRO 6000 Blackwell. But it was a high-memory workstation, not a normal 96GB-only box. |
| Measured GLM-5.2 speed? | 14.9 tok/s for FreeToken versus 7.3 tok/s for llama.cpp; comparable mean time to first token, 7.5s versus 7.8s. |
| Smallest tested system? | RTX 4060 Laptop, 8GB VRAM + 32GiB LPDDR5, serving NVFP4 Qwen3.6-35B-A3B at 39.3 tok/s. |
| Gaming-desktop tier? | RTX 5090 systems served Qwen3.6-35B-A3B at 77–83 tok/s and DeepSeek-V4-Flash at 22–25 tok/s in the main evaluation. |
| Why MoE only? | A token activates a small subset of experts, so compute can fit even when the complete expert library cannot. Dense-model offloading has different economics. |
| Is this a drop-in Ollama replacement? | Not generally. It is an Apache-2.0, NVIDIA-focused MoE serving engine with desktop and CLI paths; broad compatibility still favors established runtimes. |
The “single GPU” claim needs the rest of the specification
Parameter count is not memory usage. Quantization changes bytes per parameter; MoE changes how many parameters are active per token; offloading changes which memory tier holds them.
The paper's frontier demonstration used this stack:
| Resource | Published GLM-5.2 setup | What it does |
|---|---|---|
| Model | GLM-5.2, 753B total / 40B active | Routes each token through a small expert subset |
| Checkpoint | 433GB, NVFP4 routed experts | Complete weights that must exist somewhere |
| GPU | RTX PRO 6000 Blackwell, 96GB VRAM | Dense layers, KV cache, hot experts, GPU execution |
| Host memory | 512GiB DDR5 | Source-of-truth expert pool |
| CPU | Xeon Platinum 8559C, 48 threads in the table | Directly executes part of each expert-cache miss set |
| PCIe | Gen 5 x16; 51.5GB/s measured transfer | Streams selected experts into GPU cache |
| Host bandwidth | 178GB/s measured expert-kernel bandwidth | Supplies both CPU execution and DMA reads |
| Observed decode | 14.9 tok/s | Paper result on the math workload |
So the useful buying rule is not “753B needs one 96GB GPU.” It is: a 433GB checkpoint needs enough host memory to remain resident, enough VRAM for the dense path plus useful caches, and enough host/PCIe bandwidth to feed each token interactively.
That is also why this result does not invalidate the existing GLM-5.2 local guide. Different formats and runtimes make different capacity-speed trade-offs. FreeToken's paper evaluates its own FTW layout and NVFP4 expert path; it is not a blanket claim about every GLM-5.2 file or every workstation.
How FreeToken turns a PC into one inference platform
The architecture addresses two opposite phases of generation.
Prefill: load almost every expert without idling the GPU
During long-prompt prefill, many tokens collectively route through most experts. Sparse activation no longer saves much movement. FreeToken therefore uses two full-layer buffers: while the GPU computes layer l, the system streams layer l + 1 over PCIe.
In the paper's RTX 5090 analysis, an 8,192-token prefill chunk completed in 1.19–1.22 seconds, essentially the time needed to move the 64.4GB expert pool at the measured 52.7GB/s link rate. Turning off this double buffering cut prefill throughput by 19% at 4K tokens, 25% at 8K, and 26% at 16K.
Decode: decide whether to move an expert or compute where it lives
One generated token touches few experts, and adjacent tokens often reuse the same ones. FreeToken tracks those experts in a global LRU cache in VRAM. When an expert misses, there are two valid options:
- copy it over PCIe, run it on the GPU, and keep it cached; or
- run it directly on the CPU against host-resident weights.
The runtime profiles effective host bandwidth and pinned PCIe transfer
bandwidth on the actual machine. Its q* policy splits misses so both paths run
concurrently rather than leaving either CPU bandwidth or PCIe bandwidth idle.
The partial CPU and GPU results are merged exactly; this is a scheduling change,
not approximate expert skipping.
Agent loops: reuse state where harnesses edit context
Tool-using agents repeatedly trim thinking blocks, replace old tool output, and append new turns. That can invalidate recurrent state and force expensive re-prefill. FreeToken places state checkpoints at semantic boundaries such as tool calls and conversation turns, then recomputes only the changed suffix.
That agent-specific design is more important for builders than a single-stream tokens-per-second chart. The paper tested AIME reasoning, an OpenCode SWE task, a Claude Code-compatible multi-agent workload reaching 56K–65K tokens, and an email/calendar agent over 13 turns. For context on why runtimes and harnesses must be evaluated together, see the agent harness guide.
What the benchmarks do — and do not — prove
On RTX 5090, FreeToken reported:
- 77–83 tok/s on Qwen3.6-35B-A3B, 1.8–2.3x the strongest supported baseline;
- 22–25 tok/s on DeepSeek-V4-Flash, 1.5–1.9x the strongest supported baseline;
- decode rate within 12% of the single-turn value across three agent workloads;
- worst-turn time to first token below 44 seconds in every evaluated cell, while each baseline exceeded 150 seconds in at least one cell.
The cross-hardware result is equally useful: the 8GB RTX 4060 laptop served the official NVFP4 Qwen3.6-35B-A3B build at 39.3 tok/s. That is an impressive systems result, but it is easy to overgeneralize. Qwen3.6-35B-A3B is sparse and quantized. It does not imply an arbitrary dense 35B checkpoint will run at that speed or fit the same host-memory budget. Read the Qwen 3.6 local guide for the dense-versus-MoE quality and speed trade-off.
Three cautions belong beside the headline:
- These are author-reported v1 paper results. We reviewed the paper and public repository; explainx.ai did not reproduce them on the six listed machines.
- The comparison is specialized. Weight formats were aligned where supported, but not every baseline supported every model or multi-turn workload.
- “Interactive” is not “cheap.” A 96GB workstation GPU, 512GiB RAM, and fast storage remain serious hardware. Local avoids per-token API billing, not capital, electricity, cooling, or maintenance.
Hardware sizing: choose the model tier before the GPU
Use the paper's systems as evidence, not as shopping-list minimums.
| Goal | Evidence-backed reference configuration | Practical expectation |
|---|---|---|
| Try edge MoE on a laptop | RTX 4060 Laptop, 8GB VRAM, 32GiB LPDDR5 | Qwen3.6-35B-A3B NVFP4 at 39.3 tok/s in the paper; tight context and shared resources still matter |
| Run 35B MoE comfortably | 24–32GB GPU, 180–240GiB host RAM in evaluated servers; Qwen's full pool is much smaller than GLM | Better cache capacity and bandwidth; actual minimum depends on checkpoint format |
| Run 284B DeepSeek-V4-Flash | RTX 5090 32GB; paper evaluated 180GiB server or 192GiB desktop host memory | 22–25 tok/s in main 5090 evaluation; host RAM and PCIe are first-class components |
| Run 753B GLM-5.2 | RTX PRO 6000 96GB, 512GiB DDR5, 433GB NVFP4 checkpoint | 14.9 tok/s demonstrated; workstation-class, not consumer-GPU-only |
Before buying hardware:
- Find the exact checkpoint size, not just total parameters.
- Budget host RAM above that file size for runtime and operating-system overhead. The paper gives tested configurations, not universal minimums.
- Check PCIe lane width. The paper's laptop had Gen 4 x8 at 11.8GB/s; the 5090 desktop had Gen 5 x16 at 49GB/s measured.
- Treat DRAM bandwidth as compute capacity. More CPU cores cannot compensate for a starved dual-channel memory path.
- Plan storage for downloads and conversion. A 433GB checkpoint already exceeds many boot SSDs; FTW conversion and temporary artifacts can require additional space.
- Compare against API cost and smaller local models. The Qwen3.8-27B guide and local open-weight model setup are better starting points when privacy matters more than frontier parameter count.
How to try FreeToken without pretending we benchmarked it
The public repository is Apache 2.0 and documents Windows and Linux desktop
downloads. Its CLI path uses uv or pip:
uv pip install "freetoken[accel]"
Then launch a supported model by local path or model identifier:
ft serve --model ~/models/Qwen3.6-35B-A3B
curl http://127.0.0.1:1919/v1/models
The server exposes OpenAI-compatible routes (/v1/chat/completions,
/v1/responses) and Anthropic-compatible routes (/v1/messages). The
documented agent launcher can preview configuration changes before making them:
ft launch opencode --dry-run
Remove --dry-run only after checking the proposed provider and CLI changes.
If you prefer a more established runtime or a dense checkpoint, start with the
llama.cpp guide instead.
Who should use FreeToken now?
Good fit: a local-AI builder with an NVIDIA RTX 30/40/50-series GPU, substantial host RAM, a supported MoE checkpoint, and a reason to keep agent traffic private or avoid sustained API spend.
Poor fit: someone shopping for a normal gaming PC because “753B runs on one GPU,” a Mac-only builder, a dense-model workload, or a team that needs broad production compatibility more than experimental performance.
The durable lesson is broader than FreeToken. Open weights do not become open access merely because a download exists. Runtimes that coordinate heterogeneous memory can move the local frontier substantially — but the honest unit is the whole workstation, not the GPU badge.
Sources
- FreeToken paper on arXiv — architecture, hardware table, workload definitions, and reported results
- FreeToken repository and README — license, installation, supported hardware, and runtime overview
- FreeToken quick start — server, API, shell, and agent-launch commands
Related on explainx.ai
- Qwen 3.6 27B local development guide
- Qwen3.8-27B local model and hardware reality
- Run GLM-5.2 locally with Unsloth
- What is llama.cpp?
- MacBook vs dedicated GPU for local LLMs
- RAM prices and local-inference cost
- How to run open-source models locally in OpenCode
FreeToken version, model support, repository details, and benchmark claims were checked against arXiv v1 and the public repository on August 24, 2026. Hardware results are the authors' measurements unless explicitly labeled otherwise.