explainx / blog
GigaToken is a Rust tokenizer claiming up to 1000x faster throughput than HuggingFace Tokenizers. How the speedup works and where it matters.
Jun 21, 2026
Pake wraps any webpage in a native desktop window using Tauri and your OS's built-in WebView — no Chromium, no Electron bloat. The result is a ~5MB app that launches instantly. 55,000 GitHub stars, 63 contributors, packages for ChatGPT, Grok, YouTube, Excalidraw already built. One command to ship your own.
Jun 21, 2026
Turso rewrites SQLite in Rust — keeping full SQL and file format compatibility while adding MVCC concurrent writes, async io_uring I/O, CDC, vector search, full-text search, and an MCP server mode. 20,000+ stars, 253 contributors, in beta. Here is the full breakdown and why it matters for the agentic era.
Jul 23, 2026
A 2022 Hugging Face/BigScience project called Petals — run large language models at home, BitTorrent-style — hit the Hacker News front page again in 2026, reigniting a debate about whether peer-to-peer LLM inference is finally viable now that models are smaller, quantization is better, and newer projects like Mesh LLM and AI Horde have taken different approaches to the same problem.
TL;DR: Marcel Rød open-sourced GigaToken, a Rust tokenizer claiming up to ~1000x faster encoding throughput than HuggingFace Tokenizers on the same hardware — reaching gigabytes per second. It hit ~1,300 GitHub stars within a day. The speedup comes from SIMD-accelerated pretokenization and aggressive caching, not just "rewrite it in Rust" (HuggingFace's tokenizer and tiktoken are already Rust-based). It matters most for pretraining data pipelines and, more modestly, time-to-first-token in inference.
From the project README:
"~1000x faster than HuggingFace's tokenizers, drop-in replacement. Tokenize your text data at GB/s!"
That claim is notable specifically because both HuggingFace Tokenizers and tiktoken — the two comparison points — are already Rust-based with Python bindings, not naive pure-Python implementations. A 1000x speedup against an already-compiled, already-optimized baseline is a much stronger claim than beating a slow interpreted implementation, which is why the project's own FAQ addresses the skepticism directly:
"Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these!"
| Question | Answer |
|---|---|
| What is it? | Open-source Rust tokenizer, drop-in HuggingFace/tiktoken replacement |
| Claimed speedup? | ~1000x (native API) vs HuggingFace Tokenizers |
| Compatibility mode speedup? | ~200-300x, per the author |
| License? | MIT |
| Install? | pip install gigatoken |
| Tokenizers supported? | Wide range of commonly used tokenizers; WordPiece not yet |
| Hardware tested? | Modern x86 and ARM, consistent results across CPUs |
| GitHub stars? | ~1,300, within roughly a day of posting |
| Key techniques? | SIMD pretokenization, minimized branching, aggressive caching |
Rød's own explanation, quoted directly from the FAQ, names three specific techniques:
"The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed. Some gains are also achieved from minimizing interactions with Python, and avoiding communication between threads."
Breaking this down for readers less familiar with tokenizer internals:
The project's headline benchmark, run on an 11.9GB sample of the OpenWebText dataset:
| Hardware | GigaToken | HuggingFace Tokenizers | Speedup |
|---|---|---|---|
| Apple M4 Max (16 cores) | 1.432s, 8,327 MB/s | 16.25s, 6.15 MB/s (100MB subset) | 1,353x |
| AMD EPYC 9565 (144 cores, 2 sockets) | 0.486s, 24,532 MB/s | 4.033s, 24.80 MB/s (100MB subset) | 989x |
Note the comparison methodology: HuggingFace's number is measured on a 100MB subset for practicality, while GigaToken's is measured on the full 11.9GB file — the author validates that both produce matching output on the shared subset before reporting the speedup ratio, addressing the most obvious methodological objection (are they actually tokenizing the same thing correctly?).
At the EPYC throughput rate, the README notes you could tokenize the entirety of Common Crawl — often described as effectively "the entire internet," roughly 130 trillion tokens — in under 6.5 hours.
This was the most-debated question in the Hacker News discussion, and the honest answer is "it depends on what you're doing."
For a single inference request, tokenization genuinely is a tiny fraction of total compute — commenters cited figures around 0.1%. If you're serving one chat completion, GigaToken's speedup is close to irrelevant to end-to-end latency.
For pretraining data pipelines, it's a different story. Rød, responding directly on the thread:
"In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM."
A multi-day tokenization job that becomes a multi-hour job directly speeds up how fast researchers can iterate on data mixture changes — a meaningfully different value proposition than single-request latency.
For time-to-first-token (TTFT) specifically — not total inference time — the author shared preliminary numbers from testing against sglang on an 8B Qwen3 model on a single B200 GPU:
| Input length | TTFT (HuggingFace) | TTFT (GigaToken) | Reduction |
|---|---|---|---|
| 2,048 tokens | 30.74ms | 29.05ms | 5.5% |
| 8,192 tokens | 105.20ms | 96.36ms | 8.4% |
| 32,768 tokens | 687.05ms | 633.66ms | 7.8% |
These are preliminary, not yet in the published README, and TTFT gains are smaller than raw tokenization-throughput numbers would suggest — because tokenization is only one part of the prefill pipeline, and the GPU-bound work still dominates total time. But the gains are real and grow with input length, which matters increasingly as long-context agentic workloads become more common.
One security-adjacent use case surfaced too:
"I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request." — a Hacker News commenter running production AI infrastructure
For platforms that tokenize before routing a request to a specific model or backend, tokenization sits on the critical path even though it's a small fraction of overall compute — the same "latency-critical, not throughput-critical" distinction that shows up across systems engineering generally.
The project is transparent about what isn't finished yet:
If you run pretraining data pipelines:
If you're optimizing inference latency:
If you're evaluating build-vs-adopt for infra like this:
Primary sources: github.com/marcelroed/gigatoken · fastokens (TTFT benchmark tool)
Benchmark figures reflect the GigaToken project's published README and author comments on Hacker News as of publication. Speedup ratios vary by hardware, tokenizer, and workload — run the project's own validation and benchmark commands against your specific setup before adopting in production.
