VocabX: Specializing a Small Model's Vocabulary Cut Inference Cost 77% in Production

Fine-tuning a small open model usually starts with data and weights. A Hugging Face post from Dharma-AI argues it should start one layer earlier, with the vocabulary. The team says rebuilding the tokenizer of a fine-tuned Qwen3-4B essay grader cut the response from 350 to 101 tokens and the measured inference cost by 77%, while the quality score went up, not down. The details are in Dharma-AI's Hugging Face article, which previews a paper the authors say is still under review.

If you serve a specialized small model in volume, the ideas here are worth knowing even before the paper lands. They also fit a pattern we keep seeing: company write-ups where a specialized small model beats a general one on a measured task, as in Coinbase's fraud model and Shopify's post-trained Nemotron.

TL;DR

The problem: a vocabulary built for everyone

A tokenizer decides how text becomes tokens, and general-purpose tokenizers are trained on text from many languages and domains. For a model used in one domain, Dharma-AI names two kinds of waste.

1. Fragmentation. The tokenizer splits the text the domain repeats most into many pieces. In the grader, every response starts with the same JSON envelope, the same competency names and the same evaluation phrases, each costing several tokens where one could do. Contracts have boilerplate clauses; code has identifiers.
2. Excess. The vocabulary carries tens of thousands of entries the domain never uses, yet the model computes a score for all of them at every generated token. The grader's Qwen3-4B tokenizer has 151,669 entries, including many for languages it never writes.

Why reading is cheap and writing is expensive

The post's most useful idea is that token count is not the bill. Inference has two phases:

• Prefill processes the whole input in one pass. Each weight read is reused for every input token, so compute is the limit.
• Decode generates one token at a time, each needing a full pass. Each weight read is used only a few times, so the GPU waits on memory bandwidth.

Small models served with moderate batches are strongly decode-bound, and each decode step also reads the growing KV cache. In the grader, Dharma-AI measured a generated token at 7.4 times the cost of a read token, while noting the ratio varies with hardware, batch size and context length.

Ribbon of text sliced into beads, showing how a tokenizer splits repeated domain text into many tokens that vocabulary specialization merges

The third piece is the LM head, the last layer that scores every possible token. In a Qwen3-4B-sized model it accounts for almost 10% of parameters, and it is read and multiplied in full for every generated token. An unused vocabulary entry costs exactly as much as a frequent one, so adding tokens can lower the count and still raise the bill.

How VocabX works

VocabX makes three changes to standard vocabulary adaptation.

1. Ask about cost, not count. Each eliminated token is multiplied by the price of the field it appears in: system prompt, user message or response. Since a response token was worth 7.4 input tokens in the grader, the vocabulary is built accordingly.
2. Work on two layers under one criterion. At the sub-word layer it rebuilds how domain words are cut. At the multi-word layer it finds repeating spans, from short expressions to whole JSON envelopes, and turns each into a single macro token. Each layer has its own budget.
3. Estimate what the sample cannot see. Choosing tokens by cost needs to know how much each candidate still saves given those already chosen. Measuring exactly would mean reprocessing the corpus tens of thousands of times, so the team measures on a sample. An expression that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time. Instead of giving unseen candidates zero gain, VocabX counts a candidate's corpus-wide savings exactly and only estimates the not-yet-covered part statistically. On a contracts corpus, that change alone raised token reduction from 44.7% to 52.8%.

To find repeating spans, VocabX uses five miners for long words, whole lines, expressions, complete sentences and structures with punctuation such as JSON and tags. Dharma-AI reports the structure miner contributed the most; without all five, token reduction on the grader would fall by 55.6 points.

The grader results and one course correction

The team trained seven versions of the grader with the same base model, data, recipe and seed, differing only in tokenizer. All ran the same 500 requests on an RTX 5090 with vLLM, and cost was calculated from measured runtime.

The first VocabX attempt fused each competency's grade with the surrounding JSON into one macro token. It was the most compact but the bench score fell to 0.689. With the grade inside the macro, each grade-and-structure combination became a separate token, and the model had to choose both at once. Giving the grade its own token cost two extra tokens per response, raised the score to 0.810 and was still cheaper in the measurement: 77.0% against 73.3% for the fused version. The lesson, in the authors' words: compression has a cost when the compressed span carries the very decision the model exists to make.

Does it generalize?

Dharma-AI compared VocabX with fast vocabulary transfer, multi-word tokenization and in-place BPE expansion on five public domains (contracts, medical papers, scientific papers, news, Python code) and three model families (Qwen3, Llama-3.1, Gemma-3). It reports:

• VocabX compressed more than the strongest reference method in all fifteen combinations, by 1.2 to 9.3 points.
• In estimated inference cost, the pruned version led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3.
• On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, versus 61.8% for the best in-place BPE configuration using 207,803.

What is not shown

The last row is the practical cost. Changing the vocabulary changes the embedding and output layers, so the model must be trained again with the new tokenizer, which the team did with the same recipe. This is a lever for teams that already fine-tune and have a stable, high-volume task.

A mental model: the shipping analogy

Dharma-AI explains the idea with a shipping company that charges per box but seven times more for outgoing boxes than incoming ones. Cutting the total box count helps; cutting outgoing boxes helps seven times more. Packing each item better is the sub-word layer. Recognizing that the same twelve-item kit ships every week and sending it as one box is the macro-token layer. The vocabulary size is the warehouse: every extra shelf is more to search at every shipment, whether or not anything is stored on it. Keeping that picture in mind makes it easier to see why a method that only merges frequent input phrases misses most of the available savings for a generation-heavy task.

When it is worth considering

• High volume, narrow output. Structured JSON, classification with rationales, extraction, templated reports: the more repetitive the output, the more macro tokens save.
• Single language or domain. The fewer languages a model must write, the more of the vocabulary is dead weight.
• Decode-bound serving. Small models on moderate batches benefit most, because the savings are in generated tokens and LM-head reads.
• Not for open-ended chat. If outputs vary widely, repeated spans are rarer and the gain shrinks.

Pair it with the usual levers. Our coverage of AT&T's 56% AI coding cost cut through model routing and of NVIDIA's EdgeWise small-model delegation shows other ways to cut cost, and our small models comparison helps choose a base.

Related reading

• Coinbase's fine-tuned Qwen3.5 9B on fraud
• Shopify's post-trained Nemotron for Sidekick
• AT&T AI coding costs and model routing
• NVIDIA EdgeWise small-model delegation
• Small models comparison
• IBM Granite Timeseries Ensemble
• Official: Dharma-AI on Hugging Face · Qwen3 on Hugging Face

Figures are from Dharma-AI's October 21, 2026 article and a paper still under review; they may change before publication.