Meituan dNaViT: Open Discrete Vision Tokens for Unified Multimodal Models

Meituan LongCat published dNaViT in arXiv:2611.12426 on November 9, 2026, with open code in github.com/meituan-longcat/LongCat-Next and model assets on Hugging Face. The project asks a builder-relevant question: can finite discrete codes carry both semantics and pixels, so vision trains like language next-token prediction?

The answer in the paper is yes, with caveats: a shared vocabulary can serve reconstruction, classification, multimodal understanding, generation, and editing when training balances semantic and pixel branches.

TL;DR

The discrete vision bottleneck

Most multimodal stacks bolt continuous vision encoders onto language cores. Training infrastructure, sampling tricks, and speculative decoding patterns diverge. Chameleon, Emu3, and LongCat-Next instead push VQ-style indices that slot into autoregressive training.

The hard part is information loss: aggressive quantization hurts understanding benchmarks; gentle quantization hurts generation fidelity. dNaViT''s design targets that tradeoff explicitly.

Architecture in plain language

Frozen encoder supplies strong semantic features without retraining the whole ViT.

Residual Modulator (ResMod) fuses early and late encoder layers, then applies a zero-initialized scale-and-shift residual so training starts near the pretrained representation and gradually adds reconstruction-aware detail.

Grouped product quantization expands effective codebook capacity without lengthening the spatial token sequence — important for inference cost when you treat images as token streams.

Dual-branch losses supervise semantic reconstruction and pixel reconstruction, trained in three stages (continuous init → discrete alignment → joint optimization).

Optional Refiner modules improve perceptual quality without changing underlying discrete codes, per the README in LongCat-Next.

Open-weight distribution graphic for Meituan dNaViT tokenizer code in LongCat-Next

Reported results (from the paper)

Treat numbers as author-reported until you reproduce:

• Understanding: with Qwen3-8B, paper tables cite 885 on OCRBench and competitive document reasoning vs Qwen3-VL-8B baselines.
• Generation: GenEval2 49.5 and Qwen-Image-Bench 52.17 in listed runs.
• Reconstruction: PSNR 28.12 and SSIM 0.81 at 512×512.

The authors stress data efficiency: strong scores with smaller training budgets than some compared systems — relevant if you fine-tune tokenizers on proprietary imagery.

Multimodal fusion graphic for dNaViT vision and language token streams in one model

Training stages (why three phases matter)

The paper describes progressive training to stop semantic and pixel objectives from fighting each other:

1. Continuous initialization — align ResMod on frozen encoder features without quantization noise.
2. Discrete feature alignment — introduce grouped product quantization while keeping semantic anchors stable.
3. Joint optimization — balance representation loss and pixel reconstruction loss with gradient interference controls.

Skipping stage two is a common failure mode in VQ-VAE projects: codes collapse to a tiny subset of indices, wrecking understanding metrics. dNaViT''s claim is that ResMod + grouped PQ keeps code utilization high enough for downstream LMs.

Discrete vs continuous vision in one table

If you build RAG over screenshots or UI agents, discrete tokens can let you log visual tokens beside text in one JSONL stream — useful for multimodal knowledge agents-style evals.

Reported benchmark rows (selected)

Treat as author tables pending reproduction:

The paper argues these land with less training data than some baselines — if true, tokenizer quality not brute-force scale drove the gain. Your finetune team should still run held-out product screenshots before believing OCR numbers.

How to try dNaViT this week

1. Clone LongCat-Next and read dNaViT/README.
2. Install dNaViT-local dependencies (separate from sft/ per repo layout).
3. Run reconstruction eval scripts on a held-out image folder before you plug codes into a generative head.
4. Compare token rate and VRAM against your current continuous CLIP/SigLIP pipeline on the same downstream task.

If you already serve LongCat-2.5 APIs, dNaViT is the tokenizer research lane, not a drop-in API swap.

Connections across explainx.ai coverage

• LongCat sparse attention — same lab, different layer of the stack.
• Minimax H3 open image weights — alternative open vision generation line.
• Meta Muse wearable agents — consumer multimodal, mostly closed.
• WorldFoundry world-model benchmark — when your tokens must predict physics, not just pixels.

Integration patterns for multimodal apps

Three patterns map cleanly to production:

Understanding-only — swap your vision tower input with dNaViT codes + embedding table, keep LM frozen initially.

Generation — train or adapt an autoregressive head predicting visual token indices, similar to LongCat-Next training recipes in the parent repo.

Editing — condition on partial code sequences plus text instructions; the paper lists a dedicated editing benchmark row.

For agent harnesses, export token IDs + spatial metadata so tools can crop regions without re-encoding pixels each step — important for Gemini widget demos-style interfaces you may compare against.

Licensing and deployment notes

README states MIT for core LongCat-Next weights with dNaViT/ under Apache-2.0. Legal review should catch:

• Trademark restrictions on Meituan branding in your UI.
• Model card attribution when you ship fine-tuned tokenizers.
• Export control if you deploy in regulated regions — orthogonal to code license but common in China-origin open weights reviews.

FAQ for practitioners

Do I need LongCat-Next weights to use the tokenizer? Read dNaViT/README: reconstruction eval can run with tokenizer checkpoints before you train a full DiNA stack.

Will this run on Apple Silicon? Expect PyTorch + CUDA assumptions in research code; plan a cloud GPU finetune unless README lists MPS support when you clone.

How many visual tokens per 1024² image? Scales with downsample factor f=32 in the paper''s main config — count tokens before you promise real-time mobile generation.

Can I pair dNaViT with Claude or GPT vision APIs? Tokenizer is open; API pairing requires you to export embeddings into your own serving graph, not a plug-in swap.

Is this competitive with Gemini image stacks? Benchmarks in the paper claim strong editing and bilingual text scores; verify on your locale and font coverage.

LongCat ecosystem context

Meituan has been stacking open multimodal pieces across explainx.ai coverage:

• LongCat-2.5 API for agent serving.
• LongCat 2.5 1.6T agent model for sparse attention at scale.
• LongCat 2.0 MoE coding agent for software-focused agents.

dNaViT is the vision token layer underneath native multimodal training — not a chat API by itself. Product teams should slot it into data pipelines first, user-facing APIs second.

Hardware and throughput planning

Discrete autoregressive vision increases sequence length versus continuous patch embeddings. Before production:

• Estimate tokens per megapixel using the paper''s f=32 downsampling story.
• Profile embedding table memory for codebook size after grouped product quantization.
• Compare batch latency against your current SigLIP + cross-attn stack on identical prompts.

If throughput fails, hybridize: continuous encoder for retrieval, dNaViT codes for generation, accepting two forward passes until unified checkpoints mature.

What to watch on Hugging Face and GitHub

Watch meituan-longcat/LongCat-Next for:

• Tokenizer checkpoint revisions and vocab size changes that break saved indices.
• Issue tracker threads on ResMod training instability or codebook collapse.
• Community forks that pair dNaViT with Qwen3.5 or DeepSeek LMs — useful canaries for transfer before you commit.

When benchmarks update, re-run your document-OCR suite first; that is where discrete vision tokenizers historically underperform continuous towers.

Comparison to other open discrete vision lines

Chameleon and Emu3 proved unified autoregressive multimodal training is possible; dNaViT stresses native resolution and grouped product quantization rather than a single VQ bottleneck. Against Minimax H3 image weights, Meituan optimizes tokenizer generality first, single-shot T2I scores second — pick your integration path based on whether you need one LM vocabulary or best still-image FID today. Re-benchmark quarterly if you ship both understanding and generation in one product surface.

Limits and honest skepticism

• Benchmark tables need your own domain shift tests (medical, industrial, low-light).
• Discrete autoregressive vision still pays sampling cost at high resolution — count tokens per megapixel before production.
• License split (MIT vs Apache-2.0 in dNaViT/) matters for enterprise redistribution — read both LICENSE files.

Pin tokenizer commit hashes in your internal model registry the same way you pin LM revisions, because discrete indices are not backward compatible when codebooks change over time.

Related reading

• LongCat-2.5 API launch
• LongCat 2.5 1.6T agent model
• Minimax H3 open image model
• Multimodal Knowledge Agent benchmark
• Frontier models compared hub
• Hugging Face Hub marketplace billing

arXiv v1 and LongCat-Next README checked November 11, 2026.