Radical Numerics has published Halcyon, a family of ten open text-to-image models that were trained entirely from scratch on image-text pairs, with no internet-scale language pretraining. According to the company's October 15 research post, the 1.6B model beats a strong diffusion baseline using 275 times less pretraining compute, and a 0.52B model beats LlamaGen with 18.2 times less. The checkpoints are on Hugging Face and the work is described in a preprint titled "Text-to-Image Generative Modeling without Language Pretraining."
The headline is striking and the experiments are better controlled than most. But the models are small, the images are 256x256, and the savings depend on how compute is counted. This post separates the three.
TL;DR
| Item | Detail |
|---|---|
| Who | Radical Numerics (the lab behind the Omnii genome language models) |
| Models | Ten Halcyon models, 0.1B to 4B parameters; five sizes in the paper |
| Architecture | Decoder-only Transformer (Qwen3 architecture, random init), text and image tokens in one sequence |
| Data | 100M image-text pairs from GPIC, 256x256, one epoch (97,632 steps at batch 1,024) |
| Key result | Halcyon-1.6B: 35.3 gFD vs 41.0 for JiT-T2I-XXL/8++, 275x less total pretraining compute |
| Second result | Halcyon-0.52B: 48.0 vs 63.3 gFD against LlamaGen-XL/8, 18.2x less compute |
| Limit | 256x256 images; benchmark scores far below frontier generators |
| Released | Models, tokenizers and training recipes |
What "early fusion without language pretraining" means
Most text-to-image systems use a language model that was pretrained on web-scale text. In "late fusion," that language model is separate: its output embeddings steer an image generator through cross-attention. In "early fusion," one model handles both text and image tokens, but it is still usually pretrained on text-only data first. In both designs, the post says, language pretraining dominates the compute budget.
Halcyon asks a blunt question: is that language pretraining necessary? It trains everything from scratch on GPIC, including the image tokenizer, the text tokenizer and the generator. Text and images share one vocabulary of 80,640 entries (16,384 text tokens, 64,000 image tokens and special tokens), and the model is trained by next-token prediction. An image is a 32x32 grid of 1,024 tokens. That is the same autoregressive trick language models use, applied to pictures.
The headline results, with the numbers
A paintbrush stroke turning into shapes, illustrating Halcyon text-to-image generation from image tokens
The authors retrained two representative baselines on the same 100M-pair dataset so differences come from design, not data:
- LlamaGen-XL/8: an autoregressive model with a 1.22B FLAN-T5-XL text encoder and a 0.90B image generator. Halcyon-0.52B scores 48.0 gFD versus 63.3 (lower is better) with 18.2x less total pretraining compute.
- JiT-T2I-XXL/8++: a pixel-space diffusion model with a 1.7B Qwen3 text encoder and 1.1B image generator. Halcyon-1.6B scores 35.3 versus 41.0 with 275x less.
On prompt-following benchmarks, Halcyon-0.52B beats LlamaGen-XL/8 on GenEval (44.91 vs 30.48) and DPG-Bench (76.84 vs 67.08), and ties JiT on DPG-Bench (76.84 vs 76.85). Halcyon-1.6B reaches 53.48 on GenEval and 77.90 on DPG-Bench, and the 4B version 56.39 on GenEval with DPG-Bench staying at 77.90. The quality metric improves steadily with size: gFD falls from 144.2 at 0.1B to 28.4 at 4B.
Does language pretraining help? Three tests
The most interesting part of the post is a set of controlled experiments on whether text pretraining helps:
- A bigger text encoder. In LlamaGen, swapping FLAN-T5-Base (0.11B) for FLAN-T5-XL (1.22B) improves gFD only from 66.7 to 63.3, despite 13.5x more language-model pretraining compute.
- Text-first pretraining. Giving 10%, 20% or 50% of a fixed budget to text-only pretraining made things worse (48.0 to 51.6, 52.8 and 65.1). Adding text-only pretraining before a full image epoch, at 1.5x compute, did not help (49.2 vs 48.0).
- Continual pretraining. Starting from Qwen3-0.6B-Base, pretrained on 36T text tokens, gave no gain over training from scratch (47.8 vs 46.6).
The authors conclude that "language pretraining does not improve T2I generation" in their setting. That qualifier, in their setting, carries real weight. Captions in GPIC are produced by a vision-language model and are fairly regular. Complex, unusual or long prompts from real users may benefit from the world knowledge a large language model carries. The tests here use GenEval and DPG-Bench, which probe composition and long prompts, but at 256x256 and with modest sizes.
Captioning as a free by-product
Classifier-free guidance usually requires training some examples with the caption dropped. Halcyon moves the caption after the image instead of deleting it, so the same sequence trains both the unconditional image branch and image captioning. The model learns to caption: image-conditioned text loss drops from 18.41 to 1.10 at 0.52B. Caption alignment (scored by Qwen3-VL-Reranker-8B) rises from 0.09 at 0.1B to 0.56 at 4B, against 0.89 for real captions. So captioning works but is far from human quality, and image generation quality holds.
Why a biology lab is doing this
Radical Numerics describes itself as an AI lab for general biological intelligence, and the blog ties Halcyon directly to that. Its Omnii genome language models are trained on dozens of heterogeneous modalities, each with different statistics. Biology has no equivalent of the web-scale text corpus that makes language models easy to bootstrap, so the lab wants to know how modalities should be combined and how compute should be split between them. Text-to-image is a convenient test bed because public datasets, benchmarks and strong baselines exist.
That context explains the framing. The finding is less "you should train image models without language models" and more "do not assume a text-pretrained component is carrying weight in a multimodal system until you have measured it." The same logic applies to any pairing of modalities where one is data-rich and the other is scarce. If the result holds, a modality that has little data of its own may not need a data-rich neighbor to be modeled well, at least at small scale.
How compute is counted, and why that matters
The 275x figure counts the whole pretraining cost of the baseline: the language model, the image tokenizer and the image generator. A late-fusion system gets a language model "for free" in many products because it can reuse a model someone else trained. If you already have that encoder, your marginal cost is lower than the headline suggests. Halcyon's claim is about total cost from raw data, which matters for labs training from scratch, and for the scientific question of whether the text component is doing real work.
Other caveats readers should hold onto:
- Resolution. Everything is 256x256. Commercial generators work at much higher resolutions, where tokenizers, detail and text rendering get harder. See our coverage of Krea 3's sub-8-second 2K edits and FLUX.3 Klein open weights for what current products deliver.
- Single dataset. Models, baselines and metrics all center on GPIC, a permissively licensed 100M-image corpus. That is a cleaner test but a narrower one.
- Preprint. The results come from the authors' paper and blog, not from independent replication. The Hugging Face card also notes that its main branch is a slightly different configuration from the paper's table, so scores may not match exactly.
- Not a product. These are research checkpoints for studying pretraining, and the lab's main focus is biology, as its Omnii models show.
How to try it
- Load a checkpoint with transformers:
AutoModelForCausalLM.from_pretrained("radicalnumerics/Halcyon-4B", dtype="bfloat16"). Abaselinerevision matches the paper's standard recipe. - Use the separate tokenizer repository (
radicalnumerics/Halcyon-Tokenizers) and the documented token layout: caption tokens, a start-of-vision token, then 1,024 image tokens. - Sample with classifier-free guidance against the bare start token. The model card lists a guidance scale of about 4.5 for the 4B, and the post says the best scale falls from 7.5 at 0.1B to 4.5 at 4B.
- Treat it as a research baseline. If your goal is production images, a hosted or open product model is a better fit today.
What this means for what you build
- Small-lab image training. If the result generalizes, teams without a large pretrained language model can train competitive small image models from image-text pairs alone. That lowers the entry cost for domain-specific generators.
- Multimodal training recipes. Radical Numerics frames this as groundwork for models that mix many modalities, as in biology, where there is no giant text corpus to start from. That is a useful lens for any domain with scarce text.
- Compute accounting. Count total pretraining compute when comparing systems; reusing someone else's language model hides a cost.
For related efficiency claims, see Rigel, a 2.3B model trained with under 1% of the usual pretraining compute and Apple's tri-modal masked diffusion paper. If you generate images with commercial models, also see the jailbreak findings in White Circle's image model tests.
Claimed versus verified
| Claim | Source | Status |
|---|---|---|
| Ten open models, 0.1B to 4B | Radical Numerics post, Hugging Face | Checkable on Hugging Face |
| 275x and 18.2x less total pretraining compute | Radical Numerics | Authors' own compute accounting; not independently replicated |
| Better gFD, GenEval, DPG-Bench than the retrained baselines | Radical Numerics | Same-dataset comparison; preprint |
| Language pretraining does not help T2I | Radical Numerics | True in their 256x256, one-epoch setting only |
| Beats the best commercial generators | Nobody | Not claimed |
Details are from Radical Numerics' October 15, 2026 post and Hugging Face model cards, and may change.