Marin Mochi 535B: The Fully Open Hero Run Reportedly Finishes Pretraining, With Checkpoints Due Next Week

Marin's 535-billion-parameter open model has reportedly finished its main pretraining run. According to posts on X on November 5, 2026, the Mochi 535B-A23B mixture-of-experts model processed about 18 trillion tokens and now moves to long-context training and supervised fine-tuning. The team's lead said it hopes to release checkpoints next week, after "some EU paperwork." The project's own site still says the run is under way, so treat the finish as reported, not yet confirmed on the official page.

What is solid is the plan and the public record. Marin has shared the full build log of this "hero run" since August: the architecture, the data pipeline, the schedule and the risks. That makes this story unusual. You can check most claims against primary documents, and this post does.

What is confirmed and what is not

The project's own tracker does confirm the finish. A November 5 comment on issue 8435 from Marin contributor ClassicLarry opens with "The 535B-A23B hero finished: 390,251 steps, 18.0T tokens." It reports a final Paloma macro loss of 1.982 at 4K context, against a preregistered forecast of 2.039. That comment says nothing about next steps, the checkpoint date or EU paperwork; those come from the X posts, which we could not open. The project site still says "Right now, we're training a 535 billion parameter mixture-of-experts model," and will likely change.

What is Marin?

Marin is an open lab, described by one outlet as coming out of Stanford's Center for Research on Foundation Models. Its pitch is simple: open means sharing everything. The project site lists the code (models and infrastructure), the model weights, the source of every dataset in the training corpus, and "the results of our successful and unsuccessful experiments."

That is a higher bar than most releases called open source. A typical open-weights release gives you the final checkpoint and a model card. You cannot see the data mix, the failed runs, or why a decision was made. Marin tries to publish all three, and it has done so for earlier models: an 8B dense model and a 32B dense model, each with a written "retro" on how it was built.

We covered the lab's wider output in Laude Slingshots: 14 open research projects, where Marin's pedagogical RL work sits next to other open efforts.

The architecture, from the public build log

The most useful document is the job summary posted on the GitHub tracking issue on August 19, 2026. It gives a model spec that you can check against the W&B dashboards. The numbers below are from that summary:

• Size: 535.3B total and 22.76B active parameters per token.
• Depth and width: 48 layers, hidden dimension 6144, 48 attention heads of dimension 128.
• Experts: 384 routed experts with 8 active per token, plus 2 shared dense experts. Every block uses a mixture-of-experts layer, with no dense-only layers.
• Attention: grouped-query attention with split local and global key-value heads. Local layers use a 2,048-token sliding window, and every fourth layer is global.
• LatentMoE: tokens are compressed from 6144 to a latent size of 3072 before the experts, then expanded. The same idea appears in other recent open designs; Sebastian Raschka's breakdown of Kimi K3's LatentMoE explains the trade-off.
• Routing: a balancing scheme the team calls QB routing, which sets a per-expert bias from a histogram of routing margins and applies it on the next step, without touching gradients.
• Context during pretraining: 4,096 tokens, chosen on purpose to give better expert balance. Longer context comes later.

If those terms are new, our entry on mixture-of-experts is a good start. In short, an MoE model has many specialist sub-networks, and a router picks a few for each token. Only 23B of the 535B parameters run for any one token, so compute per token is closer to a 23B dense model while the total knowledge stored is far larger.

Diagram of a router sending a token to a few small experts, illustrating how a 535B mixture-of-experts model activates only 23B parameters per token

The data: 25.6T raw tokens, 23.1T after cleaning

A second comment on the same tracker, from a Marin contributor, describes the corpus. It starts from a raw pool of 25.6 trillion tokens across 292 sources, logged in a public provenance table on Hugging Face. More than 100 of those sources are rollout datasets from agent runs.

The team then removed 2.494T tokens with fuzzy deduplication and n-gram decontamination against common benchmarks in the LM evaluation harness and the Artificial Analysis Intelligence Index. About 250,000 documents were dropped for that decontamination step. That leaves 23.106T tokens to sample from, of which the run uses about 18T.

Documents were embedded with Microsoft's Harrier 0.6B model and grouped into 40 topic buckets. Quality labels came from a GLM 5.2 judge and were distilled into a smaller classifier. Two-phase mixing follows: according to a secondary summary, 80% of tokens are pretraining and 20% are midtraining. That split is from a third-party write-up, so check the tracker for the final mix.

Why this matters: decontamination against named benchmarks is exactly what outsiders cannot verify in most closed models. Here you can read the list and the method, then judge the final scores accordingly.

The compute: 11 GB200 racks for about 100 days

The tracker says the run uses 11 NVIDIA GB200 NVL72 racks for roughly 100 days, for about 2.7e24 training FLOPs. The team wrote its own expert-parallel implementation, because it found no faster option for JAX on XLA for GPUs. The tracker links a write-up of that design, and Marin's stated policy is to publish its code.

Two design choices stand out.

A scaling ladder at 1% of compute. Before the big run, the team trained four smaller models, from 1.6B total (61M active) on 48B tokens up to 27.7B total (1.2B active) on 926B tokens. The tracker lists five reasons: project final quality against earlier recipes, watch gradient norms and token dropping, project every eval across the whole run, keep a small-scale reference for odd behavior, and do it for about 1% of total compute. In a previous run, the ladder caught gradient norms growing past 4, which led to a fix with logit z-loss. Without it, some high-batch runs would blow up mid-run.

Token dropping as a known risk. Expert parallelism with fixed capacity drops tokens that overflow an expert. The tracker says dropping rose from about 7% to about 40% when an earlier setup moved from 4K to 65K sequences, and that the pre-run estimate for the new pooled wave design was about 3% at 4K (a later comment mentions a switch to near-dropless expert parallelism mid-run). The team kept two shared experts so that a strong backbone remains if dropping is high, and reports learning still progressed "with zero loss spikes even at 40 pct dropping" in tests.

The same log also plans for delay: if hardware failures cost up to about 25% of the token budget, the team will shorten the token horizon and change the learning-rate decay, not extend the schedule.

What happens between "pretraining done" and a release

Based on the tracker's plan, the steps after the 4K-context pretraining are:

1. Context extension. The plan was 4K to 8K around the middle of the run, 8K to 65K near the end, and a targeted push toward 262K, using the attention scaling formula described in the issue.
2. Supervised fine-tuning. The reported plan is long-context training, then SFT, before general release.
3. Post-training and RL. Marin has also released a "posttraining data commons" and pedagogical RL work, which hint at how it may tune this model.
4. Weights. The team's hope is to release checkpoints next week. Which checkpoint (base, midtrained, or tuned) is not stated.

That is why we avoid calling this a release. A 535B base checkpoint is valuable to researchers, but it is not a chat model, and the quality of an MoE base can only be judged with evals that are not published yet.

The EU paperwork angle

The reason given for the wait is "some EU paperwork." No public document says which filing that is. The European Union's rules for general-purpose AI models ask model providers to document training data and compute, and large models can face extra duties. We cannot confirm that this is the cause. What we can say is that a small open lab is delaying a release for compliance, which is itself news for anyone planning an open model in 2026. Compare how other labs handled regional rules in our coverage of Baidu's ERNIE 5.5 Flash Base open weights.

How it compares with other open MoE releases

The table compares only what each source states. It does not rank quality, because Marin has published no benchmark scores for this model.

Illustration of mixture-of-experts routing that picks a few experts per token, as used in Marin's 535B-A23B model

What to watch next

• The checkpoint drop. Look for a Hugging Face repo under the marin-community organization and a model card with a license.
• The first evals. Base-model scores on standard suites, and whether the team publishes the decontaminated test sets.
• The loss curves. The public W&B report lets anyone compare the real run with the scaling-ladder projection.
• Inference support. Runtimes such as llama.cpp need work for a new MoE layout; see the notes in llama.cpp v0.9.0 for how multi-node support is evolving.

Bottom line

Marin's hero run is the most open attempt at a frontier-scale model that we know of: the plan, the data pipeline, the failure risks and the live curves are all public. Main pretraining is reported done, and a checkpoint release may come within days. Until the weights, license and evals appear, the honest status is this: a credible, documented run that has reached a milestone, with the final quality still unknown.

Related reading

• Laude Slingshots: 14 open research projects, including Marin
• Kimi K3 architecture: LatentMoE and NoPE explained
• Baidu ERNIE 5.5 Flash Base: 280B open MoE
• Meta Muse Spark open: 816B open-weight model
• Cognition SWE-2 Flash: claimed vs verified
• llama.cpp v0.9.0: what changed