explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • What problem does Streaming DiLoCo solve?
  • How DiLoCo works, in simple terms
  • What does "streaming" add?
  • The numbers Google reports
  • Does model quality hold up?
  • How to try it in MaxText
  • Who should care, and who should wait
  • What is claimed, and what is not shown
  • What to watch next
  • Related reading
← Back to blog

explainx / blog

Streaming DiLoCo in MaxText: Train LLMs Across Metros With 65x Less Traffic

Google, MaxText, Distributed Training, TPU, Research

Part of Google Gemini and DeepMind

Google added Streaming DiLoCo to MaxText on Nov 19, 2026. On two TPU v5p slices it cut cross-slice traffic 65x and ran 9.57x faster than data parallel at 10 Gbit/s.

Nov 20, 2026·11 min read·Yash Thakker
add explainx.ai
go deep
Streaming DiLoCo in MaxText: Train LLMs Across Metros With 65x Less Traffic

Google put Streaming DiLoCo into MaxText on November 19, 2026. The technique lets you train one language model across TPU slices that sit in different data centers, or different metros, without a very fast link between them. In Google's own benchmark, cross-slice traffic fell about 65x, and training ran 9.57x faster than standard data parallel training on a throttled 10 Gbit/s link. Google says the loss matched the baseline.

This is a tooling release, not a new model. But it speaks to a real limit on frontier training: it is getting hard to put every accelerator in one building. Our reading is that Google is turning a research idea from its own labs into a config flag that any MaxText user can try. The code ships in MaxText v0.2.5 on PyPI, in the matching Docker images, and on GitHub main, per the Google Developers Blog post by Mohit Khatwani, Chris Zuo, Nova Fallen, Zachary Garrett, Zachary Charles, Alex Shraer, Deepak Patil and Andi Gavrilescu.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

What problem does Streaming DiLoCo solve?

Normal large-model training keeps all accelerators in one tightly coupled cluster. After each step, every device shares gradients with every other device. That works when the links are fast. Google's post says physical and cost limits make one ever-larger slice harder to build, so teams want to connect several slices across data centers.

The catch is the wide-area network. Inside one data center, the data center network is fast enough that JAX and XLA can hide the gradient exchange behind backward-pass compute. Across data centers or congested links, bandwidth drops and latency rises. Standard data parallel training then waits at every step for gradients, and model FLOPS utilization collapses. In plain words, expensive chips sit idle while data crawls over a thin pipe.

If you follow chip supply and power news, this fits a wider pattern. Our posts on Google's TPU 8 variants, the Cloud Next TPU 8 launch and the IEA data center outlook all point to the same constraint: compute and power come in pieces, in many places. Software that stitches the pieces together is worth watching.

How DiLoCo works, in simple terms

DiLoCo stands for Distributed Low-Communication. Google DeepMind introduced it in the paper DiLoCo: Distributed Low-Communication Training of Language Models. The idea has two loops.

  • Inner loop, local. The training run is split into M independent groups called learners. A learner is usually one TPU slice. Each learner runs a normal optimizer, typically AdamW, on its own data for H steps. In Google's experiments H is between 37 and 65. During those steps the groups send nothing to each other across the slow link.
  • Outer loop, global. Every H steps, each learner computes the difference between its current weights and the global weights from the start of the interval. The learners average these differences with an all-reduce. An outer optimizer, typically SGD with Nesterov momentum, then updates the global weights. Those weights start the next interval.

Because the groups talk H times less often, total communication drops by a factor of H, which Google describes as one to two orders of magnitude. The published results say model quality stays comparable.

What does "streaming" add?

Vanilla DiLoCo has one flaw. Every H steps, every learner sends the whole model's weight changes at once. On a thin link that creates a big periodic stall. Streaming DiLoCo, described in the paper Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch, fixes this by splitting the model into P disjoint fragments. Across H steps, each fragment syncs exactly once. The outer-loop cadence is the same, but the network sees a steady small stream, not a spike.

Google's MaxText version makes several concrete choices:

  1. Fragments. MaxText uses P equal to the number of layers plus one. One fragment holds the non-scanned parameters: token embeddings, final layer norm and output head. Each scanned transformer decoder layer gets its own fragment.
  2. Staggered sync. MaxText syncs one fragment every max(1, round(H/P)) steps. When H equals P, one fragment syncs on every step.
  3. Interleaved order. The order is spread across depth, for example fragment 0, then layer k, then layer 2k, so each round touches different depths and no part of the network goes stale.
  4. Embedding load balancing. Fragment 0 can be much larger than the others because it depends on vocabulary size. Large matrices from it are split evenly and merged into the layer fragments, so no single step creates a traffic spike.
  5. CPU syncers. Weight changes for the scheduled fragment move to M-way sharded syncer workers on the learner host CPUs. They run the distributed all-reduce in the background while the TPUs keep training, then apply the outer Nesterov update and write the fragment back. There is no central parameter server.

The numbers Google reports

All figures below come from Google's own post. They are the vendor's benchmark, run by the team that built the feature. We did not rerun them.

Google first gives a worked example with Qwen3-32B in bfloat16 and a step time of about 6.2 seconds, with P and H both 65. Data parallel exchanges about 65.5 GB of gradients every step and would need at least about 85 Gbps of sustained cross-slice bandwidth to avoid stalling. On a 10 Gbps link, the transfer would take over 50 seconds. Streaming DiLoCo sends about 1.0 GB per step, which needs only about 1.3 Gbps.

Then Google measured it. The test used Qwen3-32B with 64 decoder layers and 65 fragments, on two TPU v5p-32 slices. Linux traffic control limited the link between slices to three regimes: unthrottled, 30 Gbit/s and 10 Gbit/s.

Step time across throttled network links for Streaming DiLoCo on TPU v5pStep time across throttled network links for Streaming DiLoCo on TPU v5p Image: Google Developers Blog

table · 4 cols
Network regimeData parallelVanilla DiLoCoStreaming DiLoCo
Unthrottled5.58 s per step5.87 s5.76 s
30 Gbit/s19.41 s6.39 s5.76 s
10 Gbit/s55.11 s7.55 s5.76 s

Source: Google Developers Blog, step times read from its Figure 2.

Read the table this way. With no throttling, XLA overlaps the data parallel exchange with compute, and all three methods are close, between about 5.6 and 5.9 seconds. At 30 Gbit/s, data parallel already saturates the link and slows to 19.41 seconds, while Streaming DiLoCo stays flat at 5.76 seconds, a 3.37x speedup. At 10 Gbit/s, data parallel slows to 55.11 seconds. Vanilla DiLoCo does far better at 7.55 seconds, a 7.30x speedup, but still pays a full-model stall every 65 steps. Streaming DiLoCo stays at 5.76 seconds, a 9.57x speedup and an 89.5 percent cut in step latency.

Streaming DiLoCo cross-slice communication volume versus data parallel in MaxTextStreaming DiLoCo cross-slice communication volume versus data parallel in MaxText Image: Google Developers Blog

Traffic is where the 65x comes from. Over 65 steps, data parallel moved 4,258.8 GB across the slices. Streaming DiLoCo moved 67.0 GB. That is the 64x to 65x reduction in Google's post, and it matches the simple math: one 65th of the model per step against the full gradient set.

One thing to note: the 65x is a ratio of traffic that follows from picking H and P at 65. A larger sync period would cut traffic more, and a smaller one less. The number says more about the settings Google chose than about a fixed property of the method.

Does model quality hold up?

The common worry with low-communication training is that independent learners drift apart. That worry is sharper for mixture-of-experts models, where expert routers could diverge. Google compared Streaming DiLoCo with synchronous data parallel pretraining on two MoE models in MaxText, Qwen3-30B-A3B and Gemma4-26B-A4B, over 4,000 steps. On Qwen3-30B-A3B, after some early adjustment in the first roughly 800 steps, Streaming DiLoCo tracked and slightly beat the data parallel loss curve. On dense Qwen3-32B, with 65 fragments, Google says it matched the data parallel baseline loss with no loss in quality.

Two design details are credited with the stability. First, an outer Nesterov momentum optimizer with beta 0.9 and an outer learning rate between 0.3 and 0.7, applied to fragment pseudo-gradients. Second, warm inner optimizers: a sync only updates model parameters, so each learner keeps its AdamW moment buffers and does not lose momentum after a fragment sync.

The limit here is scale and length. 4,000 steps is a short run next to a real pretraining job, and the models are 26B to 32B parameters. The post does not claim results for trillion-parameter runs or multi-week training. Treat "matches the baseline" as a result for these runs, not a guarantee.

How to try it in MaxText

MaxText gives a native JAX and Flax NNX implementation of both vanilla and streaming DiLoCo. Google's config uses these settings:

  • enable_diloco: true and enable_streaming_diloco: true
  • num_diloco_fragments and diloco_sync_period both set to layers plus one. Google's examples are 37 for Qwen3-8B, 49 for Qwen3-30B-A3B and 65 for Qwen3-32B.
  • use_sequential_layers: false, so layers are interleaved across fragments, which Google recommends.
  • diloco_outer_lr: 0.7 and diloco_outer_momentum: 0.9
  • dcn_diloco_parallelism: 2 for two slices, or -1 to infer it from the slice count.
  • Optional dcn_bandwidth_limit: "10gbit" to emulate a slow link on a fast cluster using Linux tc.

Google points to a theory and link-sizing reference, a DiLoCo pretraining tutorial with sample recipes for Qwen3 8B and 30B, and two launch scripts: run_spmd_streaming_diloco.sh for dense models and run_olmo_qwen3_30b_streaming_diloco.sh for the MoE case. The emulation flag is a cheap way to test your own model before you rent a second site.

Who should care, and who should wait

Teams with TPU capacity in more than one place are the target. If your slices sit in different zones or data centers and the link between them is slow, this is the first option from Google that ships in the main framework.

Teams on one big cluster will see little change. The unthrottled column in Google's table shows all methods within a few percent when the link is fast.

Researchers on decentralized or volunteer training have followed DiLoCo for a while. Compare this with our look at Nvidia and Pair's open-source distributed home AI, which attacks a related problem from the inference side, and with our guide to AI chip architectures for why interconnect speed decides so much about training design.

Anyone training smaller open models may also like the full train-and-deploy loop covered in our post on Prime Intellect Model Factory, and the RL weight-sync work in Nvidia Dynamo for RL rollouts. Both deal with moving weights between machines, the same cost that DiLoCo attacks.

What is claimed, and what is not shown

  • Shown by Google: the traffic counts, the step times under traffic-control limits, and the loss curves for three open models.
  • Not shown: a run over a real long-distance link. The 30 and 10 Gbit/s numbers come from throttling a link, and real metro links add latency, jitter and loss that a throttle does not model.
  • Not shown: results beyond 4,000 steps, or on models much larger than about 32B parameters.
  • Not shown: a cost comparison. The post gives times and traffic, not dollars.
  • Not from Google: some third-party summaries say the method works across cities hundreds of miles apart without dedicated optical fiber. We did not find that claim in Google's post, so treat it as an unverified summary. The technique is designed for low bandwidth, but Google's own evidence is the emulated-link test above.

The method is also not new in concept. The original DiLoCo paper dates to 2023 and the streaming paper to January 2025. What is new on November 19 is the native implementation in MaxText, with the CPU syncer design, fragment load balancing and published configs.

What to watch next

The first thing to watch is whether an outside team reproduces the 9.57x number on a real inter-data-center link. The second is whether Google reports a large run, for example a production model trained on two sites. The third is whether other frameworks, such as those for GPU clusters, copy the fragment and CPU-syncer approach. For now, the practical step is simple: install MaxText v0.2.5, turn on the emulation flag with your own model, and see whether your step time stays flat at 10 Gbit/s.

Related reading

  • Google TPU 8 variants
  • Google Cloud Next 2026: TPU 8 and Gemini Enterprise
  • AI chip architectures compared
  • Nvidia and Pair: open-source distributed home AI
  • Prime Intellect Model Factory
  • Nvidia Dynamo and RL rollout bottlenecks
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 20, 2026

Google Broadcom TPU 8i Cut: 1M Fewer 2027 Chips — What Kuo Reported vs Google Silence

On November 20, 2026 supply-chain analyst Ming-Chi Kuo said Google cut planned 2027 shipments of Broadcom-designed TPU 8i chips by roughly one million units, mostly Whalefish 8i Max, while planning more MediaTek-designed TPU 8t silicon. Google has not confirmed the numbers.

Nov 16, 2026

PaperOrchestra: Google Research's Multi-Agent LaTeX Paper Writing Framework

PaperOrchestra is a Google Cloud AI Research open-source pipeline that converts raw research materials into conference-template LaTeX, with specialized agents for outline, literature, writing, refinement, and plotting.

Nov 9, 2026

Google TPU 8tx and 8i Max: Four Eighth-Gen Variants in One Year

On November 9, 2026 reporting attributed to Bloomberg described two new Google TPU system variants, 8tx and 8i Max, joining 8t and 8i in a four-SKU eighth generation. They target confidential compute and memory-heavy agent workloads rather than new silicon.