Edge0 Open-Sources a 35B Model That Runs in 2.5GB of Memory

A 35-billion-parameter language model, running in 1-2.5GB of peak memory. That's the headline claim behind Edge0, an on-device AI framework founded by Samuel Zeng that open-sourced its runtime, inference code, and preview model weights on September 10, 2026 under an Apache-2.0 license. The technique — streaming model weights from storage on demand instead of loading everything into RAM — is a genuinely different approach to on-device efficiency than most "small model" releases, which usually just mean fewer total parameters.

TL;DR

How "SSD expert offload" actually works

Most large language models today assume the entire set of weights is loaded into fast memory (RAM or GPU VRAM) before inference starts — that's why running a 35B-parameter model conventionally requires tens of gigabytes of memory, since every parameter needs to be immediately accessible for every token generated. Edge0's approach breaks that assumption.

The framework keeps the complete model on storage (disk or flash) rather than loading it wholesale into memory. For each inference step, it identifies which specific parameters that step actually needs — likely a subset of "expert" parameters in a mixture-of-experts-style architecture — loads just those into memory, and begins computing with them. Critically, while that computation is running, the framework prefetches the parameters the next step will need, overlapping the slow storage read with the fast compute step so the pipeline doesn't stall waiting on disk I/O each time.

This is conceptually similar to how virtual memory and paging work in traditional operating systems — treating storage as an extension of memory rather than a completely separate tier — applied specifically to the structure of a mixture-of-experts model, where different "experts" (subsets of parameters) are relevant to different inputs and don't all need to be resident simultaneously.

The real tradeoff: memory footprint versus throughput

This technique doesn't come free. Streaming parameters from storage, even with prefetching, is inherently slower than having everything already resident in fast memory — which is reflected directly in Edge0's own benchmark numbers: roughly 15-18 tokens per second for the 35B model, a usable but noticeably more modest speed than what the same parameter count would achieve with full memory residency on capable hardware.

That tradeoff is the entire point, though, not a limitation to apologize for. The target use case isn't "fastest possible inference" — it's running a genuinely large model on hardware that couldn't otherwise run it at all. A 35B-parameter model needing 60-70GB+ of memory in a conventional deployment is simply not an option on a phone, a laptop with modest RAM, or many edge devices; a 35B model needing under 3GB opens up an entirely different category of hardware, even if each token takes a bit longer to generate.

What's actually shipped versus what the demo showed

It's worth being precise here, because the promotional framing (an iPhone in the demo video) and the actual public release (macOS Apple Silicon only, per the GitHub README at the time of writing) describe two different things. The README's installation instructions target macOS specifically, list M1 through M4 Apple Silicon as the supported hardware, and note that a CUDA backend — which would be needed for most Android phones, Windows PCs, and Linux systems — is planned but not yet available. There's no documented iOS build path in the initial open-source release.

That gap between demo and shipped artifact is common enough in early-stage AI releases that it's worth flagging rather than assuming bad faith: iOS deployment likely requires additional engineering (App Store sandboxing constraints, a different memory-management model, potential Metal Performance Shaders integration specific to iOS rather than macOS) that a small team reasonably sequences after getting the core desktop runtime working and public. The underlying "SSD expert offload" technique is architecture-agnostic in principle — the constraint is engineering effort to port the runtime to each additional platform, not something fundamental to the technique that would make iPhone deployment impossible.

How this compares to other efficient on-device model approaches

Edge0's approach sits in a different lane than most of the "small but capable" open-weight models explainx.ai has covered — those typically achieve efficiency by training genuinely fewer parameters from the start, then optimizing training data quality and architecture to get the most capability out of that smaller parameter budget. Edge0 instead keeps a large parameter count (35B) and changes the deployment strategy, trading some latency for the ability to run a bigger model than would otherwise fit.

Both approaches are legitimate and largely complementary rather than competing: a genuinely small, dense model (in the spirit of efficient releases in the sub-5B range) will generally out-throughput a large model running via storage offload on the same hardware, but the offloaded large model may retain broader general capability that comes from having more total parameters to draw on, at the cost of speed. Which approach makes sense depends heavily on the specific application — a real-time voice assistant probably wants the faster, smaller dense model; an offline research or coding assistant that can tolerate a few extra seconds per response might prefer the larger model's broader capability.

What you can build with the initial release

Edge0's own demos alongside the open-source launch focused on two use cases that showcase the "genuinely useful, genuinely local" pitch:

• Natural-language game generation — describing a game idea in plain language and having the model generate playable game logic and code, entirely on-device, at roughly 2GB peak memory with no cloud calls.
• Structured 3D scene generation — describing a scene and having the model generate, then iteratively modify and refine, a structured 3D representation locally, with no project data leaving the device.

Both use cases lean into the framework's core value proposition — genuine privacy (nothing leaves the device) and zero marginal cost per generation (no API calls, no token billing) — which matters most for exactly the kind of exploratory, iterative creative work where cloud API costs and round-trip latency add friction.

What this means for builders interested in on-device AI

1. Try it on Apple Silicon today if you have access to an M-series Mac — the installation steps (pip install, fetch the model weights, edge0 serve) are straightforward per the published README, and Apache-2.0 licensing means you can experiment and build on it with minimal legal friction.
2. Don't plan a mobile (iOS/Android) product around this yet. The public release doesn't currently support those platforms — treat the demo video as a roadmap signal, not a shipped capability, until Edge0 publishes a mobile-specific runtime.
3. Watch the tokens-per-second numbers as the project matures. 15-18 tok/s for a 35B model is usable for many interactive applications but noticeably slower than a fully memory-resident deployment — if throughput matters more than raw parameter count for your use case, benchmark Edge0 against a smaller dense model on the same hardware before committing.
4. This is a genuinely different lever for on-device AI capacity than most releases, worth tracking alongside efficient small-model releases as a complementary strategy for pushing more capability onto consumer hardware without a cloud dependency.

Why Apache-2.0 licensing matters for this specific release

The choice of a permissive Apache-2.0 license, rather than a more restrictive custom license some recent open-weight releases have adopted, is worth calling out on its own. Apache-2.0 explicitly permits commercial use, modification, and redistribution with minimal obligations beyond preserving copyright and license notices — no requirement to share modifications back, no field-of-use restrictions, no revenue-based licensing tiers that kick in above a certain company size (a pattern several other "open" model releases have used).

For a technique like SSD expert offload specifically, permissive licensing plausibly matters more than usual: the value here is arguably as much in the runtime and inference engineering as in the model weights themselves, and a restrictive license on that engineering would meaningfully limit how quickly other projects could adopt or build on the underlying memory-management approach. Apache-2.0 licensing removes that friction, making it more likely the specific offloading technique gets adopted, adapted, or improved on by other projects — a genuinely different outcome than if the same technique had shipped under a more restrictive license that discouraged downstream experimentation.

What to watch next

• Whether Edge0 ships a CUDA backend and/or an iOS-specific runtime, closing the gap between the demo and the shipped platform support.
• Independent benchmarks of Edge0-35B's actual output quality against similarly-sized conventionally-deployed models, since a memory-efficient deployment technique says nothing on its own about the underlying model's capability.
• Whether other on-device AI projects adopt similar SSD-offload techniques for their own large-parameter models, which would suggest this becomes a broader pattern rather than a single project's approach.

Related reading

• Coral Edge AI Platform: The Complete Guide
• PrismML Bonsai 27B: A 1-Bit Ternary Model for Phones
• MiniCPM 5 1B: A Tiny AI Model Breakthrough
• A 4B Open-Source VLM Reportedly Beats Qwen 122B on GeoGuessr-Style Benchmarks
• OpenAI Partners With Samsung on Custom AI Chips

This post reflects the Edge0 GitHub README and edge0.ai product page as of September 10, 2026. Platform support, benchmark numbers, and licensing terms may change as the project develops — verify current details against the official repository before building on it.