Magnitude: Open-Source Local Inference Engine Claims 2x Faster Than Ollama

A new open-source tool wants to make your laptop or desktop a faster home for local models. Magnitude, backed by Y Combinator and licensed Apache-2.0, says it runs open models about twice as fast as Ollama by tuning its compute kernels to your exact chip. The headline on magnitude.dev reads "Over 2x faster than Ollama," with the Mac numbers listed as 1.32x faster prefill and 2.19x faster decode versus Ollama on MLX.

Local inference speed matters most for coding agents, which generate a lot of tokens and re-read long contexts. This post explains what Magnitude does, how to try it, and why the benchmark deserves a careful read.

TL;DR: Magnitude at a glance

How does it get faster?

Magnitude's pitch is that general engines ship one-size-fits-all kernels, because tuning each model for each chip by hand would take forever. Its site says Magnitude does the tuning automatically: when you download a model it spends up to about a minute on your actual chip trying different ways to split up the math, including how many GPU threads team up, how big a chunk each takes, and how far ahead to load data, and keeps "the fastest combination that doesn't change the model's output."

Other features listed:

• A built-in speculative decoding drafter shipped with each model.
• A quantized KV cache that the site says halves memory per agent and is freed when agents stop.
• Shared prefix caches so concurrent sessions do not slow each other.
• Hand-written kernels for the most popular open-weight families.
• Fully local operation: prompts, files and models stay on your machine.

The same idea, autotune per hardware, is familiar from GPU libraries; the novelty is applying it as a product for consumer machines. If you are new to the field, our explainer on what llama.cpp is and how to run models locally covers the baseline most of these tools build on.

How do you try it?

Per the site, install the desktop app from magnitude.dev, choose a recommended model in Discover, and connect your agent in Connections. The app includes the magnitude CLI; on Linux and macOS a server setup command runs it as a boot-time service so other computers can use it. A third-party listing adds an npm route (npm i -g @magnitudedev/cli && magnitude setup) and says setup profiles your hardware, ranks models by speed, accuracy and memory fit, downloads one and wires your harness to it. Check the project's own docs for the current commands, since the CLI was at an early 0.0.x version in that listing.

What does the benchmark actually show?

The site reports two comparisons, both against Ollama:

The footnote: "Average of 11 popular local models, 4-bit, 64k context, with speculative decoding."

Points to weigh before you rely on it:

1. One named baseline. The FAQ says Magnitude beats llama.cpp, Ollama and LM Studio, but the page gives figures only against Ollama. A site that cites "Ollama (MLX)" on the Mac and "Ollama (llama.cpp)" on the Spark is comparing against two different backends; that is fair for each platform but not a like-for-like across machines.
2. Speculative decoding is on for both, but not equally. The full benchmarks page says speculative decoding was on for both engines, with Ollama using the drafter shipped with each tag and Magnitude using its own catalog drafter (mostly "DFlash" on the Qwen and Nemotron models). It also notes that on the Mac, Ollama's MLX runner drafted very few tokens for Gemma 4 E4B, 26B and 31B at 64k, so those decode gains partly compare speculative decoding against almost none. Our coverage of llama.cpp's few-row MMA update shows how big drafter effects can be.
3. Averages hide spread. The benchmarks page lists the 11 models (Qwen 3.5 4B and 9B, Qwen 3.6 35B, Qwen 3.8 27B, Gemma 4 E2B, E4B, 12B, 26B and 31B, Nemotron 3.5 and Muse Glimmer). Per-model decode speedups on the Mac range from about 1.29x to 3.51x, and on the DGX Spark from 0.76x (Nemotron 3.5, where Magnitude is slower) to 4.84x. Prefill is slower for Magnitude on Gemma 4 E2B on Mac and Gemma 4 26B on the Spark. The site discloses these and includes them in the averages.
4. Decode is not the whole story. For agents, prefill dominates when contexts are long, and the prefill gain is the smaller one (about 1.3x). The workload is a single 64k-context request with 256 output tokens and greedy sampling; Magnitude says answer quality was not scored and that the engines do not run identical weights or numerics.
5. Vendor-run, but reproducible. The numbers come from the company, and we found no independent reproduction. The methodology lists Ollama v0.35.1 with default settings, a Mac mini M4 Pro with 48 GB and a DGX Spark, a median of two requests per result (some Ollama results are single requests from an earlier run), and says the benchmark harness is open source in the Magnitude repository, so you can rerun it.

A reasonable reading: the claim is plausible and consistent with how much headroom general engines leave on the table, and it is exactly the kind of number you can verify yourself in an afternoon.

How would you test it on your own machine?

1. Pick the two or three models you actually use, at the quantization you actually use.
2. Run the same prompt set, with the same context length, on Ollama and on Magnitude.
3. Record time to first token (prefill) and tokens per second (decode) over several runs after a warm-up.
4. Repeat with speculative decoding on and off in each tool, where available.
5. Test a concurrent scenario, such as two agents sharing a prefix, since the prefix cache is one of the selling points.
6. Check output quality on a few tasks; the site says tuning "doesn't change the model's output," which is worth confirming for your models.

What would a 2x decode gain mean in practice?

Speed claims are easier to judge as time saved. Suppose a local coding agent produces 4,000 output tokens for a task, which is typical for a moderate change with explanation. At a decode rate of 20 tokens per second that takes 200 seconds. At twice the rate it takes 100 seconds. For an agent that makes ten such calls in a session, the difference is about 17 minutes of waiting. Those numbers are illustrations, not measurements, but they show why decode speed dominates the feel of a local agent while prefill dominates the first-token wait after a large file read.

Prefill matters in a different way. An agent that re-sends a long conversation on each turn pays the prefill cost again unless a prefix cache catches it. Magnitude's shared prefix caches are aimed at that case, and the concurrency claim is the part of the pitch that a single-user benchmark would not capture. If you run two or three agents at once against a single machine, this is where to look for real gains or disappointments.

Who should try it, and who should wait?

Try it if you already run local models for coding agents on an Apple Silicon Mac, an NVIDIA box or an AMD GPU, and you are comfortable benchmarking your own setup. The cost is a download and an afternoon, the software is free and open source, and your data stays local.

Wait if you depend on a specific model that is not on Magnitude's supported list, if you need a long-term stable server, or if you rely on features of Ollama or LM Studio that Magnitude does not mention, such as particular model management workflows. The project is young, and a fast engine is only useful if it runs the model you need with outputs you trust.

The broader point is that local inference is no longer a matter of picking one tool and accepting its defaults. Engines now compete on per-chip tuning, drafters and cache design, and the gaps between them can be large enough to change which models are practical on a given machine. That makes your own benchmark the most valuable number you can get.

Where does it fit with other local-AI options?

The local stack is getting crowded. Nvidia is pushing its own path with the Llama Windows tray app and RTX Spark software; see our posts on the Llama Windows app and the Nemotron 3.5 Bolt model built for 24GB. Apple's MLX ecosystem is covered in our post on MLX over Thunderbolt 5 for local inference. Magnitude's distinguishing bet is a single app that handles hardware profiling, model choice, tuning and agent hookup, rather than leaving those to the user.

For hardware context on the DGX Spark used in the benchmark, see our post on the DGX Spark price and memory change.

What are the caveats?

• Early software. A third-party listing shows a 0.0.x CLI and a project created in June 2026; expect rough edges.
• Platform details vary by source. Most sources say Windows is supported; check the download page for your OS and chip.
• Roadmap items are not shipped. Posts about the project mention future attention pruning and expert streaming for models larger than GPU memory; we did not find them on the project's pages, so treat them as plans.
• Tuning time. Up to a minute per model on download, which is trivial unless you swap models constantly.
• Security. Any local server that exposes an API on your network should be bound to localhost unless you intend to share it, particularly if you enable the boot-time service for other computers.

Related reading

• What is llama.cpp? Run models locally
• llama.cpp Metal few-row MMA and speculative decoding on M3 Ultra
• Llama Windows app for RTX Spark
• Nemotron 3.5 Bolt 74B-A9B in 24GB
• MLX over Thunderbolt 5 for local inference
• DGX Spark price and memory change

Sources: magnitude.dev, the Magnitude GitHub repository, and a third-party listing for CLI and version details.

Benchmark figures are Magnitude's own as published on October 15 and 16, 2026.