llama-modchip: 64 GB CPU Path for DeepSeek-V4 and 30B MoE GGUF

The llama-modchip repository documents a CPU-first fork of the llama.cpp ecosystem that tries to run frontier-scale mixture-of-experts checkpoints on machines many developers already own — 64 GB DDR5 desktops, Threadrippers, Graviton cloud boxes, and Apple Silicon — often with zero discrete GPUs. Version v2.4.0 in the README centers on DeepSeek-V4-Flash Fractal GGUF near 56.56 GiB on disk and NVIDIA Nemotron-3.5-30B-A3B near 14.35 GiB, plus a ./modchip claude path that wires Claude Code to local inference with millisecond-scale KV restore claims.

This is not a lab press release. It is an open Apache-2.0 stack with patches, precompiled llama-server-modchip-linux-x86_64, benchmark JSON, and a community leaderboard. explainx.ai has no prior post on this repo; this guide follows the README and architecture docs as primary sources.

TL;DR

Why this repo exists

Small house-shaped box with a glowing core, illustrating frontier MoE inference on a home workstation

GPU scarcity and cloud bills push builders toward MLX on Apple Silicon and quantized open-weight licenses. llama-modchip argues the bottleneck is not only VRAM but how weights sit in DDR5 — cache tiling, MoE routing, speculative drafting, and aggressive quantization that stock llama.cpp builds do not ship.

The README’s blunt claim: an unpatched IQ2_XXS DeepSeek-V4 stack at 76.87 GiB OOMs on 64 GB Linux, while their Fractal-64 path lands near 58.9 GiB RSS at 256K context on listed Zen 5 hardware. You should reproduce those numbers; do not treat HN headlines or README tables as independent benchmarks.

The three pillars (maintainer framing)

The README organizes work into three pillars:

1. 284B frontier MoE in a 64 GB box — SwiGLU-H2 analytical compression, Fractal-64 MLA transcoding, AVX-512BW register tiling, and L3 whole-model tiling to raise prefill from 25 → ~44 tok/s in their DeepSeek tables.
2. 30B MoE at GPU-like CPU speeds — Nemotron-3.5-30B-A3B with Stockfish-PVS v2 plus RouteCrack speculative decoding (~96.4% acceptance claimed) and NTA L3 cache hints.
3. Local Claude Code — ./modchip claude with /dev/shm Slot-0 KV warm restore to cut multi-second agent re-prompt stalls.

Pillar three is why agent builders should care even if they never train: it attacks harness latency, a theme explainx.ai covers in vLLM vs SGLang and harness leaderboard reliability.

Supported models (README table)

Deep-dive docs live in docs/DEEPSEEK_V4_OPTIMIZATIONS.md and docs/NEMOTRON_OPTIMIZATIONS.md in the repo — cite those for operator-level breakdowns, not third-party summaries.

Compatibility trap: tensor type 40

Before you download multi-shard GGUFs, read the README warning:

Opening Fractal-64 models in unpatched llama.cpp, Ollama, or LM Studio fails with unknown tensor type 40.

Fix: use ./modchip, the release llama-server-modchip-linux-x86_64, or Hugging Face repos that bundle the matching binary — e.g. Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF and Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF paths quoted in the README.

If your pipeline assumes Ollama-only ops, this project is a parallel runtime, not a drop-in rename.

Quickstart (Linux)

From the README quickstart:

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

For manual HF downloads, the README prints hf download lines plus llama-server-modchip-linux-x86_64 invocations with -c context and -t thread flags tuned for Zen 5.

Readiness script (no full model download):

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

Use that before committing RAM on a laptop.

./modchip commands

Windows hybrid flows live under windows/ with automatic --n-cpu-moe splits when an NVIDIA GPU is present — different problem from pure CPU, but same Fractal weights.

Benchmark claims you should reproduce

Speculative decoding diagram with a small fast draft path and slower verify path, like Stockfish-PVS on llama-modchip

The README publishes headline DeepSeek-V4 numbers on Ryzen 9 9955HX, 64 GB, 0 GPU:

• Prefill ~43–44 tok/s (512-token prompt) for v2.4.0 lossless modes.
• Decode ~9–11 tok/s lossless; up to ~14.7 tok/s with Stockfish-PVS.
• Prob cosine ~1.0 and 0% PPL delta on “lossless” row — maintainer-measured, not third-party.

For Nemotron-30B, the same README claims ~25–60 tok/s decode on Zen 5 CPU and ~111 tok/s on Threadripper PRO 7985WX with -t 32 four-cores-per-CCD pinning — a reminder that topology tuning dominates once RAM fits.

Submit reproductions via the repo’s GitHub Benchmark Issue template if you want leaderboard credit.

Hardware guides in-repo

Green spiral through a funnel before a bead emerges, echoing test-time compute paths on CPU decode

docs/HARDWARE_OPTIMIZATIONS.md maps patches to silicon:

• Zen 5 — AVX-512 VPDPBUSD, 48 KB L1D tiling, NTA L3 victim isolation.
• Threadripper — per-CCD sysfs discovery; -t 32 beats -t 64 in maintainer Issue #1 notes.
• 3D V-Cache + Vulkan — hybrid -ngl 99 --n-cpu-moe splits.
• Graviton3 — SDOT / SVE paths with core pinning.
• Apple Silicon — Metal unified memory track under apple-silicon/.

Compare with explainx.ai’s Qualcomm Cloud AI 100 piece for edge accelerators — llama-modchip is the opposite bet: saturate DDR5 on a desktop you already own.

Quality gate and honesty about shortcuts

The README documents Turbo modes that trade small KL-divergence for speed — e.g. Top-5 L2-quadrature MoE pruning rows with ~0.4% PPL deltas in their table. For coding agents, read ./modchip eval output before you trust a fast profile on security-sensitive tasks.

Maintainers also ship 100/100 Quality Gate JSON under benchmarks/ for Nemotron Fractal profiles — run ./modchip eval after OS upgrades.

Relation to other local stacks

• MLX guide — Apple's path; llama-modchip adds apple-silicon/ but centers x86 AVX-512.
• Qwen3.5 4B GLB coder — small specialist models; modchip targets frontier MoE locals.
• vLLM vs SGLang — datacenter servers; modchip is workstation CLI.

Patches, ARCHITECTURE.md, and the paper folder

Beyond README tables, the repo ships ARCHITECTURE.md, a paper/ folder, and unified patch patches/deepseek-v4-flash-fractal-256k.patch for v2.4.0. Treat those as the maintainer’s design justification — not independent peer review.

Key engineering ideas the docs emphasize:

• Analytical SwiGLU-H2 compression — closed-form second-order Hessian reasoning in maintainer docs, aimed at shaving tens of GiB off MoE checkpoints.
• Fractal-64 transcoding — block-64 layouts at roughly 3.75 bpw with custom Q2_0 type 40.
• Register and cache tiling — explicit L1D / L2 / L3 column counts ({8,4,3,2} in README) to chase memory bandwidth on Zen 5.

If you contribute upstream to llama.cpp, read the patch before proposing merges — much of modchip is not upstreamable without accepting custom tensor types.

install.sh, HugePages, and /dev/shm

The ./modchip status command inspects 2MB HugePages and /dev/shm IPC per README. Those are easy to skip on a laptop and then wonder why restore times regress.

Before blaming the model:

• Confirm THP / hugepage settings match README guidance for your distro.
• Watch VmRSS during ./modchip bench — the README’s DeepSeek row is explicit about 57.61 GiB peak at 256K.
• On shared machines, ./modchip stop matters — background services hold RAM.

Gateway, eval gate, and leaderboard discipline

The repo includes gateway/sse_gateway.py for agentic SSE proxy plus a dark-mode web HUD on port 8787. ./modchip eval runs the 100-point quality, tool-calling, coding, and 8.5K needle gate described in README — use it when you change quantization mode.

The community leaderboard in README is maintainer-curated from GitHub benchmark issues. Treat leaderboard rows as repro invitations, not as audited MLPerf submissions.

Enterprise angle (README)

The README pitches air-gapped enterprise code intelligence without $30k+ multi-GPU nodes. That is a buyer story, not a security certification. Pair with your org's model allow lists and open-weight license review.

Hacker News note

As of November 8, 2026, the Show HN thread had no substantive technical comments yet — there is no developer hands-on section worth quoting. Revisit when builders post independent timings.

When comments arrive, compare them to benchmarks/results/*.json in the repo rather than to README peaks alone. Independent runs on Graviton or Threadripper will tell you whether modchip is a niche AVX-512 artifact or a broader DDR5 story.

Licensing and redistribution

Apache-2.0 on the project code does not automatically Apache-license every Hugging Face weight shard. Read each model repo’s card before redistribution. For internal mirrors, snapshot both Git commit and GGUF revision when you pass audits.

If you ship modchip inside a product, your legal review should treat custom tensor types like a forked runtime — downstream liability does not vanish because the CLI is friendly.

Related reading

• MLX Apple Silicon local LLM guide
• vLLM vs SGLang inference servers
• Open-weight model licenses explained
• How to read a Hugging Face model card
• Claude Code commands reference
• Loop engineering for coding agents
• Qualcomm Cloud AI 100 edge specs

Version numbers, speed tables, and binary names follow synchopate/llama-modchip README v2.4.0 as of November 8, 2026 — reproduce on your hardware before production use.