explainx / blog
slvDev runs a 28.9M TinyStories LLM on an ESP32-S3 at ~9.5 tok/s offline. Gemma Per-Layer Embeddings keep 25M params in flash. Architecture, numbers, limits.
Jul 26, 2026
Inflect-Micro-v2 is Apache-2.0 English TTS that fits under 10M parameters with a fixed male voice, deterministic seeds, and CPU-real-time synthesis. explainx.ai covers numbers, install, Nano vs Micro, and what the HN thread got right.
Jul 19, 2026
Pete Warden's Moonshine Micro brings voice activity detection, command recognition, and neural text-to-speech to microcontrollers — reference demo on the Raspberry Pi RP2350 (~80 cents) in as little as 470 KB RAM. MIT-licensed, TensorFlow Lite Micro, and a full Wi-Fi provisioning walkthrough.
Jun 26, 2026
Liquid AI's smallest model yet runs at 213 tok/s on a phone CPU and 42 tok/s on a Raspberry Pi 5. Pre-trained on 19T tokens with 32K context, it beats models twice its size on instruction following and tool use — and already controls a Unitree G1 humanoid on a Jetson Orin.
Most “edge LLM” demos still assume a phone SoC or a Jetson. slvDev/esp32-ai does something meaner: a 28.9 million parameter TinyStories model that generates text on the ESP32-S3 itself — roughly an $8 microcontroller class — writing tokens to a small OLED at about 9.5 tok/s, with no server.
The previous public bar on a chip like this was ~260K params. This one stores about a hundred times more by refusing to keep the whole model in fast RAM. Google’s Per-Layer Embeddings (Gemma 3n / Gemma 4) move the giant table into flash; the ESP32’s memory hierarchy becomes the product.
This explainx.ai decode is for embedded and edge builders: the memory map, the RESULTS.md ablations, the honest TinyStories ceiling, and how it rhymes with disk-backed Colibri and Moonshine Micro on RP2350 — different modalities, same “bandwidth over capacity” mindset.
| Question | Answer |
|---|---|
| Chip? | ESP32-S3 (~$8), 512KB SRAM / 8MB PSRAM / 16MB flash |
| Params? | 28.9M stored (25M in flash PLE table) |
| Speed? | ~9.5 tok/s end to end |
| Model size? | 14.9 MB at 4-bit |
| Network? | None — fully offline |
| Task? | TinyStories generation only |
| Trick? | Per-Layer Embeddings in flash |
| Prior bar? | ~260K params on similar MCU |
| Repo? | slvDev/esp32-ai |
| Steal? | Memory hierarchy as architecture |
| Parameters | 28.9M stored (25M flash lookup) |
| Chip | ESP32-S3 N16R8 class |
| Speed | ~9.5 tok/s e2e · ~9.7 tok/s compute |
| Connectivity | None |
| Artifact | 14.9 MB at 4-bit |
Core is matched across ablations at ~559K dense params — small enough that the design constraint was “fits the thinking budget,” not “fill the flash.” Vocab 32768, d_model=96, 6 layers, ple_dim=128.
Internal SRAM on the S3 is 512KB. If every weight must be hot, you stay in the sub-million-param ghetto. DaveBben’s earlier esp32-llm (~260K) is the comparison context the README cites — independent prior work, not a dependency.
The fix is philosophical: stop treating “model size” as “RAM footprint.” Embedding-style tables are read, not multiplied through every step the way a dense FFN is. Leave 25M rows in flash; fetch 6 rows/token (450 B).
SRAM (fast, tiny) thinking core — used every token
PSRAM (medium) output head + working memory / KV
FLASH (huge, slow) 25M-param PLE table — sparse reads
On-chip bandwidth benches (RESULTS.md): flash random 512B row 20.3 µs; six rows ≈ 0.12 ms per token for the table — about 0.7% of per-token memory time in the isolated bandwidth test. The output head dominates traffic (17.3 ms for a 1.5MB PSRAM scan in that bench). PLE’s bet is confirmed: the big table is nearly free; the head is the tax.
PLE (Gemma) injects embedding information per layer rather than only at the bottom. Ablations matter:
| Arm | Core | Total | Val ppl (2 seeds) | vs baseline |
|---|---|---|---|---|
| baseline | 559K | 3.7M | 12.58 | — |
| ple | 558K | 28.9M | 11.41 | +0.098 nats (~9.3% ppl) |
| fatembed | 559K | 28.9M | 11.94 | +0.052 nats |
ple_notable (plumbing, no table) is worse than baseline at small vocab — the table does the work, not the adapters.Quote carefully: 28.9M is stored parameters via a memory-hierarchy split — not a claim that this thinks like a 28.9M dense model.
4-bit PTQ: PLE’s edge is fully retained (even slightly larger vs baseline after quant) because a large redundant table with group scales is more robust than a tiny dense core where every weight is precious. No QAT required for the headline flash artifact.
Portable C runtime matched PyTorch golden to max abs diff = 0.00001 across 32,768 logits before device compile.
| Stage | ~tok/s e2e |
|---|---|
| First correct portable port | 0.57 |
| PSRAM head + scalar cleanup | ~4.6–4.8 |
| Dual-core exact head | ~5.7–6.2 |
| int8-staged head + int8 activations | ~9.5 |
Profile (int8 path, ms/token wall): head 57.6 | attn 25.6 | ple 8.5 | ffn 6.9 | input 4.4. The head is PSRAM-bandwidth-bound (~40 ms read floor at 60.7 MB/s for 2.43 MB int8 weights/token). SIMD alone maybe ~15% more; real levers are fewer bytes (int4 head + unpack) or a smaller/factorized head.
Author left the messy history in-repo — including a parameter-accounting bug that inflated an early number, then the correction. That honesty is part of why HN trusted it.
Does: coherent TinyStories continuations on-device (example from RESULTS: Once upon a time, there was a little girl named Lily…).
Does not: Q&A, instructions, code, world knowledge. Ceiling is the dense core; PLE does not move that.
HN immediately floated Inflect-class TTS on the same board for offline read-aloud — plausible param budgets, separate engineering. Another thread: layer-per-MCU clusters with RP2350 PIO — speculative; bandwidth between chips is the hard part.
| Project | Trick | Hardware | Feel |
|---|---|---|---|
| esp32-ai | PLE table in flash | $8 MCU | ~9.5 tok/s stories |
| Colibri | Weights stream from NVMe | Desktop CPU | seconds/token frontier |
| llama.cpp | mmap / quant | Laptop+ | daily driver |
| Moonshine Micro | Tiny voice nets | RP2350 | STT on MCU |
Karpathy’s llama2.c is spiritual ancestry (tiny LM in plain C) — RESULTS.md says no code/checkpoints derived from llama2.c or DaveBben; credit TinyStories + Gemma PLE design.
RESULTS.md before citing tok/s — use end-to-end 9.5, not the 58 tok/s bandwidth ceiling.firmware/esp32_llm/ — wiring notes for the 1.3" OLED (SH1106, pins 18/46 in recent commits).src/ + experiments/ if you change vocab or ple_dim.For product thinking, treat this as a sensor-side narrative generator: a badge, toy, or offline kiosk that prints a short story without a radio. Pairing with Wi-Fi sensing on ESP32 is a stretch conceptually — different RF stack — but the BOM overlap is real: same chip family, same flash discipline, same “ship without a cloud bill” constraint. If you need daily-driver chat quality, stay on llama.cpp; if you need “proof the hierarchy works,” flash this firmware and watch the OLED.
The useful mental model: capacity lives where bandwidth is sparse-friendly; compute lives where latency is low. Once you internalize that, the ESP32 demo stops looking like a stunt and starts looking like a design pattern you can steal for any flash-backed table — embeddings, codebooks, or expert routing stubs — as long as you measure the row fetch before you celebrate the param count.
Fun and learning are legitimate. Damascene constraint projects teach memory hierarchy better than slides.
Milk-V / RP2350 comments are about adjacent price points — Linux boards with more RAM change the problem; this post’s claim is specifically “PLE on ESP32 SRAM/flash,” not “best $5 AI board.”
Scale-up to NVMe + CPU is Colibri territory: same sparse-weight intuition, brutal seconds-per-token economics. Don’t confuse MCU demo tok/s with desktop disk-offload tok/s.
TTS on the same chip is a product idea, not shipped here — Inflect-Micro-class models are in the right param ballpark but need their own flash/PSRAM budget and audio path.
Primary sources: GitHub — slvDev/esp32-ai · in-repo README.md / RESULTS.md · TinyStories (Eldan & Li, arXiv:2305.07759) · Google Gemma Per-Layer Embeddings docs · Hacker News discussion (July 2026).
SRAM, tok/s, and parameter splits are as published in RESULTS.md (validated ~2026-07-21). Re-check the repo before shipping a board — firmware pins and accounting notes move.