Cloud leaderboard scores lie politely about your phone.
On August 24, 2026, Liquid AI and Artificial Analysis released Pipette — an open-source platform for benchmarking foundation models on edge devices, with a public dataset of 1,000+ lab-verified configurations and native iOS/Android benchmark clients.
Premise: on-device behavior is a property of the deployed system, not the model in isolation.
TL;DR
| Question | Answer |
|---|---|
| What it measures | Model × quantization × runtime × device × context → five performance metrics |
| Initial coverage | 30+ models; llama.cpp on macOS, iOS, Windows, Android; 256–8,192 ctx |
| Reference devices | MacBook Pro M5 Max, iPhone 17 Pro, Galaxy S26 Ultra |
| Open tools | Dashboard, shareable chart URLs, OSS clients, mobile apps |
| Community | Submit-your-results workflow in beta |
| Partner | Artificial Analysis — same shop behind cloud leaderboards |
Why another benchmark?
Mobile and laptop agents fail for boring systems reasons:
- A quant that runs on M5 Max throttles on a phone after three minutes
- llama.cpp build flags change token/sec more than a "better" model name
- Context length collapses throughput — but cloud cards rarely show your ctx
Pipette turns that into empirical, reproducible comparisons — the edge analogue of Artificial Analysis cloud evals, launched alongside their AA mobile intelligence work.
What's in the dataset
Per Liquid AI's launch post:
- Five on-device performance metrics per configuration
- 1,000+ published rows (growing)
- Shareable dashboard URLs — filter model, quant, runtime, device; link the exact chart in a PR or spec
- AMD Ryzen AI Max+ / Radeon 8060S results listed as coming soon
This feeds directly into LFM2.5-2.6B positioning — Liquid cites ~220 tok/s on M5 Max and ~30 tok/s on phone for that checkpoint, with tool-use scores competitive with much larger models.
The five metrics Pipette actually tracks
Liquid AI's launch post names five on-device performance metrics per configuration row. The dashboard exposes them as filterable columns — not a single "score" like MMLU. Treat them as a systems profile:
| Metric category | What it tells you | Why it matters for agents |
|---|---|---|
| Throughput (tok/s) | Decode speed at a fixed context length | Sets how fast tool results stream back into the loop |
| Time-to-first-token (TTFT) | Prefill latency before generation starts | Long scratchpads + RAG chunks hit this first |
| Memory footprint | RAM/VRAM pressure for model + KV cache | Phones kill background apps when you exceed budget |
| Quality proxy | Task-specific eval at that quant tier | Q4 may be 2× faster but fail structured JSON |
| Stability under load | Sustained vs burst performance | Thermal throttling shows up here, not in 30-second lab runs |
The critical detail: every row is model × quantization × runtime × device × context. Change any one variable and you get a different row. That is why a Hugging Face card showing "7B, Q4" is not a deployment spec — it omits runtime build, NPU path, and the context length your agent actually uses.
For readers coming from cloud leaderboards: Artificial Analysis cloud evals optimize for comparable API endpoints. Pipette optimizes for comparable hardware stacks. Same parent org, different measurement contract.
Pipette vs cloud leaderboards — when each lies
Neither dataset is "wrong." They answer different questions.
Cloud leaderboards lie politely about your phone when you copy a model name from a datacenter chart and expect identical behavior on-device. Quantization, runtime, and thermal envelope dominate edge deployment in ways that never appear on a vLLM leaderboard row.
Pipette lies politely about your datacenter when you use it to pick a cloud serving model. It does not measure batch prefill economics, multi-tenant scheduling, or $/million tokens at hyperscale.
| Question | Trust cloud (AA, etc.) | Trust Pipette |
|---|---|---|
| Which 30B model is smartest at 128K? | Yes | No — wrong hardware class |
| Will Q4_K_M run on iPhone 17 at 4K ctx? | No | Yes — filter and verify |
| Which llama.cpp build is fastest on M5 Max? | No | Yes — runtime is a column |
| What is my agent loop cost at 10M tokens/day? | Yes | Partial — local only |
If you are building a personal local AI system, Pipette is the decision layer. If you are routing cloud APIs for production agents, pair Pipette picks for on-device fallbacks with cloud benchmarks for server-side tiers.
What people are asking about on-device benchmarks
These questions show up in GitHub issues, Discord threads, and Hacker News comments whenever a new edge benchmark drops:
"Is 30 tok/s on phone enough for an agent?" Depends on loop depth. A single-turn assistant feels fine at 20–40 tok/s. A five-step tool loop at 200 tokens per step adds perceptible wait — which is why Liquid markets LFM-class models for millisecond-class CPU/GPU/NPU paths, not 27B cloud checkpoints squeezed onto silicon.
"Should I trust lab numbers or run my own?" Lab numbers are upper bounds under controlled thermals. Liquid publishes reproducibility protocols; the native iOS/Android clients exist so you can validate on your ship target. Community submission (beta) is the path for hardware Pipette has not lab-tested yet.
"How does Pipette compare to MLX benchmarks?" MLX is one runtime column in Pipette's matrix. Apple's MLX community posts ad-hoc numbers; Pipette standardizes across llama.cpp, ONNX, and vendor paths so you can compare builds, not blog posts. For MLX-specific workflow, see Qwen3 on Mac via MLX/ORCA.
"Does quantization always win on edge?" Not always. PrismML Bonsai pushed 1-bit ternary toward phone-class deployment, but quality cliffs vary by task. Pipette makes the speed/quality trade visible per configuration rather than assuming Q4 is the default answer.
What this means for builders shipping agents
On-device agents are a product decision, not a model download. Pipette gives you the empirical layer to make that decision defensible in a spec or PR review.
Before you commit to a model checkpoint:
- Open the Pipette dashboard and filter to your exact device class — not "Apple Silicon generically" but the SKU you ship (or the oldest device you support).
- Set context length to your agent scratchpad budget — if your harness keeps 4K tokens of tool history, do not compare 256-token rows.
- Compare two quant tiers for your top candidate — the delta in tok/s and quality proxy often matters more than swapping model brands.
- Run the native client on hardware with a realistic thermal load — screen on, other apps backgrounded, 10+ minute session if your use case is continuous.
- Document the winning row URL in your repo — Pipette's shareable chart links are designed for exactly this.
Architecture implication: fast local decode does not fix bad loop engineering. It makes tight loops tolerable. Pair Pipette numbers with harness design: minimize round-trips, cache tool schemas, stream partial results. A 3× tok/s gain from quant selection is wasted if your agent calls the same tool six times per user message.
For robotics and edge brains: Liquid's "deploy agents everywhere" positioning targets CPU/GPU/NPU across phones, laptops, and embedded controllers. Pipette is how they prove LFM checkpoints hit latency budgets before you embed weights in firmware. Validate before you bake — reflash costs more than a dashboard filter.
CI and regression: If you ship GGUF or ONNX builds across OS releases, Pipette rows become regression anchors. When iOS 20 or a new llama.cpp release drops, re-benchmark the same configuration and diff against your stored chart URL.
How builders should use Pipette
- Pick your ship target first — phone vs laptop vs robot brain; do not benchmark on Mac if you deploy Android.
- Filter to your context budget — agent scratchpads eat ctx; 256-token leaderboard rows are not your app.
- Compare quant tiers — Pipette makes Q4 vs Q8 tradeoffs visible before you bake weights into an app.
- Run the native client on your thermal envelope — lab numbers are an upper bound.
- Pair with harness design — fast local decode does not fix bad agent loops; it makes tight loops tolerable.
| Use case | Pipette helps when… |
|---|---|
| Offline assistant | You need tok/s + memory at 4k–8k ctx on iPhone |
| Laptop agent | You choose between MLX, GGUF, and ONNX builds |
| Robotics edge | You validate LFM-class models before embedding |
| CI smoke tests | You regression-test quant builds across OS releases |
Honest limitations (August 2026)
Pipette is launch-day useful, not complete:
- Community submission is beta — coverage gaps for niche hardware remain; file issues if your device class is missing.
- AMD Ryzen AI Max+ / Radeon 8060S results are listed as coming soon — Windows-on-AMD builders wait.
- Quality proxies are task-specific — Pipette does not replace your own eval set for JSON schema adherence or tool-call accuracy.
- Lab thermals ≠ user pocket — phone benchmarks in controlled conditions overestimate sustained performance in summer ambient heat.
None of that diminishes the core premise: on-device behavior is a systems property. Pipette is the first open dataset that treats it that way at scale.
Reading Pipette rows like a deployment spec
When you share a Pipette chart URL in a design doc, include these fields explicitly:
- Model checkpoint — exact Hugging Face revision, not family name
- Quant format — Q4_K_M vs Q8_0 changes memory and quality cliffs
- Runtime build — llama.cpp commit or MLX version; flags matter
- Device SKU — iPhone 17 Pro ≠ iPhone 16; M5 Max ≠ M4 Pro
- Context length — the row's ctx must match your agent's working set
Liquid AI's models page already embeds Pipette-linked charts for LFM checkpoints — use those as templates for how to cite empirical edge data alongside cloud eval scores. For LFM2.5 230M and 2.6B agentic variant coverage on explainx.ai, Pipette is the verification layer behind the marketing claims.
Related on explainx.ai
- LFM2.5-2.6B on-device agentic model
- Liquid LFM2.5 230M edge agent model
- Build your personal local AI system
- MacBook vs dedicated GPU for local LLMs
- Unsloth Desktop — train and run models locally
- Qwen3-8 27B on Mac via MLX/ORCA
- PrismML Bonsai — phone 1-bit ternary
- What is model quantization?
Sources
- Liquid AI — Introducing Pipette — August 24, 2026
- Artificial Analysis — mobile benchmarking announcement
- Liquid AI models page — Pipette-linked charts
Device list reflects Liquid AI's August 2026 launch; community submission workflow is beta. Follow @explainx_ai for on-device agent coverage.