explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • Why another "make LLM fast" repo matters
  • Results the README publishes
  • The workflow loop
  • Scripts builders actually run
  • Install for Claude Code and other agents
  • Principles that survived real climbs
  • How this compares to cloud-only tuning
  • Agent skill ecosystem context
  • When not to use this skill
  • Replication checklist
  • Inside references/levers.md
  • Unified memory on laptops matters
  • Publishing.md and the six acceptance gates
  • Failure logging as a product feature
  • Pairing with Claude Code workshops
  • Ollama and Apple Silicon paths
  • SGLang vs vLLM choice in 2026
  • When agents mis-measure
  • Enterprise angle: why Sybil open-sourced it
  • Quick FAQ for searchers asking "is local LLM worth it"
  • Related reading
← Back to blog

explainx / blog

Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent

Local AI, LLM Inference, Agent Skills, Claude Code, Open Source

Part of Local AI

Sybil Solutions open-sourced inference-optimisation-skill: an agent SKILL.md workflow to hill-climb decode, prefill, KV cache, and TTFT on any local LLM stack.

Nov 22, 2026·9 min read·Yash Thakker
add explainx.ai
go deep
Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent

The inference-optimisation-skill repository turns months of measured LLM serving hill-climbs into a portable agent workflow. Sybil Solutions published it in November 2026 with about 1,700 GitHub stars. The skill targets decode, prefill, speculative decoding, KV cache size, startup, and TTFT on any hardware and quantization, under a memory cap, with long-context stability gates before you call a build production-ready.

TL;DR

table · 2 cols
PieceRole
SKILL.mdProcedure: inventory, ceilings, one-lever loop, stop rules
references/ceilings.md, levers.md, gotchas.md, publishing.md
scripts/bench.pyOpenAI-compatible harness; writes ledger.jsonl
scripts/ceiling.pyTheoretical tok/s caps from hardware JSON
scripts/queue.shInterleaved control/candidate/control runs
Install path~/.claude/skills/inference-optimisation for Claude Code
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Why another "make LLM fast" repo matters

Most speed guides list flags. This repo argues noise is ±5–10%, so single-run wins are often drift. The authors encoded what worked on Intel Arc B390, Arc Pro B70, RTX 3090, and 4× H200 NVL, including failures in PLAN logs.

That pairs with industry moves toward agent-driven tuning, such as Together Compiler and Microsoft MAI hill-climbing in Copilot Excel. Here the agent is yours, running against your rack or laptop.

Results the README publishes

KV cache slots with one highlighted token stored across decode stepsKV cache slots with one highlighted token stored across decode steps

The README table is the credibility anchor. Selected rows:

table · 3 cols
HardwareStackBefore → after (README)
Arc B390 laptopOrnith-1.5-35B EXL3, SGLang XPU + MTP43.6 → 53 tok/s decode, 1 user
Arc B390SamePrefill 256K: 297 → 628 tok/s
Arc Pro B70Qwen3.8-27B, vLLM + custom XPU kernels91.2 tok/s decode at 1 user
RTX 3090Qwen3.6-35B-A3B EXL3 + MTP251.7 / 357.9 tok/s prose/code
4× H200 NVLDeepSeek-V4.1-Flash, vLLM + speculative decodingCrashed → 229.1 tok/s decode, 17,546 prefill

Each line points to local-ai-registry pull requests or trellis-serve docs. explainx.ai does not re-run those benches; treat numbers as author-measured until you replicate on your silicon.

The B390 example includes a ceiling calculation: ~100 tok/s decode for one user needs two useful speculative drafts, but the model accepts ~14% at depth 2, so practical ceiling ~88–97 tok/s. Hitting ~55% of physics is labeled honest progress, not failure.

The workflow loop

Small draft token running ahead of verification on parallel railsSmall draft token running ahead of verification on parallel rails

The ASCII diagram in the README flows:

  1. Measure hardware → THEORY.md bandwidth and FLOPs per token.
  2. Baseline profile → pick the lever with largest ceiling gap × likelihood.
  3. queue.sh runs serialized candidate vs neighbouring controls with memory watchdog and ≥6 samples.
  4. Accept only if beat both controls and no metric regresses >3%.
  5. Long-context needle gate and concurrent full-KV soak.
  6. Pinned clean build and six acceptance gates before publish.

That is stricter than tweeting a GGUF quant swap. It matches how Together AI and Ollama prefix-cache guidance thinks about stable prompts for caches.

Scripts builders actually run

bench.py

Hits any OpenAI-compatible server for:

  • Startup and short-prompt TTFT
  • c1 decode median and cN aggregate decode
  • Cold prefill at multiple lengths
  • Needle retrieval in long context
  • Peak memory with cap watchdog

Token counts come from server usage stats, not streamed chunks, avoiding a common benchmarking trap the README calls out.

ceiling.py

Feed a JSON like examples/ceiling-ornith-b390.json to print decode, prefill, TTFT, startup, and KV maxima. Agents are instructed to run this before chasing arbitrary tok/s targets.

queue.sh

Runs control / candidate / control interleaves so thermal drift and background jobs do not fake wins.

Install for Claude Code and other agents

bash
git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then prompt examples from the README:

  • "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap"
  • "what's the theoretical max tok/s for this model on this machine?"

For non-Claude agents, copy skills/inference-optimisation/SKILL.md or paste PROMPT.md.

Quick start without an agent:

bash
S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
python3 $S/scripts/bench.py run configs/base.json --long 131072

Principles that survived real climbs

The README highlights ideas that generalize beyond Sybil's stacks:

table · 2 cols
PrincipleWhy it matters
Ceilings firstKnow if 53 tok/s is great or hopeless
Interleaved controlsKill ±10% noise
Real tokenizer lengthsAvoid fake prefill wins
Unified memory accountingcgroups miss GPU residency on APUs
Spec decode cost per stepAcceptance rate alone misleads
Log failuresMost value is what did not work

Gotchas for vLLM, SGLang, llama.cpp, ExLlama and Intel, NVIDIA, Apple, AMD sit in references/gotchas.md.

How this compares to cloud-only tuning

If you rent Together or OpenRouter, you optimize routing and batching, not kernel levers. This skill is for teams running Qwen on Together pricing or self-hosted vLLM on owned GPUs.

For KV cache concepts, start with what is KV cache. For Rubin vs GB300 marketing numbers, see vLLM Rubin claimed vs verified.

Agent skill ecosystem context

One highlighted skill card pulled from a shelf of agent skill filesOne highlighted skill card pulled from a shelf of agent skill files

Nvidia's SkillEvaluator guide pushes CI gates on agent skills. This skill is the opposite flavor: a domain skill with Python harnesses instead of only markdown instructions.

Poki's iPhone GUI agent skill shows the same pattern: ship SKILL.md + install path + real benchmarks.

When not to use this skill

Skip it if you only call ChatGPT APIs from a laptop. Skip it if you lack root or container access to inference servers. Skip it if you cannot afford multi-hour soak tests.

Use it when on-prem Qwen, DeepSeek, or MoE models already serve agents and p95 latency blocks rollout.

Replication checklist

  1. Copy base.json config with your base_url and model id.
  2. Run ceiling.py with honest bandwidth/FLOPs JSON.
  3. Capture baseline ledger before any flag change.
  4. Change one lever documented in levers.md.
  5. Pass needle + soak before tweeting results.

Inside references/levers.md

The skill splits levers into general (apply on any stack) and stack-specific (vLLM, SGLang, llama.cpp quirks). Agents are told to load levers.md only when a ceiling gap points at a category:

table · 3 cols
Lever familyTypical symptomREADME mindset
Speculative decodingGood acceptance, slow step timeOptimize per-step cost, prune draft vocab
KV cache layoutOOM at long contextShrink bytes/token before chasing quant swaps
Prefill batchingGreat decode, slow first tokenSeparate prefill tuning from decode sweeps
MTP / multi-token predictionUnstable at 256KRun needle gate before calling win
Engine startupMinutes to first requestTrack startup separately in ledger

The Arc B70 row where SGLang beat vLLM on the same card (82.6 vs 75.7 tok/s decode) is a reminder that engine choice is itself a lever, not a one-time decision.

Unified memory on laptops matters

The README warns that cgroup memory on Intel unified memory laptops does not see full GPU residency. mem.sh exists because 40 GiB cap violations only show up when driver footprint counts.

If you run local Qwen on a 64 GB LPDDR5X machine, pair this skill with Nvidia long-context compaction paper context so you do not optimize decode while ignoring context eviction policy.

Publishing.md and the six acceptance gates

Before Sybil publishes a local-ai-registry recipe, the skill demands:

  1. Pinned engine versions and model hashes.
  2. Clean rebuild proving the binary matches the best ledger row.
  3. Needle test at declared max context.
  4. Concurrent users at declared concurrency without collapse.
  5. Regression guard on neighbour controls.
  6. Public results card linking raw ledger.jsonl.

That discipline is why the README can cite Registry Lab gates moving from 28.6 → 41.0 decode on B390 without hand-waving.

Failure logging as a product feature

PLAN.md + ledger.jsonl is where rejected configs live. Agents trained only on success posts miss that most hill-climbs are negative results. The skill explicitly tells the agent to write down failures because the next lever choice depends on knowing which knobs already failed.

For teams building internal agent platforms, copy that pattern into your MLOps tickets: attach failed bench JSON the same way you attach failed unit tests.

Pairing with Claude Code workshops

If your org is booking Claude Code workshops, add a local inference lab module: install this skill, point at a staging vLLM, and require ceiling.py output in the PR description before merging server flags.

Ollama and Apple Silicon paths

While README highlights Arc and CUDA, gotchas.md also covers Apple and AMD. Perplexity Lily Apple Silicon engine is a different approach (custom engine), but llama.cpp levers in the skill still apply when you serve through OpenAI-compatible adapters.

SGLang vs vLLM choice in 2026

Your cloud may ship Dynamo-wrapped SGLang while your laptop runs stock vLLM. The skill refuses to treat them as interchangeable: gotchas.md is per engine.

When reading SemiAnalysis-style throughput debates, remember those are rack-scale stories. This skill is single-node honest benches you can rerun tonight.

When agents mis-measure

Common traps the skill blocks:

  • Counting stream chunks instead of tokenizer tokens.
  • Using short prompts that hide prefill quadratic cost.
  • Ignoring battery power on laptops (bench flags runs on AC).
  • Calling a win when looping output detector fires.

Running python3 bench.py show after every queue should be non-negotiable.

Enterprise angle: why Sybil open-sourced it

Sybil ties the skill to public registry recipes with PR links. Open-sourcing the methodology lets other hosts replicate before/after tables without sharing proprietary kernels.

That mirrors AWS's agent skill for distributed KV cache: vendors publish playbooks, not just sliders.

Quick FAQ for searchers asking "is local LLM worth it"

Local LLM serving is worth the ops burden when any of these hold: PII cannot leave VPC, inference spend exceeds one FTE, p95 latency blocks an agent product, or you already own GPUs depreciating on the balance sheet.

This skill does not pick which model; it makes a chosen model faster without lying about numbers.

Related reading

  • Together Compiler agent tuning
  • Together AI + Ollama prefix cache
  • KV cache explainer
  • AWS distributed KV caching skill
  • Claude Code token efficiency guide

Sources: inference-optimisation-skill README (accessed November 22, 2026).

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 19, 2026

diagram-design: The Agent Skill That Draws Brand-Matched Diagrams in Claude Code and Codex

cathrynlavery/diagram-design gained about 330 GitHub stars in a day. It is an agent skill that writes diagrams as one self-contained HTML and SVG file, matched to your brand, with imports from Mermaid, draw.io, Excalidraw and PlantUML. Here is how it works, how to install it and what to check first.

Nov 15, 2026

NVIDIA SkillEvaluator: Three-Tier Agent Skill Quality Gates (Open Repo Guide)

NVIDIA/open-source SkillEvaluator packages static scanners, embedding overlap checks, and live agent evaluation for folders that follow the Agent Skills spec. Here is how the three tiers work, how to run checks without an API key, and how it connects to the older ACES lift findings.

Nov 13, 2026

Chrome DevTools MCP: Let Your Coding Agent Debug a Live Chrome Browser

Chrome DevTools for agents, shipped as the chrome-devtools-mcp repo, lets a coding agent drive and inspect a live Chrome browser through MCP or a CLI. It trended on GitHub again in week 46 with 55,000 stars. Here is what it does, how to install it, and what to switch off.