samir-vllm: Build LLM Inference From Scratch in C++ and CUDA

samir-vllm is a from-scratch large-language-model inference engine you can compile today: it loads Meta's Llama 3.2 1B Instruct weights, runs multi-layer prefill on GPU, and serves tokens with a paged KV cache and continuous batching. The GitHub repository bundles six-plus hours of YouTube walkthroughs, an 80-page handwritten PDF, and MIT-licensed CUDA kernels that mirror what production servers hide behind Python APIs.

If you have only used hosted APIs or pip-installed vLLM, this repo is a map of every buffer between prompt and next token.

TL;DR: what works in the repo

Why builders are sharing it

Hacker News surfaced the project on November 9, 2026 because it compresses months of inference coursework into runnable code. Commenters split between praise for pedagogy and skepticism about AI-assisted research platforms; we summarize that discussion below.

The README's honesty is part of the appeal. BF16 operations can diverge slightly between batch sizes, so greedy outputs may not match bit-for-bit across single and batched runs. That is exactly the kind of detail textbooks skip but production engineers learn painfully.

Architecture walkthrough

Continuous batching diagram for LLM inference learning engines

Loading: The engine memory-maps safetensors metadata, validates offsets, and copies BF16 weights without an extra host staging copy when using the mmap path.

Prefill: For each of 16 layers, the stack projects Q/K/V with cuBLAS, applies RoPE, runs grouped-query attention, projects output, applies RMSNorm and SwiGLU FFN residuals, then lands on LM-head logits.

KV cache: The paging idea follows the PagedAttention paper behind vLLM, which borrows virtual-memory paging to manage the key-value cache. Physical pages hold 16 tokens of K and V per layer. Block tables map sequences to pages so decode appends one token at a time at the correct rotary position.

Batching: Continuous mode admits FIFO prompts when page budgets fit; static mode waits for the active group to finish. The --stats flag prints peak batch size, admission waits, and pool usage.

For conceptual background, pair the repo with our KV cache explainer, speculative decoding guide, and vLLM vs SGLang comparison.

How to try it locally

Requirements from the README:

• NVIDIA GPU with CUDA Toolkit
• CMake 3.24+ and a C++17 compiler
• ICU development libraries
• Llama 3.2 1B weights and tokenizer under models/llama-3.2-1b-instruct/

Example invocations:

cmake -S . -B build
cmake --build build -j
./build/samir-vllm "Explain the KV cache in two sentences." 128
./build/samir-vllm --stats "Explain paging." 64 "Explain GQA." "Explain RoPE."

Meta's community license still applies to bundled tokenizer assets; keep attribution when redistributing weights.

What developers are saying on Hacker News

The November 9, 2026 thread was still thin when we reviewed it: one author note and a flagged reply, not a long debate. samirswr said in their comment that the handwritten notes and almost all code were theirs, with only small README fixes from automation.

That matches the repo's MIT code plus reserved PDFs. Treat skepticism as a reason to read kernels, not as proof the project is slop.

Admission control: how the scheduler avoids deadlock

KV cache pages and scheduler timing when batches compete for GPU memory

The README spends real ink on page budgets, not just kernels. Each request declares prompt_tokens + max_new_tokens - 1, rounded up to 16-token pages across 16 layers. The scheduler rejects workloads whose budget exceeds the entire pool instead of deadlocking mid-decode.

Continuous mode keeps a FIFO queue: when a running request finishes, freed pages and credits admit the next waiting prompt. Static mode batches prompts in groups, which is easier to reason about while debugging kernels but wastes silicon when one long completion blocks the queue.

The --stats flag surfaces peak active requests, largest decode batch, admission waits, and remaining free pages. Those metrics mirror what operators watch in production servers, minus Prometheus glue.

If you are porting lessons to Python frameworks, compare this conservative policy with vLLM's max_num_seqs and preemption logs described in our vLLM vs SGLang piece.

samir-vllm vs production inference stacks

Teams building on Google TPUs or Nvidia GPUs still benefit from understanding what these frameworks optimize; see semantic caching for cost and RAG reranker latency once basics click.

Licensing quirks

MIT covers src/, include/, and tests. Handwritten PDFs remain all-rights-reserved; the README blocks resale or rehosting without written permission. Fork the code, not the scans.

YouTube curriculum map

The README indexes a multi-part build series that mirrors the repo commit history:

Watching parts 8–10 together clarifies why production servers spend thousands of lines on schedulers while model math stays stable.

Tests and reference scripts

Beyond the main binary, the repo ships tokenizer-test to lock BPE behavior against the bundled Llama tokenizer file. A Python reference under python/ helps diff edge cases when C++ and notebook experiments disagree.

That split mirrors how teams validate inference servers: golden tokenizer tests first, then kernel microbenchmarks, then end-to-end perplexity or regression prompts.

Debugging tips when you port the lessons

When you adapt patterns into your own C++ or Rust runtime:

1. Log tensor shapes per layer during prefill only; decode bugs often hide in position indices, not matmuls.
2. Compare BF16 vs FP32 reductions in RMSNorm; samir-vllm documents batch-size sensitivity honestly.
3. Watch page accounting before optimizing kernels; OOM during decode usually means block table leaks.
4. Profile with Nsight after correctness; the README assumes you already know CUDA tooling basics.

If you prefer staying in Python, trace the same concepts in vLLM vs SGLang before diving into CUDA.

Where this fits explainx.ai's inference canon

We maintain parallel explainers on KV cache math, speculative decoding, semantic caching, and RULER long-context evaluation. samir-vllm is the hands-on lab tying those articles to typed pointers and GPU allocations.

Students who finish the repo should be able to read vLLM release notes about paged attention or FP8 KV without getting lost in jargon.

Hardware expectations

The README targets a single NVIDIA GPU with enough VRAM for Llama 3.2 1B BF16 weights plus KV pools. The default 128 MiB KV pool is tiny compared with production servers; --kv-cache-mib exists precisely to teach memory trade-offs.

On laptops, start with short prompts and single requests before enabling four concurrent chats. Thermal throttling shows up as silent batch-size regression during study sessions.

Performance expectations

This is not a throughput record chase. The README optimizes for clarity: one prefill at a time between decode steps, greedy decoding only, fixed 2,048 context. You will not beat vLLM tokens per second on the same GPU.

The win is inspectability. When a token looks wrong, you can printf the block table index, inspect RoPE positions, and rerun a single decode step without spelunking Python abstractions.

Contributing without forking the notes

If you improve kernels, open a GitHub PR against the MIT portions only. Do not redistribute the handwritten PDFs; link to the official release and describe fixes in prose.

Useful community contributions might include:

• ROCm or CPU-only fallbacks for students without NVIDIA cards
• Additional regression prompts in tests/
• Microbenchmarks comparing paging policies

Each contribution should keep the educational narrative intact; micro-optimizations that obscure readability will likely be rejected.

Safety and licensing reminders

Meta's Llama weights require accepting the community license. The engine never phones home, but your prompts still pass through local GPU memory; wipe VMs shared in classrooms between students.

If you demo samir-vllm at a meetup, mention that production stacks add continuous batching and auth, rate limits, and abuse monitoring that this repo intentionally omits.

Classroom use cases

Instructors can pair one video part per lab session, assign kernel modifications as homework, and grade using hidden tokenizer tests plus public generation prompts. Cap concurrent students on shared lab GPUs to avoid admission-queue exercises turning into unintentional chaos theory lessons.

Assign reading from our multimodal vision token budget guide only after students master text KV paging; otherwise two hard abstractions collide on the same week.

Record office-hour questions about RoPE and paging; the README's FAQ gaps are where course staff should contribute upstream documentation.

Star the repository if it saves you a week of sorting random CUDA snippets; upstream maintainers use stars as signal for which study notes to typeset next.

Keep a lab journal comparing your kernel timings week over week; that habit transfers directly to on-call work tuning production inference clusters.

Related reading

• What is a KV cache in LLM inference?
• Speculative decoding explained
• vLLM vs SGLang inference servers
• Vector databases for RAG compared
• Semantic caching for LLM apps
• RULER benchmark for long context