Reproducible CPU LLM inference benchmarking and optimization study around llama.cpp, focusing on quantization, KV cache, profiling, and hot-path optimization with a defined microbenchmark and end-to-end evaluation. The repo pins llama.cpp at a specific commit and provides scripts, configurations, and reports.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Reproducible CPU LLM inference performance study built around llama.cpp, focusing on quantization, KV-cache, CPU profiling, and hot-path optimization. It targets Qwen2.5-0.5B-Instruct GGUF models and includes benchmarking, profiling, and end-to-end validation with a nuanced conclusion that a local kernel optimization showed microbenchmark gains but end-to-end results were inconclusive. The project uses a pinned llama.cpp commit and provides a structured workflow across multiple phases.
How it works
The repository defines a reproducible workflow: pin the dependency (llama.cpp) to a commit, prepare specific GGUF models, run benchmarks across configurations (prefill/decode, thread scaling, quantization), profile with Linux perf, extract and compare results, and perform correctness validation. It includes microbenchmarks and paired end-to-end A/B tests to separate kernel-level gains from application-level throughput. The results are organized into phase reports and integration provenance data.
Getting started
# Python environment
python -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
git submodule update --init --recursive
# Project Debug build and tests, including Phase 8 native correctness tests
cmake --preset debug
cmake --build --preset debug
ctest --preset debug --output-on-failure
.venv/bin/python -m pytest
# Pinned llama.cpp CPU Release build verification (build first per linked guide)
.venv/bin/python scripts/verify_llama_cpp_build.py
# Q4_K_M baseline, prefill/decode, quantization, and KV/context matrices
.venv/bin/python benchmarks/run_llama_bench.py configs/cpu_baseline_q4.yaml
.venv/bin/python benchmarks/run_llama_bench.py configs/prefill_decode_scaling.yaml
.venv/bin/python benchmarks/run_llama_bench.py configs/quantization_comparison.yaml
.venv/bin/python benchmarks/run_llama_bench.py configs/kv_cache_context_scaling.yaml
# Profiling capability probe (writes ignored local output)
.venv/bin/python scripts/probe_perf_events.py \
--json profiles/perf-event-probe.json \
--markdown profiles/perf-event-probe.md
Use the Release build guide before verification and the profiling guide before collecting counters.
Recent releases
- none
Traction
13 stars
Behind the repo
No linked startup or company information provided in the repo metadata.
Caveats
- License: MIT
- Created: 2026-07-12
- Last push: 2026-07-12
- Open issues: 0
- Language: Python
- The results show that a local kernel optimization produced microbenchmark gains (1.32x–1.34x at certain depths) but end-to-end measurements were inconclusive, with no stable throughput improvement claimed. The documentation notes various limitations including CPU-only execution, environment variability, and the lack of a confirmed end-to-end benefit.