A CUDA-based, zero-dependency inference engine for Qwen 3.6 35B targeting RTX 5090/Blackwell, with dedicated GPU-optimized path and OpenAI-compatible server. Includes benchmarks, KV cache quantization, and state offload features.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
A hyper-optimized, zero-dependency C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell.
How it works
The project provides a native CUDA implementation for both 10 full-attention layers and 30 recurrent SSM layers, with dual-tier state management: VRAM KV cache quantization (2.5× smaller KV cache via --kv-quant) and DRAM/Disk state checkpoint caching (offloading recurrent states). It includes a zero-dependency OpenAI-compatible server (q36_server) and developer tools for benchmarking (q36_bench) and perplexity evaluation (q36_ppl).
Key components referenced:
- Core:
q36_engine.cu(CUDA graph setup, Blackwell kernels) - Quantization:
q36_dequant.cuh(quant formats, layouts) - Tokenizer:
tokenizer.c(native BPE) - Server:
server.c(HTTP server)
Getting started
Prerequisites include an NVIDIA RTX 5090 or RTX 6000 Pro Blackwell GPU with compute capability 12.0+, Linux with CUDA Toolkit 13.1, and the model in MXFP4 GGUF format.
Build and validation:
# Compile and run validation tests on CPU (no GPU required):
make tools
./q36_info /path/to/model.gguf
# Compile the full GPU-accelerated engine, benchmark, and OpenAI server:
make q36 q36_bench q36_server
Usage examples:
# Interactive CLI
./q36 -m /path/to/model.gguf
# OpenAI-compatible server
./q36_server -m /path/to/model.gguf --port 8080 --ctx 32768
# Benchmark
./q36_bench -m /path/to/model.gguf
KV cache & state management:
--cache-ram MB
--cache-min tokens
--cache-dir path
--no-state-cache
Multi-slot API and dynamic dual decode paths are described in the README, with commands for continuous batching benchmarks:
# Run a continuous batching simulation with up to 48 concurrent slots:
Q36_CB=48 Q36_CB_NREQ=256 ./q36_bench -m /path/to/model.gguf
Recent releases
RELEASES (latest 0): - none
Traction
Stars: 17
Behind the repo
License: AGPL-3.0. GitHub user: Ambud Sharma
Caveats
License: AGPL-3.0. Created: 2026-07-12. Last push: 2026-07-13. Open issues: 0. Forks: 0. Language: Cuda.