picolm

What it is

PicoLM is a minimal, from-scratch LLM inference engine written in C11. It runs TinyLlama 1.1B (and other LLaMA-architecture models in GGUF format) on hardware with limited RAM. It emphasizes memory mapping the model file and streaming workloads to fit within tens of megabytes of runtime memory.

How it works

PicoLM memory-maps a 638 MB model file on disk and streams one layer at a time through RAM. It uses FP16 KV cache, mmap layer streaming, and hardware-specific acceleration (NEON on ARM, SSE2 on x86). It supports multiple quantization formats (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, F16, F32) and includes a grammar-constrained JSON output mode for tool calling. The forward pass includes embedding lookup, multi-layer attention with grouped-query, and an output projection, followed by a final RMSNorm and generation steps with temperature, top-p, and optional JSON formatting.

Getting started

One-liner install (Raspberry Pi / Linux)

curl -sSL https://raw.githubusercontent.com/RightNow-AI/picolm/main/install.sh | bash

This install script detects platform, installs dependencies, builds PicoLM, downloads the TinyLlama model, and generates a PicoClaw config.

Build from source

git clone https://github.com/rightnow-ai/picolm.git
cd picolm/picolm

# Auto-detect CPU (enables SSE2/AVX on x86, NEON on ARM)
make native

# Download a model
make model

# Run it
./picolm /opt/picolm/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
    -p "The meaning of life is" -n 100

Platform-specific builds

make native      # x86/ARM auto-detect (recommended for local machine)
make pi          # Raspberry Pi 3/4/5 (64-bit ARM + NEON SIMD)
make pi-arm32    # Pi Zero / Pi 1 (32-bit ARM)
make cross-pi    # Cross-compile for Pi from x86 (static binary)
make riscv       # RISC-V (Sipeed LicheeRV, etc.)
make static      # Static binary for single-file deployment
make debug       # Debug build with symbols, no optimization

Usage

PicoLM — ultra-lightweight LLM inference engine

Usage: picolm <model.gguf> [options]

Generation options:
  -p <prompt>    Input prompt (or pipe via stdin)
  -n <int>       Max tokens to generate (default: 256)
  -t <float>     Temperature (default: 0.8, 0=greedy)
  -k <float>     Top-p / nucleus sampling (default: 0.9)
  -s <int>       RNG seed (default: 42)
  -c <int>       Context length override
  -j <int>       Number of threads (default: 4)

Advanced options:
  --json         Grammar-constrained JSON output mode
  --cache <file> KV cache file (saves/loads prompt state)

Examples

• Basic generation: ./picolm model.gguf -p "Once upon a time" -n 200
• Force JSON output for tool calls: see --json and sample outputs in README
• Pipe from stdin: echo "Explain quantum computing in one sentence" | ./picolm model.gguf -n 50

Traction

• Stars: 1895
• Forks: 238
• Open issues: 25

Getting started notes

• The release channel shows latest release as none; latest material is in README examples and build instructions.

Features (highlights)

• GGUF Native: Reads GGUF v2/v3 files directly
• K-Quant Support: Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, F16, F32
• mmap Layer Streaming: Model on disk, OS pages in per-layer
• FP16 KV Cache: Reduces memory footprint
• Flash Attention: Online softmax reduces attention buffer requirements
• Pre-computed RoPE: Tables eliminate transients from hot loop
• SIMD Acceleration: NEON on ARM, SSE2 on x86
• Fused Dot Products: Dequantize + dot-product in one pass
• Multi-threaded matmul: Parallel across cores
• Grammar-Constrained JSON: --json outputs valid JSON
• KV Cache Persistence: --cache saves/loads prompt state
• BPE Tokenizer: Score-based BPE loaded from GGUF metadata
• Top-p Sampling: Configurable seed
• Zero Dependencies: libc, libm, libpthread only
• Cross-platform: Linux, Windows MSVC, macOS; ARM, x86-64, RISC-V

Quick Start (recap)

• One-liner install: curl -sSL https://raw.githubusercontent.com/RightNow-AI/picolm/main/install.sh | bash