Bonsai-demo

What it is

The repository offers a demo setup to run Bonsai language models locally, including Bonsai (1-bit) and Ternary-Bonsai, with vision-language capabilities and tool-calling features in the 27B family. It provides quick-start instructions, environment variables, and setup scripts to install dependencies, download models, and run the servers.

How it works

• Two model families: Ternary-Bonsai (default) and Bonsai (1-bit). Users can switch sizes among 27B, 8B, 4B, and 1.7B using BONSAI_MODEL and BONSAI_FAMILY.
• The setup.sh script handles dependencies, Python virtual environment creation, model and binary downloads, MLX building (macOS), and Open WebUI installation.
• It supports multiple runtimes/backends: llama.cpp on CPU/Metal/CUDA/Vulkan, and MLX for 2-bit/1-bit configurations. The environment variables control model size, family, and various runtime knobs (BONSAI_NGL, BONSAI_CTX, BONSAI_HOST, BONSAI_SPECULATIVE, BONSAI_KV4).
• The repository includes instructions to run the models via scripts like run_llama.sh and run_mlx.sh, and demonstrates switching between Ternary and Bonsai families with commands to download models and start servers.

Getting started

• macOS/Linux quick start:
  • git clone https://github.com/PrismML-Eng/Bonsai-demo.git
  • cd Bonsai-demo
  • export BONSAI_MODEL=27B
  • export BONSAI_TOKEN="hf_your_token_here" (required for 27B private repos)
  • ./setup.sh
• Windows (PowerShell) quick start:
  • git clone https://github.com/PrismML-Eng/Bonsai-demo.git
  • cd Bonsai-demo
  • $env:BONSAI_MODEL = "27B"
  • $env:BONSAI_TOKEN = "hf_your_token_here"
  • Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
  • .\setup.ps1

Environment variables

• BONSAI_MODEL: 27B, 8B, 4B, 1.7B (default 27B)
• BONSAI_FAMILY: ternary, bonsai (default ternary)
• BONSAI_NGL, BONSAI_CTX, BONSAI_HOST, BONSAI_SPECULATIVE, BONSAI_KV4, etc.

Quick start examples

• Combine them in setup.sh invocations, e.g.:
  • ./setup.sh
  • BONSAI_MODEL=1.7B ./setup.sh
  • BONSAI_FAMILY=bonsai ./setup.sh
  • BONSAI_FAMILY=bonsai BONSAI_MODEL=4B ./setup.sh

Recent releases

• None

Traction

• Stars: 2152

Behind the repo

• The repository contains links to HuggingFace repos for Bonsai-27B, Bonsai-8B, Bonsai-4B, Bonsai-1.7B, and their Ternary equivalents, indicating a multi-model, multi-format setup.

Caveats

• Licensed under Apache-2.0
• Requires HuggingFace token for private 27B repos
• The setup downloads pre-built binaries from a specific GitHub release and may build MLX from source on macOS