Gigatoken is a Rust-based tokenizer designed for language models, offering compatibility with HF Tokenizers and Tiktoken, and claims to achieve GB/s throughput with a Python API and a fast Rust core.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Gigatoken is described as the fastest tokenizer for language modeling, with compatibility modes for HuggingFace Tokenizers and Tiktoken, and its own Gigatoken API. It emphasizes high throughput via a Rust implementation and parallelism.
How it works
The project centers on a Rust-based tokenizer that supports a wide range of CPU hardware and compatibility with common tokenizers. It relies on a Rust core that reads data directly through its API to maximize parallelism, with an emphasis on pretokenization performance and caching of pretoken mappings. The README includes extensive benchmark tables comparing gigatoken against HF tokenizers and tiktoken across multiple CPU configurations, indicating SIMD/cache-focused optimizations and reduced Python interactions as contributing factors.
Getting started
Installation is via Python package management:
pip install gigatoken
Usage in compatibility mode (easiest):
import gigatoken as gt
# Minimum change from existing HuggingFace tokenizers usage (compatibility mode)
hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
# tokenizer can be used in the same contexts as hf_tokenizer
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])
# OR with tiktoken
tiktokenizer = ...
tokenizer = gt.Tokenizer(tiktokenizer).as_tiktoken()
# Now works like existing tiktoken tokenizers
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])
Gigatoken API usage (Fastest):
import gigatoken as gt
tokenizer = gt.Tokenizer("Qwen/Qwen3-8B") # Accepts HF model names
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)
Recent releases
The Releases section lists no entries for latest version 0, implying no specific tagged releases in the provided data.
Traction
Stars: 3905 (as given). Forks: 203. Open issues: 19. Language: Rust. License: MIT. Created: 2025-11-10. Last push: 2026-07-26.
Behind the repo
No direct startup/company link provided beyond the GitHub repository in the data.
Caveats
License: MIT. Known issues section notes:
- Python iteration uses ABI3 (slower than internal CPython APIs) but may improve with version-specific optimizations.
- File sinks are not yet implemented in the Gigatoken API.
- WordPiece not supported.
- SentencePiece tokenization is not as optimized as BPE in this implementation.
- Windows has not been tested much; WSL recommended.
Citations
If used in research, cite as provided in the README's BibTeX entry.