sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill sentencepiece --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# SentencePiece - Language-Independent Tokenization Unsupervised tokenizer that works on raw text without language-specific preprocessing. ## When to use SentencePiece **Use SentencePiece when:** - Building multilingual models (no language-specific rules) - Working with CJK languages (Chinese, Japanese, Korean) - Need reproducible tokenization (deterministic vocabulary) - Want to train on raw text (no pre-tokenization needed) - Require lightweight deployment (6MB memory, 50k sentences/sec) **Performance**: - **Speed**: 50,000 sentences/sec - **Memory**: ~6MB for loaded model - **Languages**: All (language-independent) **Use alternatives instead**: - **HuggingFace Tokenizers**: Faster training, more flexibility - **tiktoken**: OpenAI models (GPT-3.5/4) - **BERT WordPiece**: English-centric tasks ## Quick start ### Installation ```bash # Python pip install sentencepiece # C++ (requires CMake) git clone https://github.com/google/sentencepiece.git cd sentencepiece mkdir build && cd build cmake .. && make -j $(nproc) sudo make install ``` ### Train model ```bash # Command-line (BPE with 8000 vocab) spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe # Python A
- When to use SentencePiece
- Quick start
- Installation
- Train model
- Encode and decode
- Language-independent design
- Whitespace as symbol (▁)
- Tokenization algorithms
- BPE (Byte-Pair Encoding)
- Unigram (default)
- Training configuration
- Essential parameters
- Character coverage
- Encoding options
Python pip install sentencepiece git clone https://github.com/google/sentencepiece.git cd sentencepiece mkdir build && cd build cmake .. && make -j $(nproc) sudo make install Command-line (BPE with 8000 vocab) spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe Python API
What does the sentencepiece skill do?
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill sentencepiece --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.