tokenizer-design
BPE, WordPiece, SentencePiece, and Unigram tokenizer training, vocabulary optimization, domain extension, and multilingual design.
npx skills add majiayu000/claude-skill-registry --skill tokenizer-design --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Tokenizer Design ## When to Use Design or modify tokenizers when training a model from scratch, adapting a model to a new domain/language with poor tokenization coverage, or optimizing inference efficiency via vocabulary tuning. ## Algorithm Selection ### Decision Table: Tokenizer Algorithm | Algorithm | Library | Strengths | Weaknesses | Best For | |-----------|---------|-----------|------------|----------| | BPE | tokenizers, tiktoken | Deterministic, widely adopted | Greedy merges can miss global optima | GPT-family, general LLMs | | WordPiece | tokenizers | Likelihood-driven merges | Slower training than BPE | BERT-family models | | Unigram | SentencePiece | Probabilistic, multiple segmentations | More complex implementation | Multilingual, T5/XLNet | | SentencePiece (BPE) | sentencepiece | Language-agnostic, raw text input | Less control over pre-tokenization | Multilingual, non-space languages | | Byte-level BPE | tokenizers | No UNK tokens, full coverage | Longer sequences for non-Latin scripts | GPT-2/3/4, Llama | ### Decision Table: Vocabulary Size | Vocab Size | Token Fertility | Training Cost | Best For | |------------|----------------|---------------|----------| | 8K-
- When to Use
- Algorithm Selection
- Decision Table: Tokenizer Algorithm
- Decision Table: Vocabulary Size
- Training BPE from Scratch
- Using HuggingFace tokenizers Library
- SentencePiece Training
- Extending an Existing Tokenizer
- Adding Domain-Specific Tokens
- Fertility Analysis
- Measuring Tokenizer Efficiency
- Gotchas and Anti-Patterns
- Vocab Size vs. Performance Tradeoffs
- Special Token Handling
What does the tokenizer-design skill do?
BPE, WordPiece, SentencePiece, and Unigram tokenizer training, vocabulary optimization, domain extension, and multilingual design.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill tokenizer-design --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.