cjk-aware-text-metrics
CJK/Latin weighted token estimation for multilingual LLM pipelines. Use when processing Japanese/Chinese/Korean text with fixed chars-per-token constants.
npx skills add majiayu000/claude-skill-registry --skill cjk-aware-text-metrics --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# CJK-Aware Text Metrics **Extracted:** 2026-02-11 **Context:** Multilingual LLM pipelines where token estimation affects cost, chunking, or rate limits ## Problem Fixed chars-per-token constants (e.g., `CHARS_PER_TOKEN = 4`) assume Latin text. Japanese/Chinese/Korean text uses ~2.5 chars/token, causing ~60% underestimation in token counts, cost previews, and chunk sizing for CJK-heavy documents. ## Solution 1. **Detect CJK characters by Unicode range:** ```python def _is_cjk(char: str) -> bool: cp = ord(char) return ( 0x4E00 <= cp <= 0x9FFF # CJK Unified Ideographs or 0x3040 <= cp <= 0x309F # Hiragana or 0x30A0 <= cp <= 0x30FF # Katakana or 0x3400 <= cp <= 0x4DBF # CJK Extension A or 0xF900 <= cp <= 0xFAFF # CJK Compatibility ) ``` 2. **Weighted token estimation:** ```python CJK_CHARS_PER_TOKEN = 2.5 LATIN_CHARS_PER_TOKEN = 4.0 def estimate_tokens(text: str) -> int: cjk_count = sum(1 for c in text if _is_cjk(c)) other_count = len(text) - cjk_count return int(cjk_count / CJK_CHARS_PER_TOKEN + other_count / LATIN_CHARS_PER_TOKEN) ``` 3. **Chunk splitting must use token-based accumulation** (not char-based): ```python # BAD: char_limit = token_limit * FIXED_CONSTANT # GOOD: accumulat
- Problem
- Solution
- When to Use
What does the cjk-aware-text-metrics skill do?
CJK/Latin weighted token estimation for multilingual LLM pipelines. Use when processing Japanese/Chinese/Korean text with fixed chars-per-token constants.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill cjk-aware-text-metrics --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.