count-dataset-tokens
This skill provides guidance for counting tokens in datasets using specific tokenizers. It should be used when tasks involve tokenizing dataset content, filtering data by domain or category, and aggregating token counts. Common triggers include requests to count tokens in HuggingFace datasets, filter datasets by specific fields, or use particular tokenizers (e.g., Qwen, DeepSeek, GPT).
npx skills add majiayu000/claude-skill-registry --skill count-dataset-tokens --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Count Dataset Tokens ## Overview This skill guides the process of counting tokens in datasets, typically from HuggingFace Hub or similar sources. These tasks involve loading datasets, filtering by specific criteria (domains, categories, splits), tokenizing text fields, and computing aggregate statistics. ## Workflow ### Phase 1: Understand the Dataset Structure Before writing any code, thoroughly examine the dataset documentation and structure: 1. **Read the README/dataset card completely** - Look for: - Available splits (train, test, validation, etc.) - Column/field definitions and their exact names - Domain or category definitions (exact values used) - Data types for each field - Any metadata subsets available 2. **Explore the actual data** - Write exploratory code to: - List all available columns - Check unique values for categorical fields (especially filter fields like "domain") - Verify field names match documentation - Examine sample records to understand data format 3. **Document findings before proceeding** - Note: - Exact field names to use - Exact categorical values for filtering - Any discrepancies between documentation and actual data ### Phase 2: Clarify Task Requir
- Overview
- Workflow
- Phase 1: Understand the Dataset Structure
- Phase 2: Clarify Task Requirements
- Phase 3: Implement Token Counting
- Phase 4: Verify Results
- Common Pitfalls
- 1. Misinterpreting Filter Criteria
- 2. Insufficient Dataset Exploration
- 3. Incomplete Documentation Review
- 4. Silent Filter Failures
- 5. Ignoring Null/Empty Values
- 6. Overconfident Assumptions
- Verification Checklist
What does the count-dataset-tokens skill do?
This skill provides guidance for counting tokens in datasets using specific tokenizers. It should be used when tasks involve tokenizing dataset content, filtering data by domain or category, and aggregating token counts. Common triggers include requests to count tokens in HuggingFace datasets, filter datasets by specific fields, or use particular tokenizers (e.g., Qwen, DeepSeek, GPT).
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill count-dataset-tokens --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.