add-dataset
Guide for adding a new dataset loader to AReaL. Use when user wants to add a new dataset.
npx skills add majiayu000/claude-skill-registry --skill add-dataset --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Add Dataset Add a new dataset loader to AReaL. ## When to Use This skill is triggered when: - User asks "how do I add a dataset?" - User wants to integrate a new dataset - User mentions creating a dataset loader ## Step-by-Step Guide ### Step 1: Create Dataset File Create `areal/dataset/<name>.py`: ```python from datasets import Dataset, load_dataset def get_<name>_sft_dataset( path: str, split: str, tokenizer, max_length: int | None = None, ) -> Dataset: """Load dataset for SFT training. Args: path: Path to dataset (HuggingFace hub or local path) split: Dataset split (train/validation/test) tokenizer: Tokenizer for processing max_length: Maximum sequence length (optional) Returns: HuggingFace Dataset with processed samples """ dataset = load_dataset(path=path, split=split) def process(sample): # Tokenize the full sequence (prompt + response) seq_token = tokenizer.encode( sample["question"] + sample["answer"] + tokenizer.eos_token ) prompt_token = tokenizer.encode(sample["question"]) # Loss mask: 0 for prompt, 1 for response loss_mask = [0] * len(prompt_token) + [1] * (len(seq_token) - len(prompt_token)) return {"input_ids": seq_token, "loss_mask": loss_mask} dataset = dataset.ma
- When to Use
- Step-by-Step Guide
- Step 1: Create Dataset File
- Step 2: Register in init.py
- Step 3: Add Config (Optional)
- Step 4: Add Tests
- Reference Implementations
- Required Fields
- SFT Dataset
- RL Dataset
- Common Mistakes
- Purpose
- How to Update
- When Dataset API Changes
What does the add-dataset skill do?
Guide for adding a new dataset loader to AReaL. Use when user wants to add a new dataset.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill add-dataset --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.