long-context
Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill long-context --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Long Context: Extending Transformer Context Windows ## When to Use This Skill Use Long Context techniques when you need to: - **Process long documents** (32k, 64k, 128k+ tokens) with transformer models - **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.) - **Implement efficient positional encodings** (RoPE, ALiBi) - **Train models** with length extrapolation capabilities - **Deploy models** that handle variable-length inputs efficiently - **Fine-tune** existing models for longer contexts with minimal compute **Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation **Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595) ## Installation ```bash # HuggingFace Transformers (includes RoPE, YaRN support) pip install transformers torch # For custom implementations pip install einops # Tensor operations pip install rotary-embedding-torch # Standalone RoPE # Optional: FlashAttention for efficiency pip install flash-attn --no-build-isolation ``` ## Quick Start ### RoPE (Rotary Position Embeddings) ```python import torch import t
- When to Use This Skill
- Installation
- Quick Start
- RoPE (Rotary Position Embeddings)
- ALiBi (Attention with Linear Biases)
- Position Interpolation for LLaMA
- Core Concepts
- 1. RoPE (Rotary Position Embeddings)
- 2. YaRN (Yet another RoPE extensioN)
- 3. ALiBi (Attention with Linear Biases)
- 4. Position Interpolation
- Method Comparison
- Implementation Patterns
- HuggingFace Transformers Integration
HuggingFace Transformers (includes RoPE, YaRN support) pip install transformers torch For custom implementations pip install einops # Tensor operations pip install rotary-embedding-torch # Standalone RoPE pip install flash-attn --no-build-isolation Clone YaRN implementation git clone https://github.com/jquesnelle/yarn cd yarn Fine-tune LLaMA with YaRN
What does the long-context skill do?
Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill long-context --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.