robots-txt
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
npx skills add kostja94/marketing-skills --skill robots --agent cursor
Same command for any agent — swap --agent for claude-code, codex, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# SEO Technical: robots.txt Guides configuration and auditing of robots.txt for search engine and AI crawler control. **When invoking**: On **first use**, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On **subsequent use** or when the user asks to skip, go directly to the main output. ## Scope (Technical SEO) - **Robots.txt**: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks - **Crawler access**: Path-level crawl control; AI crawler allow/block strategy - **Differentiation**: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See **indexing** for page-level exclusions. ## Initial Assessment **Check for project context first:** If `.claude/project-context.md` or `.cursor/project-context.md` exists, read it for site URL and indexing goals. Identify: 1. **Site URL**: Base domain (e.g., `https://example.com`) 2. **Indexing scope**: Full site, partial, or specific paths to exclude 3. **AI crawler strategy**: Allow search/indexing vs. block training data crawlers ## Best Practices ### Purpose and Limitations | Point | Note | |-------|------| | **Pur
- Scope (Technical SEO)
- Initial Assessment
- Best Practices
- Purpose and Limitations
- Crawl vs Index vs Link Equity (Quick Reference)
- When to Use robots.txt vs noindex
- Location and Format
- Core Directives
- Path Format Rules (Critical)
- Critical: Do Not Block
- AI Crawler Strategy
- Clean-param (Yandex)
- Output Format
- Related Skills
What does the robots-txt skill do?
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
How do I install it?
Run `npx skills add kostja94/marketing-skills --skill robots --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From kostja94/marketing-skills, a repository with 848 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.