site-crawlability
When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite scroll," "pagination," "masonry SEO," "AI crawler optimization," "GPTBot crawlability," "ClaudeBot crawlability," or "content not indexed." For internal links, use internal-links.
npx skills add kostja94/marketing-skills --skill crawlability --agent cursor
Same command for any agent — swap --agent for claude-code, codex, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# SEO Technical: Crawlability Guides crawlability improvements: robots, X-Robots-Tag, site structure, and internal linking. **When invoking**: On **first use**, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On **subsequent use** or when the user asks to skip, go directly to the main output. ## Scope (Technical SEO) - **Redirect chains & loops**: Fix multi-hop redirects; point directly to final URL - **Broken links (4xx)**: Fix broken internal/external links; 301 or remove - **Site architecture**: Logical hierarchy; pages within 3–4 clicks from homepage - **Orphan pages**: Add internal links to pages with no incoming links - **Pagination**: Prefer pagination over infinite scroll for crawlability - **Crawl budget**: Reduce waste on duplicates, redirects, low-value URLs (see below) - **AI crawler optimization**: SSR for critical content; URL management; reduce 404/redirect waste (see below) ## Initial Assessment **Check for project context first:** If `.claude/project-context.md` or `.cursor/project-context.md` exists, read it for site structure. Identify: 1. **Site structure**: Flat vs. deep hierarchy 2. **Framework**:
- Scope (Technical SEO)
- Initial Assessment
- Best Practices
- Redirect Chains & Loops
- Broken Links (4xx)
- Site Architecture
- Pagination vs Infinite Scroll
- Pagination (Traditional)
- Crawl Budget
- AI Crawler Optimization
- Common Issues
- Output Format
- Related Skills
What does the site-crawlability skill do?
When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite scroll," "pagination," "masonry SEO," "AI crawler optimization," "GPTBot crawlability," "ClaudeBot crawlability," or "content not indexed." For internal links, use internal-links.
How do I install it?
Run `npx skills add kostja94/marketing-skills --skill crawlability --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From kostja94/marketing-skills, a repository with 848 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.