download-gated-pdfs
Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).
npx skills add kennethkhoocy/applied-micro-skills --skill download-gated-pdfs --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Download bot-gated PDFs via Wayback id_ ## Problem Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML. ## Context / Trigger Conditions - `curl -o file.pdf <url>` succeeds but the file starts with `<!DOC` - pypdf raises `invalid pdf header: b'<!DOC'` or `PdfStreamError: Stream has ended unexpectedly` - Firecrawl `scrape` returns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content ## Solution 1. Request the file through the Wayback Machine's raw-content (`id_`) endpoint, which serves the original archived binary without rewriting: ```sh curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \ "https://web.archive.org/web/<YYYY>id_/<original-pdf-url>" -o out.pdf ``` `<YYYY>` is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The `id_` suffix after the timestamp is what requests the untouched original. 2. Verify the download with pypdf — a bot page fails immediately: ```pyt
- Problem
- Context / Trigger Conditions
- Solution
- Verification
- Example
- Notes
curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \
What does the download-gated-pdfs skill do?
Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).
How do I install it?
Run `npx skills add kennethkhoocy/applied-micro-skills --skill download-gated-pdfs --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From kennethkhoocy/applied-micro-skills, a repository with 54 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.