yc-jobs-scraper
Scrape daily job listings from YCombinator's Workatastartup platform without duplicates. Use this skill when asked to scrape YC jobs, update the YC companies list, or retrieve the latest startup jobs. It handles authentication, extracts company slugs via Inertia.js JSON payloads, falls back to public YC job pages when necessary, and maintains a local SQLite database to track historical jobs and prevent duplicates.
npx skills add Varnan-Tech/opendirectory --skill yc-jobs-scraper --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# YC Jobs Scraper This skill provides a robust architecture for scraping jobs from YCombinator and `workatastartup.com`. It is designed to run automatically, bypass login bottlenecks, and maintain state to never scrape duplicate jobs. ## Architecture The scraper uses a hybrid approach to maximize reliability and minimize bot detection: 1. **Authentication:** `scripts/auth.js` uses Playwright to let a human log in once and saves the session to `scripts/state.json`. 2. **Database:** `scripts/db.js` uses `better-sqlite3` to manage `scripts/jobs.db`. It tracks every `company_slug` and `job_id` ever seen. 3. **Primary Extraction:** `scripts/scraper.js` loads `state.json`, visits YC query URLs, and extracts company slugs from the hidden Inertia.js `data-page` JSON payload. 4. **Job Extraction (JSON):** It then visits the authenticated company pages (`/companies/[slug]`) to extract jobs from the backend JSON payload to ensure we get the real `job_id` for accurate deduplication. 5. **Job Extraction (Fallback):** If the JSON extraction fails, it falls back to parsing public HTML job cards from `ycombinator.com/companies/[slug]/jobs`. ## Workflows ### 1. First-Time Setup If this is the first
- Architecture
- Workflows
- 1. First-Time Setup
- 2. Authentication (Manual Step)
- 3. Running the Daily Scraper
- 4. Querying the Database
cd @path/scripts
npm install
npx playwright install
node auth.js
node scraper.js
node -e "const db = require('better-sqlite3')('jobs.db'); console.log('Companies:', db.prepare('SELECT COUNT(*) as count FROM companies').get().count);"What does the yc-jobs-scraper skill do?
Scrape daily job listings from YCombinator's Workatastartup platform without duplicates. Use this skill when asked to scrape YC jobs, update the YC companies list, or retrieve the latest startup jobs. It handles authentication, extracts company slugs via Inertia.js JSON payloads, falls back to public YC job pages when necessary, and maintains a local SQLite database to track historical jobs and prevent duplicates.
How do I install it?
Run `npx skills add Varnan-Tech/opendirectory --skill yc-jobs-scraper --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Varnan-Tech/opendirectory, a repository with 571 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.