data-scraper-agent
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
npx skills add mturac/everything-openai-codex --skill data-scraper-agent --agent codex
Same command for any agent — swap --agent for claude-code, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Builds a production-ready, AI-powered data collection agent for public data sources. It runs on a schedule, scrapes data, enriches items with AI, stores results, and learns from user feedback to improve over time. It uses a three-layer flow: COLLECT → ENRICH → STORE, and leverages a free Gemini Flash AI service, free GitHub Actions scheduling, and storage options such as Notion, Sheets, or Supabase.
How it works
- Step 1: Understand the goal by asking user what data to collect, how to extract fields, where to store results, how to enrich, and how often to run.
- Step 2: Design the directory architecture for the agent, including config.yaml, and folders for scraper, ai, storage, data, and GitHub Actions workflow.
- Step 3: Build the Scraper Source templates for data sources using REST, HTML, or RSS patterns. Normalize items to a consistent schema with fields like name, url, source, date_found.
- Step 4: Build the Gemini AI Client with a fallback chain across model versions and batch-friendly API calls for efficiency.
- Step 5: Build the AI Pipeline that processes items in batches, runs a prompt to analyze items, and returns enriched fields such as ai_score, ai_summary, and ai_notes.
- Step 6: Build the Feedback Learning System to persist user decisions and generate a bias prompt from positive/negative signals for future runs.
- Step 7: Build Storage (Notion example) to deduplicate by URL and push items with AI fields when available.
- Step 8: Orchestrate in main.py to fetch data, deduplicate, and store results.
When to use it
- When a user wants to scrape or monitor any public website or API.
- When tracking jobs, prices, news, repos, sports scores, events, or listings is required.
- When automation should run without hosting costs and improve over time via user feedback.
What it can touch
- Scraper sources located under scraper/sources provide fetch functions.
- AI client and pipeline exist at ai/client.py and ai/pipeline.py and interact via prompts and batch processing.
- Storage backends are wired for Notion (storage/notion_sync.py) with optional expansion to sheets or Supabase.
- Environment/config values are read from config.yaml and environment variables (e.g., GEMINI_API_KEY, NOTION_TOKEN).
Caveats
- License: MIT
- Declared risks and limitations are not enumerated beyond standard usage; the skill relies on public data access rules and free-tier quotas (Gemini Flash free tier, GitHub Actions free tier).
- It requires proper environment setup (tokens, database IDs) to operate; misconfiguration prevents execution.
# Data Scraper Agent Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time. **Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase** ## When to Activate - User wants to scrape or monitor any public website or API - User says "build a bot that checks...", "monitor X for me", "collect data from..." - User wants to track jobs, prices, news, repos, sports scores, events, listings - User asks how to automate data collection without paying for hosting - User wants an agent that gets smarter over time based on their decisions ## Core Concepts ### The Three Layers Every data scraper agent has three layers: ``` COLLECT → ENRICH → STORE │ │ │ Scraper AI (LLM) Database runs on scores/ Notion / schedule summarises Sheets / & classifies Supabase ``` ### Free Stack | Layer | Tool | Why | |---|---|---| | **Scraping** | `requests` + `BeautifulSoup` | No cost, covers 80% of public sites | | **JS-rendered sites** | `playwright` (free) | When HTML scraping fails | | **AI enrichment** | Gemini Flash via REST API | 500 req/day, 1M tokens
- When to Activate
- Core Concepts
- The Three Layers
- Free Stack
- AI Model Fallback Chain
- Batch API Calls for Efficiency
- Workflow
- Step 1: Understand the Goal
- Step 2: Design the Agent Architecture
- Step 3: Build the Scraper Source
- Step 4: Build the Gemini AI Client
- Step 5: Build the AI Pipeline (Batch)
- Step 6: Build the Feedback Learning System
- Step 7: Build Storage (Notion example)
What does the data-scraper-agent skill do?
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
How do I install it?
Run `npx skills add mturac/everything-openai-codex --skill data-scraper-agent --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From mturac/everything-openai-codex, a repository with 84 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.