CyberScraper-2077

What it is

CyberScraper 2077 is a Python web scraper with a Streamlit interface that uses AI models (OpenAI, Gemini, and local LLMs) for extraction and parsing. It supports multiple export formats and optional Tor-based scraping, plus various features for stealth and performance.

How it works

The project exposes a Streamlit app for user interaction to input a target URL or query. It leverages AI models for data extraction and supports async operations, smart parsing, and caching to optimize repeated requests. It can export data in JSON, CSV, HTML, SQL, or Excel, and can route through the Tor network or through a local browser for detection avoidance. It includes multi-page scraping (BETA) and a Google Sheets upload option.

Getting started

Installation steps from the README:

1. Clone this repository:

   git clone https://github.com/itsOwen/CyberScraper-2077.git
   cd CyberScraper-2077
   

2. Create and activate a virtual environment:

   virtualenv venv
   source venv/bin/activate  # Optional
   

3. Install the required packages:

   pip install -r requirements.txt
   

4. Install the playwright:

   playwright install
   

5. Set OpenAI & Gemini Key in your environment: Linux/Mac:

   export OPENAI_API_KEY="your-api-key-here"
   export GOOGLE_API_KEY="your-api-key-here"
   

Using LiteLLM

export LITELLM_API_KEY="your-proxy-key"
export LITELLM_BASE_URL="http://localhost:4000/v1"
export LITELLM_MODELS="my-gpt-model,my-claude-model"

The configured models appear in the sidebar as litellm:<model>.

Using Ollama

1. Setup Ollama using `pip install ollama`
2. Download Ollama from the official website: https://ollama.com/download
3. Now type: ollama pull llama3.1
4. Now follow the rest of the steps below.

🚀 Usage

1. Start the app:

   source venv/bin/activate # If not yet done
   streamlit run main.py
   

2. Open http://localhost:8501 and enter a URL or ask for data extraction.
3. Instruct the chatbot to extract data in a chosen format. Select the desired data to export.

🌐 Multi-Page Scraping (BETA)

Supports scraping multiple pages by providing URL structures or explicit ranges, with automatic pattern detection recommended.

How to use:

• Basic Usage examples for multiple pages:

  https://example.com/page 1-5
  https://example.com/p/ 1-6
  https://example.com/xample/something-something-1279?p=1 1-3
  

• Custom ranges and patterns are supported as described in the README.

🧅 Tor Network Scraping

Supports scraping .onion URLs via Tor with prerequisites for Tor installation and Python packages. Configuration options are shown in the code example, including tor_config with ports and flags.

🔐 Google Sheets setup

Steps to create OAuth credentials and configure redirect origins for Google Sheets integration are provided.

⚙️ Adjusting PlaywrightScraper settings

Optional settings include use_stealth, simulate_human, use_custom_headers, hide_webdriver, bypass_cloudflare, with example Python syntax showing a settings block.

🧪 License and disclaimer

Licensed under MIT. The README contains a disclaimer about use and safety.