ai-crawler-py

What it is

The project is titled AI-Crawler and is described as an experimental data extraction app by Oxylabs AI Studio. It uses AI algorithms to crawl a given domain, identify relevant pages based on a natural language prompt, and extract structured JSON or Markdown output data. It supports schema-based parsing for JSON output and offers multiple output formats.

How it works

The tool follows a four-step process:

1. Provide a starting URL
2. Describe the content to retrieve with a natural language prompt
3. Select the output format (JSON or Markdown)
4. If using JSON output, provide a schema to guide parsing and structuring

Key features include: starting from a URL, natural language prompts to guide extraction, AI-assisted URL selection, and output in JSON or Markdown with optional schema-based parsing.

Getting started

Installation guidance is provided:

• Ensure API key access and Python 3.10+
• Install the package with: ```bash pip install oxylabs-ai-studio

Usage provided in Python shows initializing AiCrawler with an API key, generating a schema from a prompt, and crawling with parameters like url, user_prompt, output_format, schema, render_javascript, return_sources_limit, and geo_location. Code examples include:
```python
from oxylabs_ai_studio.apps.ai_crawler import AiCrawler
import json

# Initialize the AI Crawler with your API key
crawler = AiCrawler(api_key="your_api_key")

# Generate a schema automatically from natural language
schema = crawler.generate_schema(prompt="want to parse name, platform, price")
print(f"Generated schema: {schema}")

# Crawl a website and extract structured data
url = "https://sandbox.oxylabs.io/products"
result = crawler.crawl(
    url=url,
    user_prompt="Find all Halo games for Xbox",
    output_format="json",
    schema=schema,
    render_javascript=False,
    return_sources_limit=3,
    geo_location="US",
)

# Print the crawl output as JSON
print("Results:")
print(json.dumps(result.data, indent=2))
```

It also notes a link to PyPI for additional SDK guidance.

## Recent releases
The latest releases section shows no entries (none).

## Traction
The repository has 3123 stars and 11 forks, with 1 open issue as of the provided data.

## Behind the repo
The README references Oxylabs AI Studio and a link to their services, but no corporate or startup entity details beyond that are provided in the facts.

## Caveats
- License: none listed
- Created: 2025-09-17
- Last push: 2026-04-02
- Language: not specified in the facts

## Additional notes
- Output samples show JSON structures with data items and source URLs and mention that Markdown output is also available if requested. 
- Parameters documented include url, user_prompt, output_format (default markdown), schema (required for JSON), render_javascript (default False), return_sources_limit (default 25), geo_location.