docetl

What it is

DocETL: Declarative & Agentic Map-Reduce

DocETL helps you process large collections of data (structured and unstructured) with LLMs. You write each operation in natural language, e.g., "pull out every complaint in this ticket," and DocETL

• provides the operators you need (map, reduce, filter, and more) and orchestrates them, parallelizing work across your data,
• optimizes your pipeline automatically, swapping models, rewriting prompts, decomposing operations, and replacing subtasks with code wherever possible, to raise accuracy and cut cost, and
• returns tables, easy to query in your favorite database.

How it works

The project exposes a Python API and a YAML-based low-code approach to declare pipelines. It includes operators such as map, reduce, and others, and optimizes pipelines to improve accuracy and reduce cost. Documentation sections point to guides for Python API, YAML, and operators, as well as an optimization framework referred to as MOAR in docs.

Getting started

Install via:

pip install docetl
export OPENAI_API_KEY=your_key   # or any LLM provider key

Python API (recommended)

import docetl

docetl.default_model = "gpt-4o-mini"
docetl.rate_limits = {
    "llm_call": [{"count": 500, "per": 1, "unit": "minute"}],
    "llm_tokens": [{"count": 200_000, "per": 1, "unit": "minute"}],
}

# Classify support tickets, then summarize each category
pipeline = docetl.read_json("tickets.json")

pipeline = pipeline.map(
    prompt="Classify this support ticket: {{ input.text }}",
    output={"schema": {"category": "str", "priority": "str"}},
)

pipeline = pipeline.reduce(
    reduce_key="category",
    prompt="Summarize these tickets: {% for t in inputs %}{{ t.text }}{% endfor %}",
    output={"schema": {"summary": "str"}},
)

pipeline.schema()  # {'category': 'str', 'summary': 'str'}
pipeline.show()  # run on 5 docs and print results
rows = pipeline.collect()  # full run
print(f"Cost: ${pipeline.total_cost:.4f}")

YAML (low-code)

datasets:
  tickets:
    type: file
    path: tickets.json

default_model: gpt-4o-mini

operations:
  - name: classify
    type: map
    prompt: "Classify this support ticket and assign a priority level."
    output:
      schema:
        category: str
        priority: str

pipeline:
  steps:
    - name: triage
      input: tickets
      operations: [classify]
  output:
    type: file
    path: output.json

docetl run pipeline.yaml

Documentation (Docs/Guides)

• Python API Guide
• YAML Tutorial
• Operators
• Optimization with MOAR
• DocWrangler Setup
• Claude Code Quick Start

Release notes (latest 5)

• 0.3.0 0.3.0 (2026-06-17): Claude code integration guide by @shreyashankar in https://github.com/ucbepic/docetl/pull/471; optimizer: add directives for resolve operator by @shreyashankar in https://github.c
• 0.2.6 0.2.6 (2025-12-28): website: add new showcase example by @shreyashankar in https://github.com/ucbepic/docetl/pull/403; fix showcase bug by @shreyashankar in https://github.com/ucbepic/docetl/pull/404
• 0.2.5 0.2.5 (2025-08-09): feat: Add calibration support to map operations for improved consistency by @shreyashankar in https://github.com/ucbepic/docetl/pull/365; chore: add support for docling-serve v1al
• 0.2.4 0.2.4 (2025-05-21): fix: add system prompt and other config vars to python api by @shreyashankar in https://github.com/ucbepic/docetl/pull/356; Support pandas dataframe as input by @sfc-gh-jdu in htt
• 0.2.3 0.2.3 (2025-04-29): expts: assess performance of structured outputs by @shreyashankar in https://github.com/ucbepic/docetl/pull/291; Handle DeepSeek R1 distil models: no tool calling and add think co

Traction

Stars: 3959 | Forks: 421 | Open issues: 44

About the repo

Language: Python | License: MIT | Created: 2024-07-09 | Last push: 2026-07-21 Topics: agents, data, data-pipelines, document-analysis, document-processing, elt, etl, llm, python, semantic-data, unstructured-data, unstructured-data-analysis, workflow

Caveats

License: MIT Open issues: 44