DataChain provides a Python library for turning files in S3, GCS, and Azure into versioned, typed datasets with a Dataset DB backend. It supports agent workflows via knowledge base and data harness components, and emphasizes local storage of processing state and lineage.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
DataChain: The Context Layer for Unstructured Data. A Python library that turns files in S3, GCS, and Azure into versioned, typed datasets, queryable at warehouse speed. It includes Compute Engine, Dataset DB, and optional Knowledge Base and Data Harness components for agent workflows. The knowledge base is derived from the Dataset DB and stored under dc-knowledge/.
How it works
A dataset is a named, versioned result of a pipeline step. Datasets are stored with schemas defined via Pydantic, file pointers, and lineage in a local SQLite DB at .datachain/db. Operations like read_storage, map, save create and update datasets (e.g., pets_images@1.0.0). Filters, joins, and similarity search operate directly against the Dataset DB without loading all files into memory. Vector data (e.g., embeddings) can be stored as list[float] in schemas and queried efficiently. The Knowledge Base and Data Harness enable agent integration with Claude Code, Cursor, Codex, Copilot, and Pi.
Getting started
Install:
pip install datachain
To add the agent skill (Knowledge Base + code generation):
datachain skill install --target claude # also: cursor, codex, copilot, pi
Quickstart examples illustrate constructing a dataset from storage and saving a versioned dataset, e.g., creating pets_images@1.0.0 via a read_storage+map+save pipeline. See README for detailed scripts and prompts.
Recent releases
Latest release: 0.59.5 (2026-08-04) with changes:
- fix: anchor glob patterns at the listing root by @shcheklein in https://github.com/datachain-ai/datachain/pull/1892
Previous releases include 0.59.4 (2026-08-04), 0.59.3 (2026-07-28), 0.59.2 (2026-07-17), 0.59.1 (2026-07-17) with various fixes and updates. Full changelog: https://github.com/datachain-ai/datachain/
Traction
Stars: 2803 Forks: 153 Open issues: 77
Behind the repo
Not provided in the README snippet beyond repository owner and project; no startup/company link information is included in the FACTS block.
Caveats
License: Apache-2.0 Created: 2024-06-25 Last push: 2026-08-04 Programming language: Python Supported storage backends: S3, GCS, Azure, and local filesystems Exports and datasets are stored in a Dataset DB with local SQLite backing; code examples show how to save and query datasets, with incremental updates (delta) and prefetch settings discussed in examples.