Data-Juicer is a Python-based data processing framework for foundation-model workloads, offering modular operators, YAML pipelines, and Ray-based distributed execution. Latest releases show ongoing improvements across VLA ops, robustness, and new video/table ops.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Data-Juicer is a data processing framework designed for foundation-model era workloads. It provides modular operators across text, image, audio, video, and multimodal data, with a recipe-first approach using reproducible YAML pipelines. The project emphasizes composable blocks, hot-reload of operators, and production-ready performance for large-scale AI data processing.
How it works
It exposes a collection of operators (over 200) that can be chained in pipelines defined in YAML. Pipelines can be executed in distributed Ray mode, with features like batch fusion and operator-level environment management. The README highlights capabilities such as deduplication, normalization, semantic chunking, and data profiling as part of its data intelligence suite. A Quick Start demonstrates using a YAML config or composing in Python to process datasets through a sequence of operators.
Getting started
Install and run:
uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml
Or use Python API examples shown in the README to construct and run a processing pipeline with NestedDataset and operators like TextLengthFilter and WhitespaceNormalizationMapper.
Recent releases
- v1.5.4 Release v1.5.4: HumanVBench Video OPs; Batch-local Stage Fusion; Robustness Fixes (2026-07-23)
- New OPs for human-centric video understanding; Batch-local Fusion; robustness improvements and install fixes.
- v1.5.3 Release v1.5.3: VLA Ops Enhancements; Ray Repartition Pipeline; Scalability & Robustness (2026-06-29)
- Expanded embodied-AI processing; Ray repartition pipeline; improved scalability and robustness.
- v1.5.2 Release v1.5.2: Semantic OPs; Agent Interaction Quality; Cross-Document Dedup; Robustness & Performance (2026-05-29)
- Added deduplication, agent data quality toolkit, and leaner install; semantic LLM operators.
- v1.5.1 Release v1.5.1: LaTeX OPs; Compressed Format Support; Operator Robustness Fixes (2026-03-17)
- LaTeX OPs; support for jsonl.gz; documentation and architecture upgrades.
- v1.5.0 Release v1.5.0: Partitioned Ray Executor; Embodied-AI OPs; OP-level Env Management (2026-02-26)
- Partitioned Ray executor; expanded embodied-AI operators; performance and DX improvements.
Traction
- Stars: 6826
- Forks: 403
- Open issues: 62
- Latest push: 2026-08-03
Behind the repo
Not applicable from provided data.
Caveats
- License: Apache-2.0
- Created: 2023-08-01
- Language: Python
- Active development with frequent releases and large code changes across files (e.g., 125 files changed in v1.5.4)