HuggingFace Datasets is a Python library that provides access to a large hub of ready-to-use datasets and data manipulation tools, with multiple installation options and feature support. It emphasizes one-line dataset loading, multi-format support, and integration with ML frameworks.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
🤗 Datasets is a lightweight library providing two main features:
- one-line dataloaders for many public datasets loaded from the Hugging Face Hub or local files
- efficient data pre-processing for public and local datasets in various formats
How it works
The library offers a single core function, datasets.load_dataset(dataset_name, **kwargs), to instantiate datasets. It supports streaming mode, multi-modal data, Apache Arrow backend, caching, and multi-framework interoperability (NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, Spark). It also provides JSON type support and built-in search/index features via FAISS and Elasticsearch.
Key capabilities include streaming datasets (streaming=True), multi-processing (map(num_proc=N)), and native conversion to/from multiple data formats and frameworks.
Getting started
Installation
pip install datasets
For the latest development version:
pip install "datasets @ git+https://github.com/huggingface/datasets.git"
Optional dependencies
# For audio (torchcodec)
pip install datasets[audio]
# For image/video (Pillow, torchcodec)
pip install datasets[vision]
# For PDFs/NIfTI (pdfplumber, nibabel)
pip install datasets[pdfs,nibabel]
# For PyTorch/TensorFlow/JAX integration
pip install datasets[torch,tensorflow,jax]
Quick Start
Examples show loading a dataset:
from datasets import load_dataset
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])
...and processing with map, tokenization, and streaming usage like:
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
print(example["image"])
break
Core Classes
- Dataset: in-memory / memory-mapped dataset backed by Apache Arrow
- IterableDataset: lazy, streamable dataset for large-scale processing
Add a new dataset to the Hub
A detailed step-by-step guide exists for adding datasets to the Hub, including uploading via web, Python, or Git.
Disclaimers
Users are advised to pin the revision of repositories for reproducibility.
Contributing
The project provides a Contributing Guide with guidelines on issues, PRs, code style, testing, and documentation.