Autolabel by Refuel AI is a Python library to label, clean and enrich text datasets using LLMs. It supports multiple providers, includes confidence estimation and state management, and provides a 3-step getting started workflow.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Autolabel is a Python library to label, clean and enrich text datasets with any Large Language Models (LLM) of your choice.
How it works
Features include labeling data for NLP tasks (classification, QA, NER, etc.), support for LLMs from providers such as OpenAI, Anthropic, HuggingFace, Google and more, and techniques like few-shot learning and chain-of-thought prompting. It provides confidence estimation and explanations for each output label, and includes caching and state management to minimize costs and experimentation time.
Getting started
Quick Install
pip install refuel-autolabel
Getting started workflow
Autolabel provides a simple 3-step process for labeling data:
- Specify the labeling guidelines and LLM model to use in a JSON config.
- Dry-run to make sure the final prompt looks good.
- Kick off a labeling run for your dataset!
Example usage (from README)
from autolabel import LabelingAgent, AutolabelDataset
agent = LabelingAgent(config='config.json')
Preview an example prompt that will be sent to the LLM:
ds = AutolabelDataset('dataset.csv', config = config)
agent.plan(ds)
Finally, run the labeling:
ds = agent.run(ds)
The output dataframe contains the label column, e.g. MovieSentimentReview_llm_label.
Recent releases
Latest releases include:
- v0.0.16 (2023-10-16): Fix example selector with non string inputs; change label to null label on output when appropriate.
- v0.0.15 (2023-08-28): Fix AUROC bug and allow custom Example Selectors; setting skip keys as true for hashing non basic types.
- v0.0.14 (2023-08-11): Allow label column to be null for running autolabel; version bump to 0.0.10.
- v0.0.13 (2023-08-10): Similar changes related to null label support and version bump.
- v0.0.9 (2023-07-23): Add multilabel classification to validation; consolidate compute F1.
Traction
Stars: 2328; Forks: 160; Open issues: 81
Caveats
License: MIT Created: 2023-03-23; Last push: 2025-03-05 Language: Python