A repository providing a dataset of in-the-wild jailbreak prompts for LLMs, with code and evaluation tools. It aggregates 15,140 prompts (including 1,405 jailbreak prompts) from Reddit, Discord, websites, and datasets, and offers loading via HuggingFace datasets.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
This repository presents a dataset and related tooling for in-the-wild jailbreaking prompts for large language models. It collects 15,140 prompts from December 2022 to December 2023, including 1,405 jailbreak prompts, across Reddit, Discord, websites, and open-source datasets. It references a framework called JailbreakHub and provides a dataset loader through HuggingFace Datasets.
How it works
The dataset compiles prompts from multiple platforms and sources, annotates jailbreak versus non-jailbreak prompts, and offers a loading interface via the HuggingFace Datasets library. It includes a data table summarizing sources, counts, and time ranges, and provides sample code to load the dataset:
from datasets import load_dataset
dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'jailbreak_2023_05_07', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'jailbreak_2023_12_25', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'regular_2023_05_07', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'regular_2023_12_25', split='train')
The data files are also available in the repository under data.
Getting started
The README shows how to load the dataset with HuggingFace Datasets and notes that preprocessing prompts to remove duplicates is recommended. It also provides links to the dataset page and mentions a supplementary question set for evaluation.
Recent releases
There are no releases listed for this repository.
Traction
The repository has 3763 stars and 327 forks, with 4 open issues.
Behind the repo
No explicit founder or company information is provided in the README beyond author names and affiliations in the associated paper.
Caveats
License is MIT. The README includes a disclaimer about potentially harmful content and notes the dataset is intended for research purposes only and misuse is prohibited. The project includes guidance to use the dataset for evaluation and mentions ethical considerations.