Promptsbeat is a red-teaming toolkit for LLM safety evaluation that generates adversarial cases from seeds, evaluates targets, and produces reports. The repo hosts runnable configurations, dataset templates, and documentation.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (2 so far).
What it is
Promptbeat is a scenario-driven red-teaming toolchain for LLMs and agent applications. It turns targets, scenarios, seed sources, attack recipes, and provider adapters into reproducible red-team cases, generates attacks, evaluates real targets, normalizes findings, and produces reports and dataset artifacts.
How it works
Core workflow transforms: target profile + scenario + seed files/dataset subscriptions + attack recipe + provider/adapter → generated attack cases → evaluation against real targets → normalized findings and evidence → reports/benchmark artifacts.
Key concepts include Target, Scenario, Seed, Subscription, Attack Recipe, and Provider/Adapter. Evidence is preserved through the process.
Getting started
Quick Start commands from the README:
"Download and unpack the release package for your platform, then run Promptbeat from the package root."
./bin/promptbeat --version
./bin/promptbeat validate \
--config examples/llm-basic/promptbeat.yaml
export OPENAI_API_KEY="sk-..."
./bin/promptbeat generate \
--config examples/llm-basic/promptbeat.yaml \
--output artifacts/llm-basic/cases.json
./bin/promptbeat eval \
--config artifacts/llm-basic/promptfoo.redteam.yaml \
--output-dir artifacts/llm-basic/eval
./bin/promptbeat report \
--input artifacts/llm-basic/eval/evaluation_result.json \
--output artifacts/llm-basic/report.html
Recent releases
Latest releases include:
- v0.1.2 (2026-05-31): # promptbeat v0.1.2
- v0.1.0 (2026-05-09): # promptbeat v0.1.0
For each release, the README notes a brief description: "Red-teaming toolkit for LLM safety evaluation. Generate adversarial test cases from curated seeds and evaluate model robustness." with a truncated "Features" list.
Traction
Stars: 997 Forks: 5 Open issues: 0
Caveats
License: none listed Created: 2026-05-09 Last push: 2026-06-18 Language: MDX Topics: agents, ai-security, ai-security-testing, aisecurity