RadarTopicsBuildersWeeklyReads
Open Source Radar
agentscope-ai/

OpenJudge

GitHubWebsite

OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

764stars
61forks
14issues
Apache-2.0license
2025since
Star historydaily snapshots by VibeCrowd

Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).

Alternatives & relatedmatched by topic overlap
hiyouga/
LlamaFactory
▲ 162/wk

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

74k9.0kPython
shareAI-lab/
learn-claude-code
▲ 715/wk

Bash is all you need - A nano claude code–like 「agent harness」, built from 0 to 1

73k12kPython
CherryHQ/
cherry-studio

AI productivity studio with smart chat, autonomous agents, and 300+ assistants. Unified access to frontier LLMs

49k4.7kTypeScriptgiant
NousResearch/
hermes-agent
▲ 3.5k/wk

The agent that grows with you

225k44kPython
TauricResearch/
TradingAgents
▲ 693/wk

TradingAgents: Multi-Agents LLM Financial Trading Framework

96k18kPython
bytedance/
deer-flow
▲ 1.2k/wk

An open-source long-horizon SuperAgent harness that researches, codes, and creates. With the help of sandboxes, memories, tools, skill, subagents and message gateway, it handles different levels of tasks that could take minutes to hours.

79k11kPython
Agent skills inside · 18
All skills →
redteamUse when the user wants to test their LLM/agent application for safety and security vulnerabilities — jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.Securityscriptsprompt-regressionUse when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.Design & Presentationscriptsalign-humanUse when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.Data & Analyticsscriptsrag-evalUse when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.Code Review & Qualityscriptseval-designUse when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.Testing & QAscriptsopenjudgeBuild custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.Data & Analytics
SharePost on XLinkedIn
All trending reposRevenue-verified startups →