langsmith-evaluator
INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.
npx skills add majiayu000/claude-skill-registry --skill langsmith-evaluator --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
<oneliner> Three core components: **(1) Creating Evaluators** - LLM-as-Judge, custom code; **(2) Defining Run Functions** - capture agent outputs/trajectories for evaluation; **(3) Running Evaluations** - locally with `evaluate()` or auto-run via uploaded evaluators. Python and TypeScript examples included. </oneliner> <setup> Environment Variables ```bash LANGSMITH_API_KEY=lsv2_pt_your_api_key_here # Required LANGSMITH_PROJECT=your-project-name # Check this to know which project has traces LANGSMITH_WORKSPACE_ID=your-workspace-id # Optional: for org-scoped keys OPENAI_API_KEY=your_openai_key # For LLM as Judge ``` **IMPORTANT:** Always check the environment variables or `.env` file for `LANGSMITH_PROJECT` before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one. Python Dependencies ```bash pip install langsmith langchain-openai python-dotenv ``` CLI Tool (for uploading evaluators) ```bash curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh ``` JavaScript Dependencies ```bash npm instal
- Golden Rule: Inspect Before You Implement
- Offline vs Online Evaluators
- LLM as Judge Evaluators
- Custom Code Evaluators
- Defining Run Functions
- Capturing Trajectories
- Uploading Evaluators to LangSmith
- Running Evaluations
- Common Issues
pip install langsmith langchain-openai python-dotenv curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh npm install langsmith openai List all evaluators langsmith evaluator list Upload offline evaluator (attached to dataset) langsmith evaluator upload my_evaluators.py \ Upload online evaluator (attached to project) Delete langsmith evaluator delete "Trajectory Match"
What does the langsmith-evaluator skill do?
INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill langsmith-evaluator --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.