arize-evaluator
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
Profile →npx skills add github/awesome-copilot --skill arize-evaluator --agent copilot
Same command for any agent — swap --agent for claude-code, codex, cursor.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Designs, creates, and runs LLM-as-judge evaluators on Arize. It covers creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mappings, and continuous monitoring. It is intended for use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
How it works
- Provides commands and flags to list/get/create evaluators, create evaluator versions, and update metadata. Examples show using ax evaluators create with required flags such as --name, --space, --template-name, --commit-message, --ai-integration-id, --model-name, --template, --classification-choices. It also documents creating versions and updating metadata without altering prompts.
- Describes how to create tasks that run evaluators against project spans or experiment datasets, including providing column mappings and optional query filters. Supports continuous scoring for project tasks and non-continuous backfill for project tasks, as well as experiment/dataset tasks.
- Details trigger-run commands to execute evaluations over specific data windows or experiments, with options to wait, set timeouts, and cap spans. Includes guidance on monitoring and handling run statuses.
- Notes prerequisites and troubleshooting: ax CLI availability, Arize profile credentials, space selection, and handling of missing or invalid credentials or spaces. Emphasizes not fabricating evaluation results and recommends retrying via Arize UI or credential checks when failures occur.
When to use it
Use when the user requests to create an evaluator, run evaluations, map columns for evaluation, or set up continuous monitoring of evaluations on Arize (e.g., topics like hallucination, correctness, or relevance). Trigger-run and task setup steps apply when executing evaluations on spans or experiments and when continuous scoring is required.
What it can touch
- Tools: copilot
- Requires: ax CLI and a configured Arize profile with an AI integration
- Commands referenced include:
ax evaluators create,ax evaluators create-version,ax evaluators update,ax tasks create,ax tasks trigger-run, plus variousaxflags such as--space,--template-name,--ai-integration-id,--model-name,--template,--classification-choices,--column_mappings(as part of task definitions).
Caveats
- Contains guidance to troubleshoot credential issues, space selection, and API keys without exposing or reading local files. If an evaluation task fails or yields no scores, it forbids manual scoring and suggests fixes and retries.
- Does not state any guaranteed outcomes or results from evaluations; emphasizes using Arize UI or credential checks for remediation.
# Arize Evaluator Skill > **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. This skill covers designing, creating, and running **LLM-as-judge evaluators** on Arize. An evaluator defines the judge; a **task** is how you run it against real data. --- ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see references/ax-setup.md - `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys - Space unknown → run `ax spaces list` to pick by name, or ask the user - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the
- Prerequisites
- Concepts
- What is an Evaluator?
- What is a Task?
- Data Granularity
- How trace and session aggregation works
- The {conversation} template variable
- Multi-evaluator tasks
- Basic CRUD
- AI Integrations
- Evaluators
- Tasks
- Workflow A: Create an evaluator for a project
- Step 1: Confirm the project name
Check for an existing integration first ax ai-integrations list --space SPACE Create if none exists ax ai-integrations create \ List / Get ax evaluators list --space SPACE ax evaluators get ID # accepts name or ID ax evaluators get NAME --space SPACE # required when using name instead of ID ax evaluators list-versions NAME_OR_ID ax evaluators get-version VERSION_ID
What does the arize-evaluator skill do?
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
How do I install it?
Run `npx skills add github/awesome-copilot --skill arize-evaluator --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From github/awesome-copilot, a repository with 37,432 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.