arize-experiment
Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.
Profile →npx skills add github/awesome-copilot --skill arize-experiment --agent copilot
Same command for any agent — swap --agent for claude-code, codex, cursor.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Creates, runs, exports, and analyzes Arize experiments to evaluate and compare model performance. It covers creating, exporting, and processing experiment runs, and evaluating results with metrics using the ax CLI workflows.
How it works
- Uses the ax CLI to manage experiments: list, get, export, create, and delete.
- Defines concepts: Experiment (a named evaluation run against a dataset version), Experiment Run (one dataset example’s result with output and optional evaluations/metadata), Dataset (versioned collection tied to an experiment), and Evaluation (a named metric per run).
- Provides concrete steps for running an experiment: export dataset → process examples → call the real model API for each example to generate outputs → assemble runs with example_id, output, optional evaluations/metadata → create the experiment with ax experiments create --name ... --dataset ... --space ... --file runs.json; verify with ax experiments get.
- Includes commands and flags for exporting runs, creating experiments, and comparing two experiments via exported run files, plus jq snippets to analyze evaluations.
When to use it
Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.
What it can touch
- Commands and files described under ax experiments and related ax commands (e.g., ax experiments list, get, export, create, delete).
- REST/Flight export behaviors, and example JSON run structures.
- The workflow script examples and the required data file (runs.json) with columns example_id and output.
Caveats
- Requires the ax CLI and a configured Arize profile.
- Do not fabricate outputs; for every dataset example, you must call the real model API specified by the user.
- If credentials or space/project details are missing, follow guidance to set up ax profiles, spaces, and datasets as described in the commands.
# Arize Experiment Skill > **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. ## Concepts - **Experiment** = a named evaluation run against a specific dataset version, containing one run per example - **Experiment Run** = the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata - **Dataset** = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version - **Evaluation** = a named metric attached to a run (e.g., `correctness`, `relevance`), with optional label, score, and explanation The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs. ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see references/ax-setup.md - `401 Unauthorized` / missing API key → run `ax profiles show` to inspe
- Concepts
- Prerequisites
- List Experiments: ax experiments list
- Flags
- Get Experiment: ax experiments get
- Response fields
- Export Experiment: ax experiments export
- REST vs Flight (--all)
- Create Experiment: ax experiments create
- Passing data via stdin
- Required columns in the runs file
- Delete Experiment: ax experiments delete
- Experiment Run Schema
- Evaluation fields
ax experiments list ax experiments list --dataset DATASET_NAME --space SPACE --limit 20 # DATASET_NAME: name or ID (name preferred) ax experiments list --cursor CURSOR_TOKEN ax experiments list -o json ax experiments get NAME_OR_ID ax experiments get NAME_OR_ID -o json ax experiments get NAME_OR_ID --dataset DATASET_NAME --space SPACE # required when using experiment name instead of ID ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --all ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --output-dir ./results
What does the arize-experiment skill do?
Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.
How do I install it?
Run `npx skills add github/awesome-copilot --skill arize-experiment --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From github/awesome-copilot, a repository with 37,432 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.