arize-prompt-optimization
Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.
Profile →npx skills add github/awesome-copilot --skill arize-prompt-optimization --agent copilot
Same command for any agent — swap --agent for claude-code, codex, cursor.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Prompts are extracted from LLM trace data, then the skill collects performance signals (annotations, evals, and traces) to feed a data-driven optimization loop executed with the ax CLI. It is intended for users who want to optimize, improve, or debug prompts to enhance output quality.
How it works
- States that prompts live in LLM spans and related attributes, including structured input messages or a serialized input.value fallback.
- Uses production trace data and evaluation signals (annotation..score, eval..score, eval.*.explanation) to identify failures and opportunities.
- Provides a multi-phase workflow: Phase 1 extract current prompt; Phase 2 gather performance data; Phase 3 optimize the prompt using a meta-prompt; Phase 4 iterate through an optimization loop with ax experiments and result comparisons.
- The optimization meta-prompt instructs generating a revised prompt version based on ORIGINAL BASELINE PROMPT and PERFORMANCE DATA, and prescribes an OUTPUT FORMAT as a JSON array of messages along with a brief reasoning section.
- Requires running ax commands (e.g., ax spans export, ax datasets export, ax experiments create/export) and jq queries to align inputs/outputs and evaluations.
- Includes examples of extracting prompts, reconstructing messages, and merging dataset with experiment results to locate optimization targets.
When to use it
Use when the user mentions terms like optimize prompt, improve prompt, prompt engineering, or system prompt improvement, and when you have access to production trace data, annotations/evals, and a configured Arize profile. The prerequisites section notes the need for the ax CLI and a configured Arize profile.
What it can touch
- Tools: copilot (declared)
- Commands and files: ax CLI commands for spaces, traces, datasets, experiments; jq for JSON processing
- Data sources: production trace data, datasets, experiments, and performance signals (annotations and evals)
Caveats
- Prerequisites: Requires the ax CLI and a configured Arize profile. If credentials or space/project configuration are missing, troubleshooting steps are provided (e.g., ax profiles show, ax spaces list).
- The skill instructs concrete command sequences and JSON formatting but does not guarantee outcomes; outcomes depend on data and experimental results.
# Arize Prompt Optimization Skill > **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. ## Concepts ### Where Prompts Live in Trace Data LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation: | Column | What it contains | When to use | |--------|-----------------|-------------| | `attributes.llm.input_messages` | Structured chat messages (system, user, assistant, tool) in role-based format | **Primary source** for chat-based LLM prompts | | `attributes.llm.input_messages.roles` | Array of roles: `system`, `user`, `assistant`, `tool` | Extract individual message roles | | `attributes.llm.input_messages.contents` | Array of message content strings | Extract message text | | `attributes.input.value` | Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available | | `attributes.llm.prompt_template.template` | Template with `{variable}` placeholders (e.g., `"Answer {question} using
- Concepts
- Where Prompts Live in Trace Data
- Finding Prompts by Span Kind
- Performance Signal Columns
- Prerequisites
- Phase 1: Extract the Current Prompt
- Find LLM spans containing prompts
- Export a trace to inspect prompt structure
- Extract prompts from exported JSON
- Reconstruct the prompt as messages
- Phase 2: Gather Performance Data
- From traces (production feedback)
- From datasets and experiments
- Merge dataset + experiment for analysis
Sample LLM spans (where prompts live) ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout Filter by model ax spans export PROJECT --filter "attributes.llm.model_name = 'gpt-4o'" -l 10 --stdout Filter by span name (e.g., a specific LLM call) ax spans export PROJECT --filter "name = 'ChatCompletion'" -l 10 --stdout Export all spans in a trace ax spans export PROJECT --trace-id TRACE_ID Export a single span ax spans export PROJECT --span-id SPAN_ID
What does the arize-prompt-optimization skill do?
Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.
How do I install it?
Run `npx skills add github/awesome-copilot --skill arize-prompt-optimization --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From github/awesome-copilot, a repository with 37,432 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.