arize-prompt-optimization
INVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data, evaluations, and annotations. Covers extracting prompts from spans, gathering performance signal, and running a data-driven optimization loop using the ax CLI.
npx skills add majiayu000/claude-skill-registry --skill arize-prompt-optimization --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Instructs the agent to work with production trace data to optimize, improve, or debug LLM prompts. It covers extracting prompts from various span attributes, collecting performance signals from annotations and evals, and running a data-driven optimization loop using the ax CLI.
How it works
The skill provides concrete commands and data extraction steps, including:
- Locate LLM-related spans and extract prompts from fields like
attributes.llm.input_messages,attributes.llm.prompt_template, andattributes.input.value. It specifies how to reconstruct prompts as messages and preserve template placeholders. - Gather performance signals from trace data and evaluation results, including
annotation.*.label,annotation.*.score,eval.*.label,eval.*.score, andeval.*.explanationto inform optimization. - Use the ax CLI for exporting traces, datasets, experiments, and for running optimization loops, including phase-based steps (Phase 1: Extract the Current Prompt; Phase 2: Gather Performance Data; Phase 3: Optimize the Prompt; Phase 4: Iterate).
- Provide a meta-prompt structure for prompt optimization and guidance on preserving template variables and output formats when updating prompts.
When to use it
Use when optimizing, improving, or debugging LLM prompts with production trace data, evaluations, and annotations. Trigger points include extracting current prompts from traces, collecting performance signals to identify failures, and iterating an optimization loop using ax experiments and evaluations.
What it can touch
- Tools:
claude-code(declared tool). It referencesaxCLI and JSON/JQ data processing utilities in shell blocks. - Files touched: multiple steps involve tracing data files (spans.json, traces, datasets, experiments) and JSON manipulation via
jq.
Caveats
- The skill relies on the presence of production trace data and the ax CLI being installed and configured with credentials.
- It requires environment prerequisites such as
axinstallation, API keys, and a project/context for ax commands. - All steps are described as commands and data operations; no guarantees of success or outcomes are stated in the material.
# Arize Prompt Optimization Skill ## Concepts ### Where Prompts Live in Trace Data LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation: | Column | What it contains | When to use | |--------|-----------------|-------------| | `attributes.llm.input_messages` | Structured chat messages (system, user, assistant, tool) in role-based format | **Primary source** for chat-based LLM prompts | | `attributes.llm.input_messages.roles` | Array of roles: `system`, `user`, `assistant`, `tool` | Extract individual message roles | | `attributes.llm.input_messages.contents` | Array of message content strings | Extract message text | | `attributes.input.value` | Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available | | `attributes.llm.prompt_template.template` | Template with `{variable}` placeholders (e.g., `"Answer {question} using {context}"`) | When the app uses prompt templates | | `attributes.llm.prompt_template.variables` | Template variable values (JSON object) | See what values were substituted into the template | |
- Concepts
- Where Prompts Live in Trace Data
- Finding Prompts by Span Kind
- Performance Signal Columns
- Prerequisites
- Install ax
- Verify environment
- Default Project
- Phase 1: Extract the Current Prompt
- Find LLM spans containing prompts
- Export a trace to inspect prompt structure
- Extract prompts from exported JSON
- Reconstruct the prompt as messages
- Phase 2: Gather Performance Data
List LLM spans (where prompts live) ax spans list PROJECT_ID --filter "attributes.openinference.span.kind = 'LLM'" --limit 10 Filter by model ax spans list PROJECT_ID --filter "attributes.llm.model_name = 'gpt-4o'" --limit 10 Filter by span name (e.g., a specific LLM call) ax spans list PROJECT_ID --filter "name = 'ChatCompletion'" --limit 10 Export all spans in a trace ax spans export --trace-id TRACE_ID --project PROJECT_ID Export a single span ax spans export --span-id SPAN_ID --project PROJECT_ID
What does the arize-prompt-optimization skill do?
INVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data, evaluations, and annotations. Covers extracting prompts from spans, gathering performance signal, and running a data-driven optimization loop using the ax CLI.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill arize-prompt-optimization --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.