agentsop-metric-design

What it does

Produces a metric function (and a calibration receipt) that other skills consume to guide optimization in LLM pipelines. It emphasizes decomposing evaluation into orthogonal sub-judges, with a bool return during compile and a float return during evaluation. It prescribes calibrating the metric against human judgments and mitigating judge biases.

How it works

• Activate when optimization runs, RAG evaluation, agent benchmarks, prompt tuning with quantitative metrics, or production regression tests.
• Decompose into 3–6 orthogonal sub-judges (e.g., factuality, on-topic, non-hedging, length considerations, and source-citation if needed). For each criterion, create a sub-judge using a dedicated predictor call and define an aggregation rule (AND for hard gates; weighted sum for soft scores).
• Define a single metric function that returns a bool when trace is not None (compile mode) and a float otherwise (eval mode). The bool return enforces hard constraints; the float return provides a gradient-friendly score.
• Implement a deterministic length penalty separate from the sub-judges (explicit length_score), and multiply it into the final float to counteract length bias.
• Include a bias test (length, self-preference, position, rubric-order) on a dev set before calibration.
• Calibrate with human spot-checks (≥20, ideally 30) and require per-sub-judge and aggregate agreement ≥80% before shipping.
• Ship the metric function and a calibration receipt documenting spot checks, bias probes, judge/model IDs, and dates; pair with an optimizer like GEPA or MIPROv2 depending on whether textual feedback is used.

When to use it

• Use when an optimization/benchmarking pipeline consumes metric(example, pred) -> bool|float and where the metric will gate downstream decisions (chunking, retrievers, prompting, etc.).
• Activate before production changes that rely on a quantified evaluation signal to avoid bias-driven artifacts.

What it can touch

• Tools and concepts named exactly in the skill: the metric function, sub-judges via dspy.Predict(Assess), and the compile/eval modes (trace not None vs None) to switch bool/float semantics.
• It references outputs like a calibration receipt, a JSON artifact next to the compiled artifact, and potential optimizers such as dspy.GEPA or MIPROv2.

Caveats

• The metric is the optimizer’s objective; a wrong metric leads to biased or undesired downstream behavior.
• Requires careful calibration (≥80% agreement) before compilation; otherwise, compilation is refused.
• Length penalties are deterministic and must not be delegated to the LLM judge.
• Bias probes and cross-family judge strategies are recommended to reduce self-preference and length biases.