llm-calibration-logprobs
LLM token logprobs and calibration. Per-decision confidence, ECE, Brier, reliability diagrams, low-confidence triage.
npx skills add majiayu000/claude-skill-registry --skill llm-calibration-logprobs-scdenney-open-science-skills --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Gauges a model's internal confidence per decision by reading emitted token log-probabilities, and compares calibration against ground truth. It explains how to collect logprobs, aggregate them across multi-token labels, map probabilities to confidence tiers, and use calibration metrics.
How it works
The skill instructs you to:
- Turn on logprob return at the API/server level (logprobs=True, top_logprobs=K; K ≤ 20). Ensure the response includes per-token logprobs and top alternatives.
- Fix decoding for reproducibility with temperature=0 and a fixed seed (e.g., seed=42). Note that this does not guarantee identical logprob values across calls on hosted APIs.
- For multi-token labels, reconstruct the full output string, locate the label span, map tokens to that span, and aggregate logprobs into one per-item confidence. Options include:
- Sum of label-token logprobs (logprob_code_sum)
- Mean per-token logprob (token-count-normalized)
- First-token logprob (logprob_first_token) State which aggregation you used and why.
- Capture top alternatives at the first label position (top_alternatives_json) to assess margins between chosen and runner-up.
- Ensure the output uses a rigid schema (e.g., JSON with a "code" field) to locate label tokens.
- Map aggregated logprob to confidence tiers:
- p ≥ 0.90 auto-accept
- 0.65 ≤ p < 0.90 accept but flag for spot-check
- p < 0.65 route to human review Also consider the margin to the top alternative as a secondary trigger.
- Route low-confidence items to humans rather than discarding them, and propagate uncertainty downstream when used in downstream analyses.
- Report the share of the corpus in each tier and the calibration metrics (ECE, Brier) along with a reliability diagram.
- Reproduce and report: model identifier/version, decoding settings, aggregation function, raw per-token logprobs, and the derived metrics.
When to use it
Use this skill when you need within-model confidence signals and calibration checks to decide how to handle each item (auto-accept, flag, or human review) and to quantify calibration quality with ROC-like reliability diagrams and Brier/ECE metrics. Do not substitute logprobs for ground-truth calibration without external validation.
What it can touch
The skill specifies tools and outputs for:
- Accessing and returning per-token logprobs and top alternatives from the model API (logprobs, top_logprobs, K).
- Structuring outputs with fields: logprob_first_token, logprob_code_sum, top_alternatives_json, and the label span mapping for aggregation.
Caveats
- Proprietary logprob availability may vary by API; logprob values can drift across model snapshots or infrastructure even with temperature=0. Pin model version and record run date if possible.
- Verbal confidence is not a reliable substitute for logprob; treat chain-of-thought or verbal ratings as a weaker, separate signal.
- Calibration metrics depend on ground-truth labels; ensure evaluation uses held-out human-labeled data.
- Calibration applicability is higher for closed-set tasks with enumerable labels; open-ended generation yields weaker aggregation usefulness.
# Reading a Model's Own Uncertainty from Token Log-Probabilities ## Instructions This skill covers **within-model** confidence: how sure a single model is about each decision it makes, read off the token log-probabilities it emits, and whether that internal confidence is *calibrated* against ground truth. It pairs with the `model-council-voting` skill, which handles the complement — **between-coder** agreement across several independent models. Use both: one model's high self-reported confidence on an item, and three models independently agreeing on that item, are different kinds of evidence, and a careful pipeline reports both. This skill does not cover codebook design or human-validation statistics (κ, F1) — those live in the `text-classification` skill; cross-reference it rather than re-deriving them here. ### 1. What a Token Log-Probability Is, and When to Use It - A generative LLM produces each output token by sampling from a probability distribution over its vocabulary. The **log-probability** (logprob) of the token it actually emitted is `log P(token | context)`; exponentiating gives a probability in `[0, 1]`. A logprob of `0.0` means probability 1.0 (the model treated that
- Instructions
- 1. What a Token Log-Probability Is, and When to Use It
- 2. Collecting Logprobs and Aggregating Multi-Token Labels
- 3. Confidence Tiers, Margins, and Triage
- 4. Calibration Assessment
- 5. Using Uncertainty in the Pipeline
- 6. Caveats
- 7. Reproducibility and Reporting
- Quality Checks
What does the llm-calibration-logprobs skill do?
LLM token logprobs and calibration. Per-decision confidence, ECE, Brier, reliability diagrams, low-confidence triage.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill llm-calibration-logprobs-scdenney-open-science-skills --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.