llm-calibration-logprobs
Analyze LLM token logprobs and calibration. Use for per-decision confidence, ECE, Brier scores, reliability diagrams, and low-confidence triage.
npx skills add majiayu000/claude-skill-registry --skill llm-calibration-logprobs --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Analyzes LLM token logprobs and calibration. Use for per-decision confidence, ECE, Brier scores, reliability diagrams, and low-confidence triage.
How it works
- Focuses on within-model confidence by reading emitted token logprobs and assessing calibration against ground truth, paired with a separate cross-model voting skill for between-coder agreement.
- Provides guidance on collecting logprobs by enabling logprobs in API requests (e.g., OpenAI with logprobs and top_logprobs), and on reproducible decoding settings (temperature=0, fixed seed).
- Describes aggregation strategies across multi-token labels: sum of label-token logprobs, mean per-token logprob, or first-token logprob, and instructs storing corresponding fields (logprob_code_sum, logprob_first_token) and top alternatives (top_alternatives_json).
- Advises capturing top alternatives to measure margins (p_top − p_second) and to locate labels within a structured output to anchor token spans.
- Outlines a three-tier confidence scheme mapped to actions (auto-accept, accept with flag, route to human), with caveats about calibration before relying on thresholds.
- Recommends triage behavior, uncertainty propagation downstream, and reporting requirements (tier shares, calibration metrics).
- Includes reproducibility and reporting requirements: model version, decoding settings, output schema, aggregation rationale, raw per-token logprobs, and calibration metrics (ECE, Brier, reliability diagram).
When to use it
Use when you need within-model confidence signals and calibration assessments for a decision, particularly to supplement or replace hard labels with a probabilistic, calibrated approach. Use alongside the model-council-voting skill for a full picture of reliability across models.
What it can touch
- Tools: claude-code
- Includes instructions to enable logprobs and top_logprobs in API calls, and to store per-token logprob fields and top alternatives for analysis.
Caveats
- Proprietary logprob availability may vary across APIs; logprob values can drift with model updates, batch processing, or non-determinism even with temperature=0. Pin model version and record run date.
- Verbal confidence phrases are not the same as logprobs and should not substitute for the logprob-based calibration signal.
- Calibration metrics require external ground-truth labeling; logprobs alone do not guarantee correctness.
- Best suited for fixed label sets; may be weaker for open-ended tasks.
# Reading a Model's Own Uncertainty from Token Log-Probabilities ## Instructions This skill covers **within-model** confidence: how sure a single model is about each decision it makes, read off the token log-probabilities it emits, and whether that internal confidence is *calibrated* against ground truth. It pairs with the `model-council-voting` skill, which handles the complement — **between-coder** agreement across several independent models. Use both: one model's high self-reported confidence on an item, and three models independently agreeing on that item, are different kinds of evidence, and a careful pipeline reports both. This skill does not cover codebook design or human-validation statistics (κ, F1) — those live in the `text-classification` skill; cross-reference it rather than re-deriving them here. ### 1. What a Token Log-Probability Is, and When to Use It - A generative LLM produces each output token by sampling from a probability distribution over its vocabulary. The **log-probability** (logprob) of the token it actually emitted is `log P(token | context)`; exponentiating gives a probability in `[0, 1]`. A logprob of `0.0` means probability 1.0 (the model treated that
- Instructions
- 1. What a Token Log-Probability Is, and When to Use It
- 2. Collecting Logprobs and Aggregating Multi-Token Labels
- 3. Confidence Tiers, Margins, and Triage
- 4. Calibration Assessment
- 5. Using Uncertainty in the Pipeline
- 6. Caveats
- 7. Reproducibility and Reporting
- Quality Checks
What does the llm-calibration-logprobs skill do?
Analyze LLM token logprobs and calibration. Use for per-decision confidence, ECE, Brier scores, reliability diagrams, and low-confidence triage.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill llm-calibration-logprobs --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.