text-classification

What it does

Design and validate LLM-based text classification. Used for codebooks, prompts, validation samples, agreement statistics, and reporting model-coded data.

How it works

• Provides structured instructions across multiple phases:
  • Codebook Design: specify population, sampling frame, treatment, and build codes with Label, Definition, Clarification, Negative clarification, and Examples. Keep codes small (3-6), allow multi-labels if needed, include a residual category, and iterate via pilot testing.
  • Choosing a Learning Regime: outlines zero-shot, few-shot, fine-tuning, instruction-tuning, and encoder-only fine-tuning; recommends testing multiple regimes and selecting by empirical performance.
  • Model Selection and Reproducibility: favors open-weight models, requires exact model identifiers for proprietary models, sets temperature to 0 for classification, and suggests reliability checks like repeated runs and per-language validation.
  • Prompt Construction: place the codebook in the system prompt, specify exact output format (labels only, comma-separated if multi-label), use JSON for inputs/outputs, and separate response text with a consistent delimiter.
  • Pilot Testing and Validation: tasks include Halterman & Keith stage tests, hand-coding 50–100 responses, inter-coder reliability, self-coding diagnostics, per-category precision/recall/F1 reporting, and diagnostic prompts for misclassified cases.
  • Hybrid Human-LLM Workflows: implement classification with human adjudication, flag uncertain cases, and plan for 10–15% human review, with ensemble design guidance and NA handling.
  • Analysis and Interpretation: report prevalence, co-occurrence, measurement-error considerations, and design corrections; treat LLM outputs as one instrument, not ground truth.
  • Reporting: document model names/versions, parameters, prompts, codebook, and classification runs; report metrics and archive prompts/code.

When to use it

• Use when designing or validating LLM-based text classification workflows for social science research, especially where formal codebooks, multi-step prompting, and rigorous validation are required. Triggered when you need a repeatable, auditable pipeline with explicit coding schemas, reliability testing, and hybrid human-in-the-loop adjudication.

What it can touch

• Tools: claude-code is listed as a declared tool

Caveats

• License: MIT
• Notes: The skill emphasizes open-weight models for reproducibility, explicit versioning for proprietary models, and careful handling of multi-language validation, data privacy, and potential model refusals on sensitive content. It prescribes no guaranteed outcomes and relies on validation metrics (precision, recall, F1, Cohen's Kappa) and pilot testing thresholds as house defaults rather than universal guarantees.