text-classification

What it does

Guides an AI agent to design and apply an LLM-based text classification workflow for social science data, including codebook structure (labels, definitions, clarifications, negatives, examples), learning regimes (zero-shot, few-shot, fine-tuning, instruction-tuning, encoder-only fine-tuning), model selection and reproducibility practices, prompt construction, pilot testing/validation steps, hybrid human-LLM workflows, analysis and interpretation plans, and comprehensive reporting requirements.

How it works

1. Codebook Design: specifies population and sampling constraints; structures each code with Label, Definition, Clarification, Negative clarification, and Examples; recommends 3–6 codes, allows multi-labels if instructed, includes a residual category (none_of_above or uncodeable); iterates via pilot testing.
2. Choosing a Learning Regime: provides decision framework mapping document characteristics to prompting or training approaches (Zero-shot, Few-shot, Fine-tuning, Instruction-tuning, Encoder-only fine-tuning); suggests testing multiple regimes when possible.
3. Model Selection and Reproducibility: advocates open-weight models for publishable research; if using proprietary models, require exact model identifiers and set temperature to 0; recommends running ~50 responses twice to measure agreement and report variance.
4. Prompt Construction: instructs to place the codebook in the system prompt with all components, specify exact output format (code labels, comma-separated if multi-label), constrain no extra text, and prefer JSON for structured inputs/outputs; include input text in the user message with a consistent delimiter.
5. Pilot Testing and Validation: outlines stages to test label recovery, definition recovery, and in-context labeling, hand-code 50–100 responses with two independent coders, report Cohen’s κ, consider self-coding, compare LLM outputs to ground truth with per-category metrics, and perform error analysis with justifications for misclassifications; specifies an iterative path if zero-shot F1 is inadequate.
6. Hybrid Human-LLM Workflows: describes a hybrid approach with a high target accuracy (93%+), flags for human review using confidence, disagreements, residuals, and boundary cases; estimates 10–15% of responses requiring human review and discusses codebook refinement from flagged cases; provides ensemble workflow guidance.
7. Analysis and Interpretation: recommends reporting code prevalence by subgroups, cross-tabulations, and accounting for measurement error when LLM labels serve as variables in downstream analyses; advises design-based corrections and pre-registration considerations.
8. Reporting: requires full documentation of model name/version, generation parameters, prompt text, codebook, and run dates; mandates validation metrics (per-category precision/recall/F1, overall accuracy, Cohen’s Kappa), variance tests, human-review rates; and archiving prompts, codebooks, and classification code.

When to use it

Use when designing an LLM-based text classification study in social science, especially with a predefined codebook structure and a need for rigorous validation, reproducibility, and hybrid human-in-the-loop workflows.

What it can touch

The skill references tools and concepts such as: codebook components (Label, Definition, Clarification, Negative clarification, Examples), model regimes (Zero-shot prompting, Few-shot prompting, Fine-tuning, Instruction-tuning, Encoder-only fine-tuning), model identifiers for proprietary models, and outputs formatted as code labels (comma-separated for multi-label) or JSON. It instructs running ~50 responses for variance checks and using two independent coders for pilot tests.

Caveats

It notes license and risk considerations around data privacy and proprietary model behavior, suggests open-weight models for reproducibility, and emphasizes reporting risk of future deprecation for proprietary models. It specifies attention to per-category F1 thresholds (0.7 as a practical benchmark) and acknowledges potential limitations in cross-language validation and classifier reliability. It requires documenting full classification pipeline and acknowledging measurement error in downstream analyses.