scientific-data-preprocessing

What it does

This skill instructs the agent to consult before any data preprocessing task to prevent catastrophic errors, using multi-level feature analysis, data leakage detection, and semantic validation across grouped data (time-series, panel). It mandates consultation for planning, reflection, and auditing, and provides concrete patterns for data type classification, within-group vs global processing decisions, and various preprocessing steps such as imputation, scaling, encoding, feature engineering, and sliding window features. It emphasizes that the task remains limited to preprocessing and does not decide modeling choices.

How it works

• Triggered before, during, and after preprocessing to ensure audit and validation at each stage.
• Guides the agent through a structured workflow: classify features (binary, categorical, continuous), assess data structure (grouped time-series, panel data), validate data quality, handle missing values within groups, encode categoricals, standardize features within groups or globally depending on scope, and create window-based features for momentum.
• Includes concrete code patterns and examples for within-group interpolation, within-group standardization, one-hot encoding, and sliding window features, as well as checks for data leakage (ID leakage, causal inversion, meaningless numeric codes) and data quality validation.
• Provides a decision tree to choose processing scope (within_group vs global) using intraclass correlation-based logic and a pattern for automatic scope detection.
• Supplies guardrails and boundaries: does not handle purely cross-sectional data, does not choose models, and assumes data fits in memory.

When to use it

• Trigger keywords include: preprocess, preprocessing, data cleaning, data preparation, standardize, normalize, scale, transform, impute, fill missing, handle NaN, encode, one-hot, categorical, feature engineering, feature selection, feature construction, train test split, cross validation split, interpolate, smooth, aggregate.
• Use at the mandatory checkpoints: BEFORE planning preprocessing strategy, DURING reflection and error checking, AFTER comprehensive validation and audit.
• Applicable to data with groups (e.g., match_id, patient_id, session_id) and when goal is within-group or cross-group comparison guided by stability of variance.

What it can touch

• The skill references tools/libraries in examples such as pandas (pd), scikit-learn (StandardScaler, OneHot), numpy, scipy for interpolation and scaling, and groupby transforms. It also shows usage of functions like get_dummies, interpolate, rolling, and StandardScaler.
• It specifies inputs required before proceeding: confirmation of groups, understanding of within-group vs cross-group goal, and domain constraints on data ranges/units.

Caveats

• License: Apache-2.0 (as per the repository metadata).
• It does not make domain-specific feature engineering decisions; business logic must be supplied by the user.
• It does not handle distributed/big data infrastructure; assumes data in memory.
• It warns about potential pitfalls like cross-group interpolation, time-travel leakage, and misinterpreting IDs as numeric features.