nan-safe-correlation
Per-feature NaN-safe Spearman/Pearson correlation across many features (genes, proteins, variants) with missing values. Covers why bulk matrix shortcuts fail, correct pairwise deletion, degenerate input filtering, and large-dataset performance. Use statistical-analysis for test choice; shap-model-explainability for interpretability.
npx skills add BioTender-max/awesome-bio-agent-skills --skill nan-safe-correlation --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# NaN-Safe Correlation Computation ## Overview Computing correlations across many features (genes, proteins, variants) when missing values are present is error-prone. The most common mistake is using bulk matrix shortcuts that silently mishandle NaN, producing incorrect correlation values. This guide covers correct per-feature pairwise computation, degenerate input filtering, and performance optimization. ## Key Concepts ### Pairwise vs Listwise Deletion - **Pairwise deletion**: For each feature pair, remove only samples where either value is NaN. Each feature uses the maximum available data. - **Listwise deletion**: Remove any sample with NaN in any feature. Wastes valid data and biases results if missingness is not completely random. - **Rule**: Always use pairwise deletion for per-feature correlations. ### Why Bulk Matrix Shortcuts Fail Different features have different missing value patterns across samples. Bulk methods handle this inconsistently: | Method | Problem | |--------|---------| | `DataFrame.rank()` then `corrwith()` | `rank()` assigns NaN ranks; `corrwith()` may drop globally or per-column inconsistently | | `DataFrame.corrwith(method='spearman')` | Implementation va
- Overview
- Key Concepts
- Pairwise vs Listwise Deletion
- Why Bulk Matrix Shortcuts Fail
- Impact of Incorrect Computation
- Degenerate Inputs
- Decision Framework
- Best Practices
- Common Pitfalls
- Workflow
- Reference Implementation
- Anti-Patterns
- Performance Optimization (> 10,000 features)
- Further Reading
What does the nan-safe-correlation skill do?
Per-feature NaN-safe Spearman/Pearson correlation across many features (genes, proteins, variants) with missing values. Covers why bulk matrix shortcuts fail, correct pairwise deletion, degenerate input filtering, and large-dataset performance. Use statistical-analysis for test choice; shap-model-explainability for interpretability.
How do I install it?
Run `npx skills add BioTender-max/awesome-bio-agent-skills --skill nan-safe-correlation --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From BioTender-max/awesome-bio-agent-skills, a repository with 135 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.