sn-da-large-file-analysis
万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet 转换加速、内存优化、分块处理和大文件写入模式。**遇到以下任一情况就主动使用本 skill**：①数据行数 ≥ 10k（由 sn-da-excel-workflow 的行数评估步骤触发）；②用户出现触发词：大文件 / 大数据量 / 性能优化 / 内存不足 / OOM / 百万行 / 十万行 / 流式读取 / Parquet / 分块处理 / large file / big data / streaming read / chunked processing；③直接使用 pd.read_excel() 导致超时或内存溢出；④用户明确要求对大规模数据集进行高性能处理。仅不用于：小于 10k 行的常规 Excel 分析（使用 sn-da-excel-workflow 即可）。
npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Large Scale Excel Analysis Skill ## Mandatory Rules > **When total rows >= 10,000, you MUST use the methods in this skill.** | Data Scale | Read Strategy | Reason | |-----------|---------------|--------| | < 10k rows | `pd.read_excel()` directly | No memory pressure | | 10k–100k rows | `pd.read_excel()` → convert to Parquet → `pd.read_parquet()` for analysis | Avoid repeated slow reads | | 100k–1M rows | **openpyxl `read_only` + `iter_rows` streaming** → Parquet | `pd.read_excel()` will OOM or timeout | | > 1M rows | Streaming read + **multi-sheet split** (Excel max 1,048,576 rows per sheet) | Must chunk | **Prohibited:** - Do NOT use `pd.read_excel()` to fully load 100k+ row files - Do NOT search for fonts with `fc-list`, `find ... fonts`, or install packages with `pip install` - Do NOT use `df.iterrows()` on large DataFrames (use `itertuples()` or vectorized ops) - Do NOT use `df.apply(lambda...)` for operations that can be vectorized --- ## Environment Setup ```python import pandas as pd import numpy as np import os import gc pd.options.mode.copy_on_write = True # CJK font setup (fixed paths — do NOT search for fonts) # ⚠️ Copy this block as-is. Do NOT use fc-list, find, subpr
- Mandatory Rules
- Environment Setup
- Core Method 1: Inspect File Structure (Without Loading Data)
- Core Method 2: Streaming Read → Parquet (100k+ Rows)
- Core Method 3: Medium File Parquet Conversion (10k–100k Rows)
- Core Method 4: Memory Optimization (Type Downcasting)
- Core Method 5: Large File Writing
- Example 1: 100k-Row Table — Column Distribution + Chart
- Example 2: 1M-Row Table — Streaming Read + Filter + Export
- Vectorized Operations Cheat Sheet
- Memory Estimation
- Best Practices
What does the sn-da-large-file-analysis skill do?
万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet 转换加速、内存优化、分块处理和大文件写入模式。**遇到以下任一情况就主动使用本 skill**：①数据行数 ≥ 10k（由 sn-da-excel-workflow 的行数评估步骤触发）；②用户出现触发词：大文件 / 大数据量 / 性能优化 / 内存不足 / OOM / 百万行 / 十万行 / 流式读取 / Parquet / 分块处理 / large file / big data / streaming read / chunked processing；③直接使用 pd.read_excel() 导致超时或内存溢出；④用户明确要求对大规模数据集进行高性能处理。仅不用于：小于 10k 行的常规 Excel 分析（使用 sn-da-excel-workflow 即可）。
How do I install it?
Run `npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From OpenSenseNova/SenseNova-Skills, a repository with 4,855 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.