matlab-choose-bigdata-solution
Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building cust
npx skills add matlab/matlab-agentic-toolkit --skill matlab-choose-bigdata-solution --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Choose Big Data Solution Help users and agents select the right MATLAB tool for large **tabular data in file-based formats** (CSV, Parquet, delimited text, spreadsheets, MDF). The skill encodes a decision flowchart — recommend one clear path, not a menu of options. ## When to Use - User or agent has large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) - User or agent says data is "large", "big", or "huge" (even without a specific size) - User or agent hits an out-of-memory error with `readtable`, `parquetread`, or similar - User or agent has multiple tabular files to process as a single dataset - User or agent has multiple tabular files to process independently (per-file) - User or agent asks how to scale up an existing workflow on tabular file data - User or agent asks about datastores, tall arrays, or mapreduce - User or agent needs to build a custom datastore class for a proprietary or non-standard format - User or agent has large XML, JSON, HTML, or Word document files — `readtable` supports these formats but the built-in datastores do not. Scaling these requires a custom datastore (see "Formats Without a Built-in Datastore" and "Custom D
- When to Use
- When NOT to Use
- Pre-Flight Check
- Decision Flowchart
- Datastore + Tall Arrays (continuous dataset)
- Datastore + Transform (per-file)
- Datastore + Transform (per-row-group, Parquet)
- Parallelizing readall (transform workflows)
- Parallel Pool (local parallelism for tall arrays)
- Mapreducer (Hadoop/Spark only)
- Key Functions
- Formats Without a Built-in Datastore
- Custom Datastores
What does the matlab-choose-bigdata-solution skill do?
Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building cust
How do I install it?
Run `npx skills add matlab/matlab-agentic-toolkit --skill matlab-choose-bigdata-solution --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From matlab/matlab-agentic-toolkit, a repository with 868 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.