pyfixest-grid-sharding
Diagnose and fix slow pyfixest regression GRIDS (many feols/fepois calls run sequentially) that stay slow despite demeaner_backend="cupy64" and an idle GPU. Use when: (1) a script looping dozens of pf.feols models on a 100k+ row panel takes ~1 min/model, (2) process inspection shows ~1-1.5 cores busy and nvidia-smi shows ~0% GPU utilization with a resident cupy context, (3) planning any worker prompt that will run a model grid (robustness variants x FE structures x domains). Root cause: per-model CPU-side single-threaded fixed costs (formulaic model-matrix build, interaction construction, sing
npx skills add kennethkhoocy/applied-micro-skills --skill pyfixest-grid-sharding --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# pyfixest Grid Sharding ## Problem A regression grid (e.g. 2 measures x 3 FE structures x pooled+per-domain x 3 label variants ~ 70 models) on a 327k-row panel with high-cardinality director FE ran ~55 s/model sequentially — ~65 min wall — on an RTX 5080 machine with `demeaner_backend="cupy64"` on every call. The GPU was NOT the bottleneck. ## Context / Trigger Conditions - Measured signature (verified 2026-07-21, H5 seat-loss rerun): job process at ~1.4 cores CPU (37.7 CPU-min in 27 wall-min), `nvidia-smi` 0% utilization with ~4 GB resident (cupy context loaded, idle), one pyfixest singleton warning per completed model ticking by in the log. - Any orchestration prompt that asks a worker to "rerun every headline cell under variants A/B/C" without specifying execution structure. ## Solution 1. Diagnose before blaming the GPU: check process CPU-minutes vs wall-clock (~1 core => serial CPU-bound) and GPU utilization (near 0% => demeaning is not the constraint). The cupy64 kwarg is still correct; it just cannot fix a CPU-dominated pipeline. 2. Shard the GRID, not the data: split the model list across N OS processes (`--shard i --nshards N` over the model index, one output part-file ea
- Problem
- Context / Trigger Conditions
- Solution
- Verification
- Measured GPU-saturation verdict (2026-07-21 escalation experiment)
- Notes
What does the pyfixest-grid-sharding skill do?
Diagnose and fix slow pyfixest regression GRIDS (many feols/fepois calls run sequentially) that stay slow despite demeaner_backend="cupy64" and an idle GPU. Use when: (1) a script looping dozens of pf.feols models on a 100k+ row panel takes ~1 min/model, (2) process inspection shows ~1-1.5 cores busy and nvidia-smi shows ~0% GPU utilization with a resident cupy context, (3) planning any worker prompt that will run a model grid (robustness variants x FE structures x domains). Root cause: per-model CPU-side single-threaded fixed costs (formulaic model-matrix build, interaction construction, sing
How do I install it?
Run `npx skills add kennethkhoocy/applied-micro-skills --skill pyfixest-grid-sharding --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From kennethkhoocy/applied-micro-skills, a repository with 54 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.