zarr-python
Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.
npx skills add LeonChaoX/qinyan-academic-skills --skill zarr-python --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Converts the Zarr Python skill into a description of what the agent is instructed to do: manage large N-dimensional arrays stored in chunked formats, with optional compression, and cloud/local backends. It emphasizes creating, opening, reading, writing, and resizing arrays, configuring chunking strategies, and using various storage backends (local, in-memory, ZIP, S3, GCS). It also covers integration with NumPy, Dask, and Xarray, plus metadata handling and parallel synchronization.
How it works
The skill guides the agent to:
- Install and import the zarr package and related IO modules for cloud storage (e.g., s3fs, gcsfs).
- Create arrays with zarr.create_array, zarr.zeros/ones/full, or zarr.array, specifying shape, chunks, and dtype, and optionally store paths.
- Open arrays with zarr.open_array or zarr.open, in modes 'r', 'r+', or 'w', and handle reading and writing via standard NumPy-like indexing.
- Use chunking strategies and topics like minimum chunk size (~1 MB), align chunks with access patterns, and optional sharding via shards to group chunks.
- Configure per-chunk compression using codecs like BloscCodec, GzipCodec, BytesCodec, including specific parameters (cname, clevel, shuffle).
- Select storage backends: LocalStore, MemoryStore, ZipStore, and cloud stores via S3Map, GCSMap; manage credentials, and consolidate metadata with zarr.consolidate_metadata and zarr.open_consolidated.
- Work with groups and hierarchies, including H5py-like API for groups and datasets.
- Attach and read JSON-serializable attributes on arrays and groups.
- Integrate with NumPy, Dask, and Xarray for analysis, lazy computations, and backends.
- Use synchronization primitives for thread/process safety when writing to chunks, and understand when synchronization is required.
- Profile and debug with z.info, and inspect storage sizes and compression statistics.
When to use it
Use this when you need:
- Large N-dimensional arrays that exceed memory and require parallel I/O.
- Cloud-native workflows with S3 or GCS backends.
- Integration with NumPy, Dask, and Xarray for analysis.
- Fine-grained control over chunking, compression, and metadata consolidation to optimize performance.
What it can touch
- Local storage via LocalStore and file paths.
- In-memory storage via MemoryStore.
- ZIP-based storage via ZipStore.
- Cloud storage via s3fs (S3Map) and gcsfs (GCSMap).
- It references codecs and tools like BloscCodec, GzipCodec, ZstdCodec, and synchronization primitives (ThreadSynchronizer, ProcessSynchronizer).
Caveats
- Metadata consolidation reduces I/O but can become stale if updates aren’t re-consolidated.
- Not suitable for frequently-updated datasets without re-consolidation.
- Multi-writer scenarios may have inconsistent reads; synchronization is only needed for writes that span chunk boundaries.
- Requires Python 3.11+ and optional cloud packages for cloud storage support.
# Zarr Python ## Overview Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray. ## Quick Start ### Installation ```bash uv pip install zarr ``` Requires Python 3.11+. For cloud storage support, install additional packages: ```python uv pip install s3fs # For S3 uv pip install gcsfs # For Google Cloud Storage ``` ### Basic Array Creation ```python import zarr import numpy as np # Create a 2D array with chunking and compression z = zarr.create_array( store="data/my_array.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="f4" ) # Write data using NumPy-style indexing z[:, :] = np.random.random((10000, 10000)) # Read data data = z[0:100, 0:100] # Returns NumPy array ``` ## Core Operations ### Creating Arrays Zarr provides multiple convenience functions for array creation: ```python # Create empty array z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4', store='data.zarr') # Create filled arrays z = zarr.ones((5000, 5000), chunks=(500, 500)) z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100)
- Overview
- Quick Start
- Installation
- Basic Array Creation
- Core Operations
- Creating Arrays
- Opening Existing Arrays
- Reading and Writing Data
- Resizing and Appending
- Chunking Strategies
- Chunk Size Guidelines
- Aligning Chunks with Access Patterns
- Sharding for Large-Scale Storage
- Compression
uv pip install zarr
What does the zarr-python skill do?
Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.
How do I install it?
Run `npx skills add LeonChaoX/qinyan-academic-skills --skill zarr-python --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From LeonChaoX/qinyan-academic-skills, a repository with 759 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.