zarr-developers/zarr-python - CodSpeed

zarr-developers

zarr-python

Blog Docs Changelog

Performance History

Latest Results

perf: vectorize shard_dict build in _encode_partial_sync (fix write regression) CodSpeed flagged test_sharded_morton_write_single_chunk regressing ~38-39% (writing one 1x1x1 chunk into a 32^3 = 32768-chunk shard). Both main and this branch do a full shard rewrite for a partial write, so the rewrite itself is not the regression — and it is NOT the removed byte-range fast path (that path was gated out here anyway: write_empty_chunks defaults to False -> skip_empty=True). The cause: the sync _encode_partial_sync rebuilt the in-memory shard_dict with a per-coordinate __getitem__ loop over all 32768 chunks (O(n_chunks) Python overhead + try/except per chunk), whereas main's async _encode_partial_single builds the same dict with a single vectorized index lookup via _ShardReader.to_dict_vectorized. Switched the sync path to to_dict_vectorized (a plain, non-async method; _shard_reader_from_bytes_sync already returns a _ShardReader), matching the async path. The dict's key order is immaterial (the physical layout is decided downstream by the subchunk_write_order loop in _encode_shard_dict_sync), so the merge loop — which looks up by coordinate, not order — is unaffected. Local micro-benchmark (32^3 shard, single 1x1x1 chunk write): 59.4 -> 40.0 ms/write (~1.5x), matching the CodSpeed delta. Correctness: full sharding + pipeline-parity suites pass (581), so Fused still matches Batched byte-for-byte. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

d-v-b:perf/prepared-write-v2

1 day ago

Merge branch 'main' into perf/prepared-write-v2

d-v-b:perf/prepared-write-v2

1 day ago

ig/pread

3 days ago

perf: build order-coord tuples via .tolist(); document dual representation `morton_order_coords` / `lexicographic_order_coords` built their tuple-of- tuples with a row-by-row `tuple(int(x) for x in row)` comprehension. Using `map(tuple, arr.tolist())` instead does the int conversion in a single C-level call, producing byte-identical native-int tuples ~8-9x faster (~16ms -> ~1.9ms cold build at 32^3). It is a per-shape cached build, so this only speeds the first write to each shard shape, but it is free. Also document in `to_dict_vectorized` why the chunk coordinates are needed in two forms — a numpy array for the vectorized index lookup and hashable tuples for the dict keys — since numpy rows are unhashable and a tuple list can't be used for the vectorized modulo/advanced-indexing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

d-v-b:perf-sharding-coord-cache

5 days ago

refactor(indexing): make morton_order_iter the lazy primitive too Per @mkitti: the morton pair was backwards in the same way the lexicographic pair was. Invert it to match — `morton_order_iter` is now the lazy generator primitive and `_morton_order_keys` collects it into a cached tuple, mirroring `lexicographic_order_iter` / `_lexicographic_order_keys`. No behavioral change for the in-tree consumers (all fully consume the sequence) and the Z-order is identical; this keeps the two coordinate- order families symmetric and gives morton the same lazy/early-exit option lexicographic now has. Co-authored-by: Mark Kittisopikul <mkitti@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

d-v-b:perf-sharding-coord-cache

5 days ago

chore(deps): bump the actions group across 1 directory with 8 updates (#4014) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](https://github.com/prefix-dev/setup-pixi/compare/1b2de7f3351f171c8b4dfeb558c639cb58ed4ec0...5185adfbffb4bd703da3010310260805d89ebb11) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/57e3a136b779b570ffcdbf80b3bdc90e7fab3de2...e79a6962e0d4c0c17b229090214935d2e33f8354) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](https://github.com/github/issue-metrics/compare/c9e9838147fd355dace335ba787f01b6641a400a...1e38d5e62363e14db8019ed7d106b9855bdba6cc) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](https://github.com/j178/prek-action/compare/6ad80277337ad479fe43bd70701c3f7f8aa74db3...bdca6f102f98e2b4c7029491a53dfd366469e33d) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v7...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v7...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.13.0...cef221092ed1bacb1cc03d23a2d87d1d172e277b) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/b1d7e1fb5de872772f31590499237e7cce841e8e...5f14fd08f7cf1cb1609c1e344975f152c7ee938d) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

main

7 days ago

perf: cache lexicographic chunk coords in sharding codec The subchunk_write_order feature (#3826) regressed sharded write performance: _encode_partial_single rebuilt the full per-shard chunk coordinate grid on every write via `np.array(list(_subchunk_order_iter(..., "lexicographic")))`, and `to_dict_vectorized` rebuilt a tuple key per row with `tuple(coords.ravel())`. For a single-chunk write into a shard with tens of thousands of chunks this roughly doubled write time (~22ms -> ~40ms on test_sharded_morton_write_single_chunk, matching the -44% CodSpeed regression). Add cached `_lexicographic_order` (array) and `_lexicographic_order_keys` (tuples) helpers in indexing.py, mirroring `_morton_order`/`_morton_order_keys`, and pass the cached keys into `to_dict_vectorized` instead of deriving them row-by-row. This restores write throughput to the pre-#3826 baseline while preserving identical chunk ordering (verified equal to np.ndindex across shapes including 0-d and empty). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

d-v-b:perf-sharding-coord-cache

12 days ago

fix: handle NaN fill_value in array metadata equality (#3999) Frozen dataclass __eq__ compared fill_value directly, so two identical metadata objects with a NaN fill value compared unequal (NaN != NaN under IEEE 754). Compare the JSON-serialized form instead, which treats matching NaN and infinite fill values as equal. Fixes #2929 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

main

14 days ago

Latest Branches

×3.1

perf: phased codecpipeline#3885

1 day ago

298295f

d-v-b:perf/prepared-write-v2

0%

perf: try `pread` like `zarrs`'s use of `File::read_exact_at`#4039

3 days ago

72cb294

ig/pread

+40%

perf: cache lexicographic chunk coords in sharding codec#4012

5 days ago

2742897

d-v-b:perf-sharding-coord-cache

© 2026 CodSpeed Technology

Home Terms Privacy Docs