Harmonize encoding from h5netcdf with netcdf4 - #11067
Conversation
56c6396 to
71aa924
Compare
| "zlib", | ||
| "szip", | ||
| "bzip2", | ||
| "blosc", | ||
| "zstd", |
There was a problem hiding this comment.
todo: learn why this is a different bug?
bea7ba4 to
255d849
Compare
|
closing since being more aggressive seems to be ok |
c0d66e3 to
2006346
Compare
0526993 to
25b1b0d
Compare
kmuehlbauer
left a comment
kmuehlbauer
left a comment
There was a problem hiding this comment.
@hmaarrfk typo? It's h5netcdf 1.8.0
yes thanks, i kinda just played off memory. |
8d8694a to
482a647
Compare
| # tmp_file, | ||
| # engine="h5netcdf", | ||
| # encoding={"x": {"compression": "lzf", "zlib": True}}, | ||
| # ) |
There was a problem hiding this comment.
i guess i need to revive this error.
482a647 to
70833d0
Compare
| tmp_file, | ||
| engine="h5netcdf", | ||
| encoding={"x": {"compression": "lzf", "zlib": True}}, | ||
| ) |
There was a problem hiding this comment.
in this case, "compression": "lzf" wins now.
like test_compression_check_encoding_h5py
|
the CI should be fixed in #11366 |
70833d0 to
ff1d4e0
Compare
ff1d4e0 to
e4240a3
Compare
|
Just trying to see if there is still interest in this PR |
|
sorry to bump this again, i just think this is a small quality of life improvement when switching between backends. |
|
Thanks @kmuehlbauer. Sorry to keep bumping this, but I think this is an important one that hits long time users of xarray. I'm happy to help build and review a few of these maintenance PRs. let me know what else i can do to expedite this. I given the 6 month thing, i would appreciate it if the final reviewer just rebased / merge to resolve the conflicts when they feel ready to do so. I've been rebasing with AI alot and it just really helps. Let me know if you think that is unreasoable, so i can rebase after the final review. |
|
no concerns at all (sorry, I missed the ping a month ago). The min-deps CI says we can bump to @kmuehlbauer, since you know the |
<details><summary>Claude's draft</summary> Report compression and filter settings from the h5netcdf backend in the variable ``encoding`` consistently with the netCDF4 backend, using h5netcdf's ``Variable.filters()``. Data compressed with codecs such as ``zstd`` or ``blosc`` now keeps its compression when re-saved instead of being silently written uncompressed. - ``_extract_nc4_variable_encoding`` translates the boolean netCDF4-style compression flags (zlib/szip/bzip2/blosc/zstd) produced by ``filters()`` into a single h5py-style ``compression`` string. When both styles are supplied, the explicit ``compression`` takes precedence. - Bump the minimum supported h5netcdf to 1.8.0 (which introduced these compatibility features) and add h5netcdf to the policy ``ignored_violations`` so the intentionally-newer pin does not fail the Minimum Version Policy check. - Drop the now-unused ``requires_h5netcdf_1_7_0_or_above`` test helper. - Document the change in whats-new.rst. xref: pydata#10657 Co-authored-by: Claude <noreply@anthropic.com> Resume this Claude session: ``` claude --resume 9faa34e9-aaf5-4e94-a2b7-b03f468175f8 ``` </details>
1876e3a to
3a8d799
Compare
|
Cis refreshed and green. |
<details><summary>Claude's draft</summary> Report compression and filter settings from the h5netcdf backend in the variable ``encoding`` consistently with the netCDF4 backend, using h5netcdf's ``Variable.filters()``. Data compressed with codecs such as ``zstd`` or ``blosc`` now keeps its compression when re-saved instead of being silently written uncompressed. - ``_extract_nc4_variable_encoding`` translates the boolean netCDF4-style compression flags (zlib/szip/bzip2/blosc/zstd) produced by ``filters()`` into a single h5py-style ``compression`` string. When both styles are supplied, the explicit ``compression`` takes precedence. - Bump the minimum supported h5netcdf to 1.8.0 (which introduced these compatibility features) and add h5netcdf to the policy ``ignored_violations`` so the intentionally-newer pin does not fail the Minimum Version Policy check. - Drop the now-unused ``requires_h5netcdf_1_7_0_or_above`` test helper. - Document the change in whats-new.rst. xref: pydata#10657 Co-authored-by: Claude <noreply@anthropic.com> Resume this Claude session: ``` claude --resume 9faa34e9-aaf5-4e94-a2b7-b03f468175f8 ``` </details>
3 participants
We were trying to figure why certain compression features were not included in our h5netcdf backend.
This PR to h5netcdf should provide the filters variable, but I understand that it is too bleeding edge for you all to pin to h5netcdf/h5netcdf#307
xref: #10657
Ultimately, I don't think you can change the backend without ensuring it is "drop in replacement" at least in what concerns metadata.
Critically, if you compress certain data with zstd, and you try to resave it, it would be saved as 'raw' ruining any gains you had from the compression.
whats-new.rstapi.rstI used AI (Claude) to help me fix the linter problems.