Skip to content

towards new h5netcdf/netcdf4 features - #9509

Merged
kmuehlbauer merged 21 commits into
pydata:mainfrom
kmuehlbauer:h5netcdf-new-features
Oct 2, 2024
Merged

towards new h5netcdf/netcdf4 features#9509
kmuehlbauer merged 21 commits into
pydata:mainfrom
kmuehlbauer:h5netcdf-new-features

Conversation

@kmuehlbauer

@kmuehlbauer kmuehlbauer commented Sep 17, 2024

Copy link
Copy Markdown
Contributor

Comment thread doc/user-guide/io.rst
Comment thread doc/user-guide/io.rst
Invalid netCDF files
~~~~~~~~~~~~~~~~~~~~

The library ``h5netcdf`` allows writing some dtypes (booleans, complex, ...) that aren't

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It boils down, that we are approaching feature equality between netcdf4-python and h5netcdf. It seems, that only reference objects can't be handled by netcdf-c.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo: check boolean enums

Comment thread xarray/backends/h5netcdf_.py Outdated
Comment thread xarray/backends/netCDF4_.py Outdated
Comment thread xarray/coding/variables.py
Comment thread xarray/tests/test_backends.py Outdated
@kmuehlbauer
kmuehlbauer marked this pull request as ready for review September 18, 2024 08:21
@kmuehlbauer kmuehlbauer reopened this Sep 18, 2024
@kmuehlbauer kmuehlbauer added the run-upstream Run upstream CI label Sep 18, 2024
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

@ZedThree @bzah it would be great if you could have a look here. This brings nc-complex (netcdf4/h5netcdf) and enum handling (h5netcdf) to the backends. This is not released yet in h5netcdf, but in latest main-branch (checked with upstream CI run). If you have anything to add, please let me know.

@ZedThree ZedThree left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread doc/user-guide/io.rst
Comment thread xarray/backends/api.py
compute: bool = True,
multifile: Literal[False] = False,
invalid_netcdf: bool = False,
auto_complex: bool | None = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does auto_complex = None mean? Is it the same as auto_complex = False?
EDIT: Ah, to avoid passing the argument to h5netcdf? But invalid_netcdf is just a plain bool, so is that handled differently for netCDF4?

Should we also consider defaulting to True? I'm definitely interested in finding out if there are any cases where this results in unwanted behaviour

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, not sure, if I handled that correctly. It's more or less a security measure against feeding the kwarg to versions which can't handle those yet. I think this was the simplest solution without adding a whole bunch of boilerplate code. But maybe there is an easier solution to that.

Comment thread xarray/backends/netCDF4_.py
Comment thread xarray/backends/netCDF4_.py Outdated
Comment thread xarray/backends/netCDF4_.py
Comment thread xarray/coding/variables.py
Co-authored-by: Peter Hill <zed.three@gmail.com>
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

Thanks @ZedThree for taking the time to review. I've added couple of comments.

Comment thread xarray/backends/api.py
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

@pydata/xarray This seems to be ready now for review. This brings in adaptions for changes ongoing in h5netcdf/netcdf4-python wrt complex numbers and enums.

@kmuehlbauer kmuehlbauer changed the title MNT: towards new h5netcdf/netcdf4 features towards new h5netcdf/netcdf4 features Sep 23, 2024
@dcherian
dcherian requested a review from ZedThree September 24, 2024 05:29

@ZedThree ZedThree left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @kmuehlbauer!

I do think it would be clearer with having auto_complex just be bool (I don't think allowing it to be None gets us anything over setting it False by default), and with a check that when setting it we are using the netcdf4 engine, but that's a minor quibble.

@kmuehlbauer kmuehlbauer added the plan to merge Final call for comments label Sep 30, 2024
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

Upstream and upstream mypy failures are unrelated. I'm going to merge this the next day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

plan to merge Final call for comments run-upstream Run upstream CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writing complex numbers to netCDF Add writing complex data to docs

2 participants