Reduce the time taken to import `cfdm`

[davidhassell]

Importing cfdm is quite slow ... on my local computer it takes 1.4 seconds.

Another system I have tried (JASMIN) can take between 6 and 10 seconds!

Having had a look at this, there seem to be two main culprits for the slow import:

• Doc string rewriting
  • At import time, every docstring (of which there are currently 5569) is inspected for doc string substitutions, replacing any that are found
• Importing external modules that themselves have a slow import
  • The main problems here are dask, scipy, s3fs, zarr, h5netcdf, uritools, netCDF4

These can be improved on:

• Doc string rewriting
  • Only apply substitutions when necessary, rather than trying every possible substitution for every doc string. Only 3009 of the doc strings need rewriting, and each one of those only utilises a small number of the 110 possible substitutions.
• Importing external modules that themselves have a slow import
  • Move the dask, scipy, s3fs, zarr, h5netcdf, uritools to run time, rather than import time. Many will not ever get imported, and when they do, the time is usually negligible compared to the operation being run.

Results

By applying these two changes, my local import time reduces to 0.2 seconds (from 1.4 seconds - a factor of 7 speed-up). On the other system I tried, the time reduces to between 1.5 and 2.5 seconds (from between 6 and 10 seconds).

These are good enough improvements for a PR, I think ...

[sadielbartholomew]

@davidhassell good thinking. I have noticed cf-python is also very slow to import - is that due to the slowness of cfdm, do you think, or will you address cf-python separately in this respect?

[davidhassell]

I have noticed cf-python is also very slow to import - is that due to the slowness of cfdm, do you think, or will you address cf-python separately in this respect?

Mainly cfdm, yes! I have a cf-python PR coming, which reduces its import down to 0.25 seconds.

[davidhassell]

The rule of thumb I've applied is to only import built-in packages at cfdm import time, unless they're _really_required to make the cfdm import work.

There are already exceptions to this - numpy and cfunits - these are pretty cheap to import, and are used so much that repeated imports (millions) in methods/functions could potentially add up to a noticeable runtime delay.

[davidhassell]

A useful way to analyse import times is

$ python -X importtime -c "import cfdm"

The last line of the output give the total import time in microseconds:

import time:      1050 |     217006 | cfdm