docker: compile core Ubuntu 24.04 dependencies from source

[ninsbl]

In the current Ubuntu 24.04 docker image we have both NumPy 2 (from pip) and NumPy 1.26 (from Ubuntu). This PR is an attempt for a cleaner solution. It:

• removes the usage of the UbuntuGIS-unstable repository in Ubuntu docker build
• gets all direct Python-dependencies from PyPi (not the distro)
• adds code to compile core GIS related dependencies (PROJ, GEOS, GADL, PDAL) from source of the latest release instead (we would have to activate renovate for keeping versions up to date (help/suggestions welcome))
• adds a build script for the dependencies that could be used in CI workflows as well

Main advantage is that we ship more recent Python packagages compared to the distro-based python3-x giving more flexibility to downstream usage of the docker image (in my case actinia). Esp. NumPy > 2.0 requires to build GDAL from source and PDAL is build from source in other images as well...

Also we get more control about version of important direct dependencies...

Still, this is just a suggestion for a possible improvement. An alternative I considered was to use just system packages (and no pip install at all), but I found the alternative here significantly more useful downstreams. A question is how much additional maintenance it may cause. As mentioned we compile PDAL and the GDAL-GRASS plugin other places as well...

[echoix]

I took a look, and the reasoning for using source builds and versions we decide make sense for Docker images.

I'm just not too sure the caching won't bust our caching allowance as is now, (because every change of the layers here might be too big for the cache, so our real layers we need cached will be thrashed-out (see: cache thrashing))

[echoix]

The last remaining thing here is that the third argument of the build script is unused. It is for the numthreads. Ninja will always use the most. But we might not always use this one (but for now it is hardcoded in the cmake -D options used), so kinda useless. I don't care keeping it or not, but I think how it is written now, the third argument is required.

Otherwise, I read a bit the linked GDAL dockerfile. Seems useful only to see the dependencies they use?

If it works for this CI run, we could merge as is.

[echoix]

Quick question: Should we be building unit tests if we are not executing them? For example, right now the build is waiting on GEOS to finish building its unit tests in order to continue building more layers in the Dockerfile.

[echoix]

A couple of suggestions, probably not complete

[echoix]

(I'll remove the commits changing the docker.yml script and do another PR, I have an opportunity to see new failures on @ninsbl's fork that I can't set up myself on my account as I already have tags and branches correctly defined)

[echoix]

Currently, the last commit works on a fork, let's see on a PR to this repo if it fails or not. If it doesn't, these last two commits could be made to another PR and removed here.

[echoix]

For now, the unit tests removed (we forgot to remove GEOS) saved 2.5 on two separate runs (probably different hardware too)

[ninsbl]

Nice! Thanks for your effort @echoix !

[echoix]

I removed the commits fixing the workflow in forks and created #6506 instead

[echoix]

@ninsbl Here you go

[wenzeslaus]

What is the purpose of the # ARG DIGEST="sha256:... lines? I would prefer these either not there if not needed, or with a comment explaining why the line is commented out. Commented-out lines are bad for maintenance when they lack explanation (which may be obvious in author's head, but not necessarily in everyone's head).

[echoix]

What is the purpose of the # ARG DIGEST="sha256:... lines? I would prefer these either not there if not needed, or with a comment explaining why the line is commented out. Commented-out lines are bad for maintenance when they lack explanation (which may be obvious in author's head, but not necessarily in everyone's head).

If they were there, they have a signification on what the base image is. It's kinda impossible now to keep updated, even in this PR. They wouldn't be updated by renovate, unless more work is made to create a custom regex for it.

[echoix]

Do you have an idea in mind to solve #6384 following what we just merged here?

Would using the same Dockerfile for the gui be the good move?

[ninsbl]

Do you have an idea in mind to solve #6384 following what we just merged here?

Would using the same Dockerfile for the gui be the good move?

Yes, I think that would be a good solution as it would reduce the number of dockerfiles to maintain... Also, the compile script could be used in Ubuntu 24.04 CI where we now use PDAL from UbuntuGIS...

[echoix]

Do you have an idea in mind to solve #6384 following what we just merged here?

Would using the same Dockerfile for the gui be the good move?

Yes, I think that would be a good solution as it would reduce the number of dockerfiles to maintain... Also, the compile script could be used in Ubuntu 24.04 CI where we now use PDAL from UbuntuGIS...

I filed issues for both these ideas