Assorted bootstrap LLVM refactors (part 4/N)

[Kobzol]

Continuation on #161247.

This PR completely removes handling of git changes or LLVM downloads from config parsing, and moves it into the LlvmFromCi step. Thanks to that, we now also allow downloading LLVM for non-host targets.

There is one annoyance related to that, and that is that download-ci-llvm now applies to all targets for which you try to build LLVM (d'uh), but that also means that if (for whatever reason) LLVM fails to be downloaded from CI, the build will fail. So if you build for target T2 from target T1:

• If you want to download T1, but build T2, that's not possible to express.
• If T2 fails to be downloaded, the build fails, even if it could be built locally.

I think that we mostly have four options how to deal with this:

1. Just ignore it and wait to see if someone complains.
2. Revert the change and always download only for the host target. Worked so far.
3. Allow specifying download-ci-llvm per target in the target config section. So that you can say that you want to download for T1, but build for T2.
4. Make download failures non-fatal, and cause them to trigger a local build. This would also help with removing the hacky is_ci_llvm_available_for_target logic, which hard-codes a bunch of targets to "know" which ones offer LLVM and which don't. We could just try to download, and if the result is 404, then we print a warning and continue with building.

I think that 4. would be the best. If we get a 404, there's no way we can download, so we build instead. If we get a different error, we still make the failed download fail the build.

Already before this PR, we did this:

// If download-ci-llvm=true we also want to check that CI llvm is available
b && llvm::is_ci_llvm_available_for_target(&dwn_ctx.host_target, asserts)

so if LLVM wasn't available, we just silently reverted from download-ci-llvm=true to download-ci-llvm=false. The 4. proposal would just generalize that, to actually check whether the LLVM files are present on the CDN or not.

After I opened this PR, the CI failure showed something quite tricky: how to handle LLVM downloads during check builds. Bootstrap has an optimization, where if we are doing x check:

• We want to avoid building LLVM, and thus do not pass some LLVM-related build flags
• Unless LLVM is already built, in which case we do want to pass the build flags, to ensure better consistency (and thus less invalidations) between check/build commands.

The question is what we should do about downloaded LLVM in this check. Should an LLVM that can be downloaded, but hasn't been downloaded yet, be considered "already built"? This comes up in commands like x check --target T1, T2, T2 core, where previously we would just consider LLVM to be in "has to be built" mode, and so the build flags were not passed. But not, when LLVM can be actually downloaded, the situation is different.

What the (currently two) places who have the check optimization want is something like:

1. If LLVM is already built, pass the flags
2. If LLVM is already downloaded, pass the flags
3. If there is an external llvm-config, pass the flags

We could implement this more explicitly, however, there is one issue with checking 1., because for that we have to checkout the src/llvm-project submodule, to have a fully robust check if the local LLVM has been built or not (unless we want to just check that its local output directory exists or something). And of course checking out that submodule if you only do x check is something that we also want to avoid...

Based on #161247 (last three commits are new).

r? jieyouxu

[rust-log-analyzer]

The job pr-check-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#15 2.879 Building wheels for collected packages: reuse
#15 2.880   Building wheel for reuse (pyproject.toml): started
#15 3.095   Building wheel for reuse (pyproject.toml): finished with status 'done'
#15 3.096   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132857 sha256=6a78a69541e3df9c7c106367722bab90241125ee6b7cdb81c5e40637268f85a9
#15 3.096   Stored in directory: /tmp/pip-ephem-wheel-cache-ojjz232s/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#15 3.098 Successfully built reuse
#15 3.099 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#15 3.488 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#15 3.488 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#15 DONE 3.6s