bootstrap: Ensure documentation is built before tarball is created

[alistair23]

run_default_doc_steps() is called to ensure the documentation is built by Docs::run() and it should build the documentation if it isn't already built.

When running the install src command I'm seeing failures as the builder.doc_out(host) directory does not exist. This is because match_paths_to_steps_and_run() doesn't actually build any documentation as the paths.is_empty() causes an early return. This results in install failures as the */doc src directory doesn't exist.

This patch passes the paths to run_step_descriptions() when building documentation to ensure it is correctly built.

This fixes installing the Rust source code in OpenEmbedded.

[rustbot]

r? @Zalathar

rustbot has assigned @Zalathar.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Zalathar]

r? bootstrap

[jieyouxu]

@alistair23 can you say more on:

• What's the cli invocation you used, and
• What configuration you had?

I'm trying to see if it's possible to snapshot the doc steps that ./x install src triggers on default.

[alistair23]

@alistair23 can you say more on:

* What's the cli invocation you used, and

* What configuration you had?

I'm trying to see if it's possible to snapshot the doc steps that ./x install src triggers on default.

It's part of the OpenEmbedded build system, so it isn't super easy to split out a simple reproducible test case.

It's calling

./x install
./x install src

This is my config.toml

# EXTRA_OECARGO_PATHS
paths = [

]

# Local mirror vendored by bitbake
[source.bitbake]
directory = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/cargo-native/1.92.0/sources/rustc-1.92.0-src/vendor"

[source.crates-io]
replace-with = "bitbake"
local-registry = "/nonexistent"

[http]
# Multiplexing can't be enabled because http2 can't be enabled
# in curl-native without dependency loops
multiplexing = false

# Ignore the hard coded and incorrect path to certificates
cainfo = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/cargo-native/1.92.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt"


# HOST_SYS
[target.x86_64-unknown-linux-gnu]
linker = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/cargo-native/1.92.0/wrapper/target-rust-ccld"

[build]
# Use out of tree build destination to avoid polluting the source tree
target-dir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/cargo-native/1.92.0/build/target"

[term]
progress.when = 'always'
progress.width = 80

[jieyouxu]

(I meant the bootstrap config, be it bootstrap.toml or env var / bootstrap flags, not the cargo config, should've been more clear)

[alistair23]

(I meant the bootstrap config, be it bootstrap.toml or env var / bootstrap flags, not the cargo config, should've been more clear)

Whoops, wrong file.

If I copy these out from OE I can bootsrap Rust independently

config.toml

change-id = 116881

[target.x86_64-unknown-linux-gnu]
llvm-config = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr/bin/llvm-config"
cxx = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/wrapper/target-rust-cxx"
cc = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/wrapper/target-rust-cc"
linker = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/wrapper/target-rust-ccld"

[llvm]
static-libstdcpp = false
download-ci-llvm = false

[rust]
rpath = true
remap-debuginfo = true
download-rustc = false
llvm-tools = false
lld = false
use-lld = false
channel = "stable"
optimize = true
verbose-tests = true

[build]
submodules = false
docs = false
rustc = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/rust-snapshot/bin/rustc"
cargo = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/rust-snapshot/bin/cargo"
extended = false
vendor = true
target = ["x86_64-unknown-linux-gnu"]
host = ["x86_64-unknown-linux-gnu"]
build = "x86_64-unknown-linux-gnu"

[install]
prefix = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr"
bindir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr/bin"
libdir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr/lib"
datadir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr/share"
mandir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/usr/share/man"
sysconfdir = "/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/image/var/mnt/scratch/alistair/yocto/oe-master/build/tmp/work/x86_64-linux/rust-native/1.92.0/recipe-sysroot-native/etc"

.cargo/config.toml

[source.crates-io]
replace-with = "vendored-sources"

[source."git+https://github.com/rust-lang/team"]
git = "https://github.com/rust-lang/team"
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"

But that doesn't reproduce the issue. I'm not clear on what the difference is between the OE build environment and a standalone one which triggers the issue

[jieyouxu]

That feels a bit surprising, is OE using the same build == host == target setup?

[jyn514]

We saw a failure like this in Ferrocene and the root cause was that we had run x dist —-host=, which disables all Steps that are marked HOST_ONLY = false. Maybe OE did the same?

[jieyouxu]

Oh, interesting, that isn't a possibility I considered 👀