Repository navigation
Support for "cargo vendor" broke -Zbuild-std for custom targets. Or at least, the current behavior is now different #8945
Description
(Disclaimer: Rust noob here. Don't kill me if I've misinterpreted a correct behaviour.)
Problem
Building a crate with -Zbuild-std=... and --target <target> where <target> is a custom target linked in Rustup like this (rustup link mytarget /...path.../) is broken since approx the time when the changes for "cargo vendor" landed.
Steps
- Build a custom rust toolchain. For example, this one.
- Register the just-built toolchain as a custom target in rustup as described in the above repo.
- Try to build a crate (for example, this one) by using
cargo build -Z build-std=std,panic_abort -Z build-std-features=panic_immediate_abort - Problem 1: Building the crate will fail with complaint
error: could not read vendor path. Apparently, because of this code on line 55. - Problem 2: If you make an empty "vendor" dir and re-run the build, it will fail again now with the dreaded:
error[E0463]: can't find crate for `std`
|
= note: the `xtensa-esp32-none-elf` target may not be installed
... which is due to the fact that cargo does not pick up the std lib from the sysroot of the custom target at all. An indication of that is the fact that it downloads and compiles the rustc-std-workspace-* crates from GitHub rather than picking up their patches from the sysroot of the custom target. I.e. when run with --verbose:
rustc --crate-name rustc_std_workspace_alloc --edition=2018 /Users/...path.../.cargo/registry/src/github.com-1ecc6299db9ec823/rustc-std-workspace-alloc-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=c8dee4e2d9f82246 -C extra-filename=-c8dee4e2d9f82246 --out-dir /Users/<censored>/rust-esp32-std-hello/rust/target/xtensa-esp32-none-elf/release/deps --target xtensa-esp32-none-elf -Z force-unstable-if-unmarked -L dependency=/Users/...path.../rust-esp32-std-hello/rust/target/xtensa-esp32-none-elf/release/deps -L dependency=/Users/...path.../rust-esp32-std-hello/rust/target/release/deps --cap-lints allow (exit code: 1)
Apparently, the above is caused by the logic here (line 67 and onward), but I've not analyzed (yet) what needs to be fixed in detail.
Possible Solution(s)
The following workaround seems to fix the issue:
cd rustmkdir vendorcd vendorln -s ../library/rustc-std-workspace-alloc rustc-std-workspace-allocln -s ../library/rustc-std-workspace-core rustc-std-workspace-coreln -s ../library/rustc-std-workspace-std rustc-std-workspace-std
Yet, this just does not feel right.
I just want to use my custom target with -Zbuild-std and that's it. Ideally, I should not be even aware of the vendor machinery, let alone symlinking the rustc-std-workspace-* patch crates in there.
Or is this how it all is supposed to work with custom targets, now that cargo vendor has landed?
Activity
alexcrichton commentedon Dec 7, 2020
The changes to
cargo vendoris that we expect the Rust libstd sysroot crates to be vendored in the source directory. This probably means that we either need specific code for "you're working with a local toolchain" or the Rust build system needs a new build target for "make sure to build a source tree that's compatible withcargo -Zbuild-std"ivmarkov commentedon Dec 8, 2020
What does that mean? When working with custom targets, cargo is already very well aware of the following two locations:
<rust-github-copy>/library<rust-github-copy>/vendor(this one actually does not really exist with custom targets, but cargo tries to access it anyway)It is aware of those because when the custom target is build (e.g.
./x.py build --stage2), there is a symlink autocreated as follows:<rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/src/rust-> symlinks back to<rustc-github-copy>/.... where the custom target root is
<rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2(as I'm not installing the custom target after building it but just linking it into Rustup by doingrustup link xtensa <rust-github-copy>/build/x86_64-unknown-linux-gnu/stage2)So to summarize Cargo does have access to the
/libraryand/vendordirs even with custom targets. It also knows that the 'std' and 'core' code is in/libraryand not in/vendor. However - since the changes that introduced cargo vendor - it somehow does expect therustc-std-workspace-*hacks specifically to be in/vendorwhile they are are actually inside/library. (Hence why I have to symlink them from inside/vendorin my workaround.)Why are the
rustc-std-workspace-*crates now expected to be in/vendor, and only these are expected to be in/vendorfrom all the std/crate/etc. machinery while the rest is still (correctly) looked up from/library?Sounds like a bug to me?
alexcrichton commentedon Dec 9, 2020
Yes I believe your diagnosis of the situation is all correct, and this is due to changes in #8834. Cargo doesn't expect the rustc-std-workspace crates to be in vendor themselves, but it's expecting the copies from crates.io is vendored in there, since that's what
cargo vendorgenerates.It's not clear if this should be a bug for Cargo to fix or a bug for rust-lang/rust and/or rustup to have an extra step when you create a local toolchain.
ehuss commentedon Dec 18, 2020
I'm going to close this as it has been reverted in #8968. Will continue tracking in rust-lang/wg-cargo-std-aware#23.