deadlock detected when using -Zthreads=8

[MartinquaXD]

I just tried out the new parallelization feature of the compiler and it randomly panicked because a deadlock was detected.
I tried a couple of times and could not reproduce it unfortunately.

Meta

rustc --version --verbose:

cargo 1.75.0-nightly (7046d992f 2023-11-08)

Error output

thread '<unnamed>' panicked at compiler/rustc_query_system/src/query/job.rs:543:9:
deadlock detected

<Backtrace>

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked -Z threads=8

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
deadlock handler panicked, aborting process
error: could not compile `bigdecimal` (lib)

Caused by:
  process didn't exit successfully: `/Users/martin/.rustup/toolchains/nightly-aarch64-apple-darwin/bin/rustc --crate-name bigdecimal /Users/martin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bigdecimal-0.3.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=128 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --cfg 'feature="serde"' -C metadata=36f9c58248b94912 -C extra-filename=-36f9c58248b94912 --out-dir /Users/martin/work/backend/side/target/debug/deps -L dependency=/Users/martin/work/backend/side/target/debug/deps --extern num_bigint=/Users/martin/work/backend/side/target/debug/deps/libnum_bigint-3d0a0c3120349810.rmeta --extern num_integer=/Users/martin/work/backend/side/target/debug/deps/libnum_integer-ed949aa408650e16.rmeta --extern num_traits=/Users/martin/work/backend/side/target/debug/deps/libnum_traits-a22352494166d956.rmeta --extern serde=/Users/martin/work/backend/side/target/debug/deps/libserde-2f69e42262cb5ef0.rmeta --cap-lints allow -Z threads=8` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...

Backtrace

stack backtrace:
   0:        0x100ee12b8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h052d4a6449303d0c
   1:        0x100f23268 - core::fmt::write::hd7376fa11f842969
   2:        0x100ed79ec - std::io::Write::write_fmt::h8962cd07816e935b
   3:        0x100ee10f4 - std::sys_common::backtrace::print::h97fb0a8a4077c481
   4:        0x100ee3af4 - std::panicking::default_hook::{{closure}}::h3b4672ac61ca5355
   5:        0x100ee383c - std::panicking::default_hook::h609e6a125a90b4e7
   6:        0x10a2eabb8 - <alloc[e142c9bf9c63cfc4]::boxed::Box<rustc_driver_impl[1a98898d3f3a38ed]::install_ice_hook::{closure#0}> as core[4e0a19b8f4e7b450]::ops::function::Fn<(&dyn for<'a, 'b> core[4e0a19b8f4e7b450]::ops::function::Fn<(&'a core[4e0a19b8f4e7b450]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[4e0a19b8f4e7b450]::marker::Send + core[4e0a19b8f4e7b450]::marker::Sync, &core[4e0a19b8f4e7b450]::panic::panic_info::PanicInfo)>>::call
   7:        0x100ee416c - std::panicking::rust_panic_with_hook::h9625542414b348eb
   8:        0x100ee3f10 - std::panicking::begin_panic_handler::{{closure}}::hb4fa343c314105dc
   9:        0x100ee1720 - std::sys_common::backtrace::__rust_end_short_backtrace::he263cec7b035e782
  10:        0x100ee3cd0 - _rust_begin_unwind
  11:        0x100f3d0b8 - core::panicking::panic_fmt::h624e311b541e9a0d
  12:        0x10b6fac6c - rustc_query_system[2c5cf56d08163fd7]::query::job::deadlock
  13:        0x10a2fba58 - std[f37625d93cba7df1]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[6fd153b9b61224af]::util::run_in_thread_pool_with_globals<rustc_interface[6fd153b9b61224af]::interface::run_compiler<(), rustc_driver_impl[1a98898d3f3a38ed]::run_compiler::{closure#0}>::{closure#0}, ()>::{closure#2}::{closure#1}, ()>
  14:        0x10a2cccfc - <<std[f37625d93cba7df1]::thread::Builder>::spawn_unchecked_<rustc_interface[6fd153b9b61224af]::util::run_in_thread_pool_with_globals<rustc_interface[6fd153b9b61224af]::interface::run_compiler<core[4e0a19b8f4e7b450]::result::Result<(), rustc_span[7d3d88934d9d268]::ErrorGuaranteed>, rustc_driver_impl[1a98898d3f3a38ed]::run_compiler::{closure#1}>::{closure#0}, core[4e0a19b8f4e7b450]::result::Result<(), rustc_span[7d3d88934d9d268]::ErrorGuaranteed>>::{closure#2}::{closure#1}, ()>::{closure#1} as core[4e0a19b8f4e7b450]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  15:        0x100eec368 - std::sys::unix::thread::Thread::new::thread_start::h6af77cae4d08b8df
  16:        0x18f69ffa8 - __pthread_joiner_wake

rustc-ice-2023-11-10T12_41_14-4307.txt

[SparrowLii]

Could you provide the minimal example causes the ICE?

[MartinquaXD]

The project I compiled was huge and I was not able to consistently reproduce it so reducing my case to a minimal example might take forever.
Buy I noticed the issue you linked already contains a fairly small example code so it's probably alright to close my ticket since it seems to be a duplicate of that other ticket anyway.

[SparrowLii]

Can you reproduce this bug using the latest nightly version to get the new error output?

[MartinquaXD]

Yes. It appears to be the same issue (albeit while compiling a different dependency but that probably doesn't matter):

thread '<unnamed>' panicked at compiler/rustc_query_system/src/query/job.rs:548:13:
deadlock detected! current query map:
{
    QueryJobId(
        271116,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "collect_and_partition_mono_items",
            span: Some(
                Span {
                    lo: BytePos(
                        0,
                    ),
                    hi: BytePos(
                        0,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: None,
            def_kind: None,
            ty_adt_id: None,
            dep_kind: DepKind {
                variant: 250,
            },
            hash: 15975767348252564715,
        },
        job: QueryJob {
            id: QueryJobId(
                271116,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    270390,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        270390,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "collecting exported symbols for crate `0`",
            span: Some(
                Span {
                    lo: BytePos(
                        0,
                    ),
                    hi: BytePos(
                        0,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: None,
            def_kind: None,
            ty_adt_id: None,
            dep_kind: DepKind {
                variant: 249,
            },
            hash: 2769749618247426339,
        },
        job: QueryJob {
            id: QueryJobId(
                270390,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: None,
            latch: Some(
                QueryLatch {
                    info: Mutex {
                        data: QueryLatchInfo {
                            complete: false,
                            waiters: [
                                QueryWaiter {
                                    query: None,
                                    condvar: Condvar { .. },
                                    span: Span {
                                        lo: BytePos(
                                            0,
                                        ),
                                        hi: BytePos(
                                            0,
                                        ),
                                        ctxt: #0,
                                    },
                                    cycle: Mutex {
                                        data: None,
                                    },
                                },
                            ],
                        },
                    },
                },
            ),
        },
    },
}
stack backtrace:
   0:        0x102dbcfa0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb8d18225a1661929
   1:        0x102dfeeb8 - core::fmt::write::h66d9d95386951700
   2:        0x102db3764 - std::io::Write::write_fmt::h9d6553bedbe91688
   3:        0x102dbcddc - std::sys_common::backtrace::print::h39028e36ec7b8685
   4:        0x102dbf7dc - std::panicking::default_hook::{{closure}}::h2f257dae1f0f72ca
   5:        0x102dbf524 - std::panicking::default_hook::h6488302641c8f9a5
   6:        0x10c184e38 - <alloc[e0ed195e94aa089e]::boxed::Box<rustc_driver_impl[ef5e448ef3e95e63]::install_ice_hook::{closure#0}> as core[12693b1b2ef4f00a]::ops::function::Fn<(&dyn for<'a, 'b> core[12693b1b2ef4f00a]::ops::function::Fn<(&'a core[12693b1b2ef4f00a]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[12693b1b2ef4f00a]::marker::Send + core[12693b1b2ef4f00a]::marker::Sync, &core[12693b1b2ef4f00a]::panic::panic_info::PanicInfo)>>::call
   7:        0x102dbfe54 - std::panicking::rust_panic_with_hook::h7d37ace162996714
   8:        0x102dbfc1c - std::panicking::begin_panic_handler::{{closure}}::h22ef26c89493a874
   9:        0x102dbd408 - std::sys_common::backtrace::__rust_end_short_backtrace::h12707e46a10e13f1
  10:        0x102dbf9b8 - _rust_begin_unwind
  11:        0x102e19120 - core::panicking::panic_fmt::h6b4a24417ba9d930
  12:        0x10d55ff58 - rustc_query_system[d5549c59454d906e]::query::job::deadlock
  13:        0x10c171e90 - std[1def8dc791e8ed96]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[7c00d2f3d4e7e51d]::util::run_in_thread_pool_with_globals<rustc_interface[7c00d2f3d4e7e51d]::interface::run_compiler<core[12693b1b2ef4f00a]::result::Result<(), rustc_span[bc478db2eb69d712]::ErrorGuaranteed>, rustc_driver_impl[ef5e448ef3e95e63]::run_compiler::{closure#0}>::{closure#0}, core[12693b1b2ef4f00a]::result::Result<(), rustc_span[bc478db2eb69d712]::ErrorGuaranteed>>::{closure#2}::{closure#1}, ()>
  14:        0x10c172730 - <<std[1def8dc791e8ed96]::thread::Builder>::spawn_unchecked_<rustc_interface[7c00d2f3d4e7e51d]::util::run_in_thread_pool_with_globals<rustc_interface[7c00d2f3d4e7e51d]::interface::run_compiler<core[12693b1b2ef4f00a]::result::Result<(), rustc_span[bc478db2eb69d712]::ErrorGuaranteed>, rustc_driver_impl[ef5e448ef3e95e63]::run_compiler::{closure#0}>::{closure#0}, core[12693b1b2ef4f00a]::result::Result<(), rustc_span[bc478db2eb69d712]::ErrorGuaranteed>>::{closure#2}::{closure#1}, ()>::{closure#1} as core[12693b1b2ef4f00a]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  15:        0x102dc803c - std::sys::unix::thread::Thread::new::thread_start::haf9ae65b2a0bdc21
  16:        0x1918e7fa8 - __pthread_joiner_wake

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please attach the file at `/Users/martin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustls-0.21.6/rustc-ice-2023-11-29T08_18_08-20115.txt` to your bug report

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked -Z threads=8

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
deadlock handler panicked, aborting process
error: could not compile `rustls` (lib)

Caused by:
  process didn't exit successfully: `/Users/martin/.rustup/toolchains/nightly-aarch64-apple-darwin/bin/rustc --crate-name rustls --edition=2021 /Users/martin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustls-0.21.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=114 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --cfg 'feature="default"' --cfg 'feature="log"' --cfg 'feature="logging"' --cfg 'feature="tls12"' -C metadata=67fcfb13440bd256 -C extra-filename=-67fcfb13440bd256 --out-dir /Users/martin/work/backend/main/target/debug/deps -L dependency=/Users/martin/work/backend/main/target/debug/deps --extern log=/Users/martin/work/backend/main/target/debug/deps/liblog-aa4856433adc5a5e.rmeta --extern ring=/Users/martin/work/backend/main/target/debug/deps/libring-745c94965474232e.rmeta --extern webpki=/Users/martin/work/backend/main/target/debug/deps/libwebpki-9dedcb39c13b0664.rmeta --extern sct=/Users/martin/work/backend/main/target/debug/deps/libsct-075319b689d94f4d.rmeta --cap-lints allow -Z threads=8 -L native=/Users/martin/work/backend/main/target/debug/build/ring-d4681648d316f62e/out` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...