failed to lookup `SourceFile` in new context - incremental compilation

[ijackson]

Steps to reproduce

git clone https://salsa.debian.org/iwj/otter
cd otter/
git checkout origin/wip.pre-ice.cargo-toml
# ^ 92577e37c70e11f975dc8fd3aabd104feeb0c4bb
cargo test --workspace --locked
# ^ builds many packages, and tests pass
git checkout origin/wip.ice.cargo-toml
# ^ e65778e27f65fd4de4355bb2308137e46dfb0e7d
cargo test --workspace --locked
# ^ ICE

Code

The code is my entire project, unfortunately. The ICE appears to be related to incremental compilation, so I'm not sure how to minimise it.

Background

(There was a large section here which is irrelevant now I have "steps to reproduce" and a repro on current nightly.)

Meta

rustc --version --verbose:

rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ rustc --version --verbose
rustc 1.52.0-nightly (acca81892 2021-03-13)
binary: rustc
commit-hash: acca818928654807ed3bc1ce0e97df118f8716c8
commit-date: 2021-03-13
host: x86_64-unknown-linux-gnu
release: 1.52.0-nightly
LLVM version: 12.0.0
rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ 

Error output

rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ cargo test --workspace --locked
   Compiling otter-api-tests v0.0.1 (/volatile/rustcargo/Otter/Ice/otter/apitest)
thread 'rustc' panicked at 'failed to lookup `SourceFile` in new context', compiler/rustc_middle/src/ty/query/on_disk_cache.rs:697:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (acca81892 2021-03-13) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [optimized_mir] optimizing MIR for `main`
#1 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack
error: could not compile `otter-api-tests`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ 

Backtrace

rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ RUST_BACKTRACE=1 cargo test --workspace --locked
   Compiling otter-api-tests v0.0.1 (/volatile/rustcargo/Otter/Ice/otter/apitest)
   Compiling otter-webdriver-tests v0.0.1 (/volatile/rustcargo/Otter/Ice/otter/wdriver)
thread 'rustc' panicked at 'failed to lookup `SourceFile` in new context', compiler/rustc_middle/src/ty/query/on_disk_cache.rs:697:22
stack backtrace:
thread 'rustc' panicked at 'failed to lookup `SourceFile` in new context', compiler/rustc_middle/src/ty/query/on_disk_cache.rs:697:22
stack backtrace:
   0: rust_begin_unwind
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/core/src/panicking.rs:92:14
   2: core::option::expect_failed
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/core/src/option.rs:1321:5
   3: rustc_middle::ty::query::on_disk_cache::CacheDecoder::file_index_to_file
   4: rustc_middle::ty::query::on_disk_cache::<impl rustc_serialize::serialize::Decodable<rustc_middle::ty::query::on_disk_cache::CacheDecoder> for rustc_span::span_encoding::Span>::decode
   5: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_SourceInfo::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::SourceInfo>::decode
   6: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_Statement::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::Statement>::decode
   7: rustc_serialize::serialize::Decoder::read_seq
   8: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_BasicBlockData::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::BasicBlockData>::decode
   9: rustc_serialize::serialize::Decoder::read_seq
  10: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_Body::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::Body>::decode
  11: rustc_middle::ty::query::on_disk_cache::OnDiskCache::try_load_query_result
  12: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  13: rustc_data_structures::stack::ensure_sufficient_stack
  14: rustc_query_system::query::plumbing::get_query_impl
  15: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::optimized_mir
  16: rustc_middle::ty::<impl rustc_middle::ty::context::TyCtxt>::instance_mir
  17: rustc_mir::monomorphize::collector::collect_neighbours
  18: rustc_mir::monomorphize::collector::collect_items_rec
  19: rustc_session::utils::<impl rustc_session::session::Session>::time
  20: rustc_mir::monomorphize::collector::collect_crate_mono_items
  21: rustc_mir::monomorphize::partitioning::collect_and_partition_mono_items
  22: rustc_query_impl::<impl rustc_query_system::query::config::QueryAccessors<rustc_query_impl::plumbing::QueryCtxt> for rustc_query_impl::queries::collect_and_partition_mono_items>::compute
  23: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  24: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  25: rustc_data_structures::stack::ensure_sufficient_stack
  26: rustc_query_system::query::plumbing::force_query_with_job
  27: rustc_query_system::query::plumbing::get_query_impl
  28: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::collect_and_partition_mono_items
  29: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  30: rustc_interface::passes::QueryContext::enter
  31: rustc_interface::queries::Queries::ongoing_codegen
  32: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  33: rustc_span::with_source_map
  34: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (acca81892 2021-03-13) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [optimized_mir] optimizing MIR for `main`
#1 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack
error: could not compile `otter-api-tests`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
   0: rust_begin_unwind
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/core/src/panicking.rs:92:14
   2: core::option::expect_failed
             at /rustc/acca818928654807ed3bc1ce0e97df118f8716c8/library/core/src/option.rs:1321:5
   3: rustc_middle::ty::query::on_disk_cache::CacheDecoder::file_index_to_file
   4: rustc_middle::ty::query::on_disk_cache::<impl rustc_serialize::serialize::Decodable<rustc_middle::ty::query::on_disk_cache::CacheDecoder> for rustc_span::span_encoding::Span>::decode
   5: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_SourceInfo::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::SourceInfo>::decode
   6: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_Statement::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::Statement>::decode
   7: rustc_serialize::serialize::Decoder::read_seq
   8: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_BasicBlockData::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::BasicBlockData>::decode
   9: rustc_serialize::serialize::Decoder::read_seq
  10: rustc_middle::mir::_DERIVE_rustc_serialize_Decodable_D_FOR_Body::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::mir::Body>::decode
  11: rustc_middle::ty::query::on_disk_cache::OnDiskCache::try_load_query_result
  12: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  13: rustc_data_structures::stack::ensure_sufficient_stack
  14: rustc_query_system::query::plumbing::get_query_impl
  15: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::optimized_mir
  16: rustc_middle::ty::<impl rustc_middle::ty::context::TyCtxt>::instance_mir
  17: rustc_mir::monomorphize::collector::collect_neighbours
  18: rustc_mir::monomorphize::collector::collect_items_rec
  19: rustc_session::utils::<impl rustc_session::session::Session>::time
  20: rustc_mir::monomorphize::collector::collect_crate_mono_items
  21: rustc_mir::monomorphize::partitioning::collect_and_partition_mono_items
  22: rustc_query_impl::<impl rustc_query_system::query::config::QueryAccessors<rustc_query_impl::plumbing::QueryCtxt> for rustc_query_impl::queries::collect_and_partition_mono_items>::compute
  23: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  24: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  25: rustc_data_structures::stack::ensure_sufficient_stack
  26: rustc_query_system::query::plumbing::force_query_with_job
  27: rustc_query_system::query::plumbing::get_query_impl
  28: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::collect_and_partition_mono_items
  29: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  30: rustc_interface::passes::QueryContext::enter
  31: rustc_interface::queries::Queries::ongoing_codegen
  32: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  33: rustc_span::with_source_map
  34: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (acca81892 2021-03-13) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [optimized_mir] optimizing MIR for `main`
#1 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack
error: build failed
rustcargo@zealot:/volatile/rustcargo/Otter/Ice/otter$ 

[ijackson]

@rustbot modify labels +A-incr-comp +E-needs-mcve

[Aaron1011]

@ijackson: Neither origin/wip.ice.cargo-toml nor commit e65778e27f65fd4de4355bb2308137e46dfb0e7dESC exists when I clone that repository.

[ijackson]

@Aaron1011 Err, sorry. Not sure what happened there. Maybe I deleted the remote branch by mistake somehow. I have the commit locally so I have pushed it again. In case of other confusion I also retried my "steps to repro" and it still goes wrong as I described.

Also I noticed that the commitids there have a spurious "ESC" at the end (!), the result of a git+less misbehaviour. I have deleted those from the summary above.

[Aaron1011]

This will be fixed by #83132