Bit-for-bit deterministic / reproducible builds

[infinity0]

It would be good if rustc could generate bit-for-bit reproducible results, even in the presence of minor system environment differences. Currently we have quite a large diff: e.g. see txt diff for 1.9.0 or perhaps txt diff for 1.10.0 a few days after I'm posting this. (You might want to "save link as" instead of displaying it directly in the browser.)

Much of the diff output is due to build-id differences, which can be ignored since they are caused by other deeper issues and will go away once these deeper issues are fixed. One example of a deeper issue is this:

│   │   │   ├── ./usr/lib/i386-linux-gnu/librustc_data_structures-e8edd0fd.so
│   │   │   │   ├── readelf --wide --file-header {}
│   │   │   │   │ @@ -6,15 +6,15 @@
│   │   │   │   │    OS/ABI:                            UNIX - System V
│   │   │   │   │    ABI Version:                       0
│   │   │   │   │    Type:                              DYN (Shared object file)
│   │   │   │   │    Machine:                           Intel 80386
│   │   │   │   │    Version:                           0x1
│   │   │   │   │    Entry point address:               0x4620
│   │   │   │   │    Start of program headers:          52 (bytes into file)
│   │   │   │   │ -  Start of section headers:          338784 (bytes into file)
│   │   │   │   │ +  Start of section headers:          338780 (bytes into file)
│   │   │   │   │    Flags:                             0x0
│   │   │   │   │    Size of this header:               52 (bytes)
│   │   │   │   │    Size of program headers:           32 (bytes)
│   │   │   │   │    Number of program headers:         8
│   │   │   │   │    Size of section headers:           40 (bytes)
│   │   │   │   │    Number of section headers:         30
│   │   │   │   │    Section header string table index: 29
│   │   │   │   ├── readelf --wide --program-header {}

Here are the system variations that might be causing these issues. I myself am not that familiar with ELF, but perhaps someone else here would know why the section header is 4 bytes later in the first vs second builds.

[steveklabnik]

In general, being reproducible is something we're interested in; we try to tackle it bug by bug.

[eefriedman]

This probably has some obvious cause:

│   │   │   │ -<h4 id='method.unzip' class='method'><code>fn <a href='../../core/iter/trait.Iterator.html#method.unzip' class='fnname'>unzip</a>&lt;A, B, FromA, FromB&gt;(self) -&gt; (FromA, FromB) <span class='where'>where FromB: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;B&gt;, FromA: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;A&gt;, Self: <a class='trait' href='../../core/iter/trait.Iterator.html' title='core::iter::Iterator'>Iterator</a>&lt;Item=(A, B)&gt;</span></code></h4>
│   │   │   │ +<h4 id='method.unzip' class='method'><code>fn <a href='../../core/iter/trait.Iterator.html#method.unzip' class='fnname'>unzip</a>&lt;A, B, FromA, FromB&gt;(self) -&gt; (FromA, FromB) <span class='where'>where FromB: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;B&gt;, Self: <a class='trait' href='../../core/iter/trait.Iterator.html' title='core::iter::Iterator'>Iterator</a>&lt;Item=(A, B)&gt;, FromA: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;A&gt;</span></code></h4>

If someone wants to look at the code generation differences, it's probably best to start with libcore. The reproducible-builds.org diff isn't that useful for that because it doesn't recognize .rlib files as archives.

[infinity0]

Thanks for the heads up, I just added AR support in diffoscope and hopefully the diff output will see that in the next few weeks, when the website updates.

[infinity0]

Here's a diff of rust.metadata.bin from libcore.rlib, from 1.10.0. (2.0 MB, ~1700 lines, might make your browser slow.) Mostly ordering differences. @eddyb has suggested HashMap ordering as a culprit, and that would be replaced with FnvHashMap instead.

[eddyb]

If it's a HashMap, I don't know which one. cc @rust-lang/compiler

[infinity0]

I tried replacing HashMap with FnvHashMap in src/librustc_metadata/loader.rs but it didn't seem to help. Here is the Makefile i'm using to automate my rebuilds:

TRIPLET = x86_64-unknown-linux-gnu

all: libcore.diff

libcore.diff: rust.metadata.bin.1 rust.metadata.bin.2
    diffoscope --html-dir "$@" rust.metadata.bin.1 rust.metadata.bin.2 \
        --max-diff-block-lines 5000 \
        --max-diff-input-lines 10000000 \
        --max-report-size 204800000; true # diffoscope returns 1 for "there were diffs"

rust.metadata.bin.1 rust.metadata.bin.2: Makefile
    rm -f $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/libcore-*.rlib
    rm -f $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/stamp.core
    $(MAKE) $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/stamp.core
    ar x $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/libcore-*.rlib rust.metadata.bin
    mv rust.metadata.bin "$@"

.PHONY: all rust.metadata.bin.1 rust.metadata.bin.2

perhaps i'm Doing It Wrong.

[infinity0]

As background, HashMap in rust is non-deterministic to protect against certain types of DoS attack. You can switch it to the deterministic FnvHashMap if you're sure your code will always be called in a safe manner. This ought to be true for rustc itself. (I notice some online "try-it-yourself" rust web services let me run "ls /" and other shell commands, so I could also exploit this HashMap ordering issue, but I also assume that they're clever enough to set a ulimit and/or containerise the thing.)

[jonas-schievink]

I wrote a small script to diff metadata (I couldn't really get the makefile to work).

It looks like even more is changing between two compiles using the current master (or nightly): The crate hash and/or disambiguator, which are stored right after the target triple. @infinity0 can you confirm?

EDIT: That might get fixed by #35854

[jonas-schievink]

Okay, apparently replacing Hash{Map,Set} in resolve with FnvHash{Map,Set} got rid of the first (smaller) blocks of the diff. The big block at the bottom is still there. Will continue to randomly sed the compiler and report back ;)

[jonas-schievink]

The remaining issue is that (some?) predicates are encoded in non-deterministic order. Maybe we need to do this for all bounds? Not sure what would be the correct way to do that (or why these are nondeterministic in the first place).

[michaelwoerister]

Just for reference, there's this: #34805

[jonas-schievink]

@michaelwoerister Thanks! Too tired to think about it currently. Will have a look tomorrow :)