Spurious failures in sccache

[alexcrichton]

Outstandling failures:

Failed to spawn child broken pipe

maybe fixed in #40809?

• https://travis-ci.org/rust-lang/rust/jobs/210449425
• https://travis-ci.org/rust-lang/rust/jobs/210356090
• https://travis-ci.org/rust-lang/rust/jobs/210674231
• https://travis-ci.org/rust-lang/rust/jobs/217614672
• https://travis-ci.org/rust-lang/rust/jobs/217513855
• https://travis-ci.org/rust-lang/rust/jobs/217609418

error: failed to execute compile
caused by: failed to spawn child
caused by: Broken pipe (os error 32)
make[3]: *** [bin/llvm-config] Error 2

Notes:

• no theories maybe suspicion: EPIPE coming out of register on OSX tokio-rs/mio#582?
• Always OSX so far
• Both 32 and 64 bit
• presumably happens here

Error 254

Suspected bug in sccache. Hidden by mozilla/sccache#82 by accident. Sccache update in #40676

maybe fixed in #40809?

• https://travis-ci.org/rust-lang/rust/jobs/208053065
• https://travis-ci.org/rust-lang/rust/jobs/208060953
• https://travis-ci.org/rust-lang/rust/jobs/209158635
• https://travis-ci.org/rust-lang/rust/jobs/209740993
• https://travis-ci.org/rust-lang/rust/jobs/213667930

notes:

• One job failed nul bytes in env, args, or program passed to Command when creating preprocessor. This seems crazy. Memory corruption?
• OSX or Linux so far, no Windows.
• Most indicate just one compilation failing, the last log indicates the entire server crashed w/ a segfault or something

Fixed failures

Unexpected EOF

Hoping to be fixed by mozilla/sccache#79

• https://travis-ci.org/rust-lang/rust/jobs/210484803
• https://travis-ci.org/rust-lang/rust/jobs/207425298
• https://travis-ci.org/rust-lang/rust/jobs/207606309
• https://travis-ci.org/rust-lang/rust/jobs/208643512
• https://travis-ci.org/rust-lang/rust/jobs/208893289
• https://travis-ci.org/rust-lang/rust/jobs/208849166
• https://travis-ci.org/rust-lang/rust/jobs/209162867
• https://travis-ci.org/rust-lang/rust/jobs/210063204
• https://travis-ci.org/rust-lang/rust/jobs/210145670
• https://travis-ci.org/rust-lang/rust/jobs/210484803
• https://travis-ci.org/rust-lang/rust/jobs/210685018
• https://travis-ci.org/rust-lang/rust/jobs/211605305
• https://travis-ci.org/rust-lang/rust/jobs/211341835
• https://travis-ci.org/rust-lang/rust/jobs/211214150
• https://travis-ci.org/rust-lang/rust/jobs/211758389
• https://travis-ci.org/rust-lang/rust/jobs/212004583

error: failed to execute compile
caused by: error reading compile response from server
caused by: IoError(Error { repr: Custom(Custom { kind: Other, error: StringError("unexpected EOF") }) })
caused by: unexpected EOF
make[3]: *** [tools/llvm-readobj/CMakeFiles/llvm-readobj.dir/ELFDumper.cpp.o] Error 2

Forcibly closed by remote host

Suspected fixed in mozilla/sccache@5f6932b

PR sent in #40676

• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2344/job/6ky8r85djtr9l202
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2366/job/xnj3xxp6plfqig8p
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2368/job/4x9x6h3an4cb331c
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2377/job/e0s6p5yx4abub3hw
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2474/job/8x4h54hyj4uh16td

error: failed to execute compile
caused by: Failed to send data to or receive data from server
caused by: IoError(Error { reprError { : Os { code: 10054repr: Os { code: 10054, message: "An existing connection was forcibly closed by the remote host." } })

notes:

• Only windows so far
• Both 32 and 64 bit
• Ignore segfaults, they're likely Remove dependency on named_pipe mozilla/sccache#83
• Both beginning and end of the build
• appears that the server process is crashing
• suspect, crossbeam segfaulting (there are known segfaults in there)

Wedged in build logs

wedged in mio, likely fixed in #40809

• https://travis-ci.org/rust-lang/rust/jobs/214641046
• https://travis-ci.org/rust-lang/rust/jobs/214659698

Errors not at the fault of sccache

ar.exe/ranlib errors

More tracked at #40546 suspected to be unrelated to sccache and instead related to MinGW makefiles (and how they're not always the best...)

Hoping to be fixed by #40548

• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2279/job/28y2keij5k2nggbm - ranlib file format not recognized
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2286/job/9f765q3opo03djpo - ar.exe no such file or directory
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2305/job/k56s0ybh90dbtl0j - ar.exe unable to rename
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2314/job/4u931mk93qpx78e3 - ar.exe unable to rename
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2325/job/s8x6tfi1hs2tfl48 - ar.exe permission denied
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2397/job/xmn5g0alr50n86uv - ar.exe permission denied

[alexcrichton]

First attempt to debug this landed in #40240, although that didn't cover OSX/Windows which seem to be failing a lot as well. Second attempt to debug this is #40442

[alexcrichton]

I've made a PR to sccache to hopefully squash at least some of the spurious failures we've been seeing (I don't think it will fix all of them though)

[alexcrichton]

The first failure above is getting an EPIPE on this line which baffles me. I have no idea how an EPIPE would arise during process creation.

[Mark-Simulacrum]

Looking at the two lines above, there's Stdio::piped(), which implies that during process creation we're opening the stdout and stderr file descriptors as pipes, which may be the cause of these problems. Not sure if that's the case though.

[alexcrichton]

I've found a segfault in sccache, but I don't think it's affected the bots all that much. I've seen a segfault here and there but not nearly as commonly as the above error conditions.

[luser]

The first failure above is getting an EPIPE on this line which baffles me. I have no idea how an EPIPE would arise during process creation.

There's also the pipe that's created in Command::spawn to report failures back from exec.

[alexcrichton]

These logs are brand new with all sccache fixes I know of so far.

My guess is memory corruption somewhere, which would also explain EPIPE I think.

@luser yeah the problem with that pipe though is that I don't know how it's possible for the parent process to receive EPIPE :. If we're randomly rewriting what fd you're writing to though (or randomly rewriting the error coming out) then I could see this being a problem...

[aturon]

Just saw this one:

mkdir C:\cache\rustsrc
A subdirectory or file C:\cache\rustsrc already exists.
Command exited with code 1
cat %CD%/sccache.log
cat: 'C:\projects\rust/sccache.log': No such file or directory
Command exited with code 1

#40842

[aturon]

Nevermind that latest, it was introduced in 0347ff5

[alexcrichton]

I have not seen any failures related to sccache over the past ~24 hrs, let's keep this open though just to make sure.