Document guarantees (or lack thereof) regarding sign, quietness, and payload of `NaN`s

[ecstatic-morse]

NaNs can behave in surprising ways. On top of that, a very common target is inherently buggy in more than one way. But on all other targets we actually follow fairly clear, if improperly documented, rules. See here for the current status.

Original issue

Several issues have been filed about surprising behavior of NaNs.

• Wrong signs on division producing NaN #55131, in which the sign of the result of 0.0 / 0.0 changed depending on whether the right-hand side came from a function argument or a literal.
• i686 floating point behavior does not agree with unit tests in debug mode #73288/Some num tests fail for {i586, i686, x86_64}-unknown-linux-gnu #46948, in which the result of f32::from_bits(x).to_bits() was not always equal to x.

The root cause of these issues is that LLVM does not guarantee that NaN payload bits are preserved. Empirically, this applies to the signaling/quiet bit as well as (surprisingly) the sign bit. At least one LLVM developer seems open to changing this, although doing so may not be easy.

Unless we are prepared to guarantee more, we should do a better job of documenting that, besides having all 1s in the exponent and a non-zero significand, the bitwise value of a NaN is unspecified and may change at any point during program execution. In particular, the from_bits method on f32 and f64 types currently states:

This is currently identical to transmute::<u32, f32>(v) on all platforms.

and

this implementation favors preserving the exact bits. This means that any payloads encoded in NaNs will be preserved

These statements are misleading and should be changed.

We may also want to add documentation to {f32,f64}::NAN to this effect, see #52897 (comment).

cc #10186?

[ecstatic-morse]

This also affects the documentation for the methods in #72568.

[RalfJung]

@ecstatic-morse wrote elsewhere

Indeed. The underlying cause is clear. I wonder what we should do here, though? Does Rust currently guarantee that extended precision is not used for operations on f64? If so, this is technically a miscompilation. However, I don't know whether it's worth fixing. Maybe we should just document the status quo and move on?

I don't think we can easily just "move on" -- as mentioned here, what LLVM currently does seems incoherent and is likely just plain unsound (but miscompilations are hard to trigger). In that sense this is similar to #28728: LLVM in its current state makes it impossible to build a safe language on top of it with reasonable effort, which means fixing this will be a lot of work, but from a Rust perspective that's nevertheless a critical soundness bug.

Cc @rust-lang/lang

[ecstatic-morse]

That issue does not involve NaN, and that comment is not applicable here.

[RalfJung]

Fair. But I feel #72327 is related in the broader sense of "our FP semantics are a mess". Looks like we actually have two problems here:

• LLVM doesn't preserve NaN bits (which is likely incoherent). That's this issue.
• LLVM uses the x87 instructions on i686 which behave different from how IEEE floats should behave. That's Comparing to infinity is buggy on x87 #72327.

[RalfJung]

I created rust-lang/unsafe-code-guidelines#237 to collect FP issues. That's indeed off-topic here, sorry for that.