Missing optimization when transmuting enums

[Jarcho]

Transmuting a fieldless enum to it's integer type causes the optimizer to lose track of it's possible values. Note that as casts are optimized correctly.

Example:

#[repr(u32)]
enum E { X = 0 }
fn f(x: E) {
    assert_eq!(unsafe { core::mem::transmute::<E, i32>(x) }, 0); // Not optimized out
    assert_eq!(x as i32, 0); // optimized out
}

cc Lokathor/bytemuck#175

[nikic]

Godbolt: https://rust.godbolt.org/z/ExbzPGf7s

cc @scottmcm in case your scalar transmute changes are relevant.

Not an LLVM problem because the necessary information is never provided to LLVM.

[clubby789]

rust/compiler/rustc_mir_build/src/build/expr/as_rvalue.rs

Lines 267 to 268 in 90a9f69

	assumer(range.end, BinOp::Ge);
	assumer(range.start, BinOp::Le);

We emit a couple of assume's when casting an enum with as but no extra info is emitted for a transmute

[erikdesjardins]

If we could just put range metadata on arguments, we could emit

define noundef i32 @f(i32 noundef range(!0) %x)
...
!0 = !{i32 0, i32 1}

and then this and other cases (e.g. NonZeroX::get) would work without us having to add assumes for every conversion.

I actually have a partial implementation of this here. Though I assume allowing attributes to depend on metadata nodes would be untenable, they're probably separate intentionally? (@nikic thoughts?)

(My implementation also doesn't fully work because of this problem with RAUW on forward-declared metadata: erikdesjardins/llvm-project@c1342ac#diff-100c1b4a714c7d260fc41ac27cc8d2d5857d54021f16d73debc71125d110e7daR2809-R2831)

The alternative would be something like range(0, 1, 4, 5), where it doesn't use metadata to represent the ranges, and stores it in a bespoke attribute type, but I think this would also be a bit awkward. All range metadata handling code depends on the metadata representation (this could be refactored for sure), and attributes aren't equipped to hold variable-size arguments except strings, so I think some significant changes to Attribute would be required.

(Of course the other alternative would be to just add the assumes, but it feels a bit unsatisfying)

[scottmcm]

Transmuting a fieldless enum to it's integer type causes the optimizer to lose track of it's possible values.

If you do this directly to the repr type, it should no longer be a problem, since we MIR-optimize a transmute like that to a discriminant read: https://github.com/rust-lang/rust/pull/109612/files#diff-4f93c014037e17032815fe49c647952b4faf26aff2007afd9a36106f8d0f39f6R86-R87.

Hmm, playground is only on the 2023-04-02 nightly, so I can't look there to see if my change improved things.

[scottmcm]

With #109843, this:

// CHECK-LABEL: @check_to_enum(
#[no_mangle]
pub unsafe fn check_to_enum(x: i8) -> SmallEnum {
    transmute(x)
}

// CHECK-LABEL: @check_from_enum(
#[no_mangle]
pub unsafe fn check_from_enum(x: SmallEnum) -> i8 {
    transmute(x)
}

just emits

; Function Attrs: uwtable
define noundef i8 @check_to_enum(i8 noundef %x) unnamed_addr #0 {
start:
  ret i8 %x
}

; Function Attrs: uwtable
define noundef i8 @check_from_enum(i8 noundef %x) unnamed_addr #0 {
start:
  ret i8 %x
}

since it's passed as an immediate.

Loading an enum puts !range metadata on the load, but since there's no load here the OperandValue stuff I'm doing in the new code doesn't do anything with the niches.