Tracking issue for the to_bytes and from_bytes methods of integers

This tracks the stabilization of two methods on each primitive integer type, added in PR #49871:

impl $Int {
    pub fn to_bytes(self) -> [u8; mem::size_of::<Self>()] {
        unsafe { mem::transmute(self) }
    }
    pub fn from_bytes(bytes: [u8; mem::size_of::<Self>()]) -> Self {
        unsafe { mem::transmute(bytes) }
    }
}

Previous issue message:

I’d like to propose adding to the standard library between various integer types $Int and byte arrays [u8; size_of::<$Int>()] (which by the way is literally a valid type today). The implementation would be exactly transmute, but since the signature is much more restricted and all bit patterns are valid for each of the types involved, these conversions are safe.

Transmuting produces arrays with the target platform’s endianness. When something different is desired, the existing to_be/to_le/from_be/from_le methods can be combined with these new conversions. Keeping these concerns orthogonal (instead of multiplying ad-hoc conversions) allows to keep the API surface small.

Wrapping specific forms of transmute into safe APIs makes good candidates for the standard library IMO since they can save users from needing writing (and reviewing and maintaining) unsafe code themselves. See Box::into_raw for example. Together with the existing {to,from}_{be,le} methods and TryFrom<&[T]> for &[T; $N] impls, these new conversions would cover much of the functionality of the popular byteorder crate with little code and a relatively small API surface.

What I’m less certain about (and why this isn’t a PR yet) is what API should we expose these conversions as. Options are:

Impls of the From trait, or
Named methods, similar to f32::to_bits and f32::from_bits. The advantage over From is that we can give specific names to these conversions in order to communicate what they do. The downside is that we need to pick names.
- I initially thought of to_native_endian_bytes and from_native_endian_bytes but that’s not great because:
  - It’s somewhat inconsistent with to_be and friends which are much more abbreviated. (But maybe they shouldn’t be. It is worth adding to_big_endian & co and deprecating the short ones?)
  - It looks weird when combining them for writing portable code: n.to_be().to_native_endian(): now "native endian" is inaccurate, but that’s partly the fault of to_be for changing the meaning of a value without changing its type.
- Another idea is simply to_bytes and from_bytes, but that’s uninformative enough that they could just as well be From impls.

@rust-lang/libs or anyone, any thoughts?

I definitely like the idea of having these. Because of the multiple possible interpretations I'd lean towards inherent methods rather than From impls. I think to_bytes and from_bytes are reasonable names - to_native_endian is a bit confusing I agree.

Two recent internals threads with thoughts around this area:

It seems to me like there's a general common theme here of "safe, but a bit weird and rather transmute-y" conversions: this thread's uN <=> [u8; N/8], the first thread's u16x8 <=> u32x4 and u32 <=> f32, some parts of as like u32 <=> i32 that currently don't have a method version, and extended versions of that like &'a u32 <=> &'a i32 that are never exposed as safe today (but could be).

So here's a sketch of an idea using #[marker] traits:

#[marker] unsafe trait InplaceReinterpretAs<T> {}
unsafe impl<T> InplaceReinterpretAs<T> for T {}
unsafe impl InplaceReinterpretAs<[u8; 4]> for u32 {}
unsafe impl InplaceReinterpretAs<i32> for u32 {}
unsafe impl InplaceReinterpretAs<u32> for i32 {}
unsafe impl<T, U> InplaceReinterpretAs<*const U> for *const T {}
unsafe impl<T, U> InplaceReinterpretAs<*mut U> for *mut T {}
unsafe impl InplaceReinterpretAs<u16x8> for u32x4 {}
unsafe impl InplaceReinterpretAs<u32x4> for u16x8 {}

#[marker] unsafe trait ReinterpretAs<T> {
    // Because it's a marker trait, these cannot be overridden,
    // and thus their behaviour is always predicatable
    fn reinterpret(self) -> T {
        unsafe {
            let r = ptr::read_unaligned(&self as *const Self as *const T);
            mem::forget(self);
            r
        }
    }
    unsafe fn reinterpret_unchecked(x: T) -> Self {
        let r = ptr::read_unaligned(&x as *const T as *const Self);
        mem::forget(x);
        r
    }
}
unsafe impl<T, U> ReinterpretAs<U> for T where T: InplaceReinterpretAs<U> {}
unsafe impl<'a, T, U> ReinterpretAs<&'a U> for &'a T where T: InplaceReinterpretAs<U> {}
unsafe impl<'a, T, U> ReinterpretAs<&'a mut U> for &'a mut T where T: InplaceReinterpretAs<U> {}
unsafe impl ReinterpretAs<u32> for [u8;4] {} // not ok in-place, but fine as memcpy

Certainly std is generally adverse to introducing these using traits, but I think the recursiveness of the scenario makes the trait version more compelling than normal in this case. If one can turn a u32 into a [u8; 4] safely, why not also be able to turn a &[u32] into a &[[u8; 4]] safely?

(Name inspired by C++'s reinterpret_cast, obviously.)

@scottmcm

@scottmcm There’s probably something interesting there, but this feels like the definition of scope creep and I’m not gonna personally pursue it today. Byte arrays/slices are "special" because they’re what’s used for I/O, let’s keep this particular issue specifically about them.

@sfackler

@sfackler Alright, from_bytes and to_bytes sound good. Remaining questions:

Should we have these methods for signed integers? The Reference already specifies that they’re two’s complement.
Should we have them for pointer-sized integers? The portability story with arrays of different size feels more iffy than with endianness.

It definitely makes sense for signed integers. The case for usize/isize is interesting. I think we should probably still have them and maybe just note in the docs that people should be careful about using literal sizes.

Rollup merge of rust-lang#49871 - SimonSapin:int-bytes, r=sfackler

@rfcbot

This landed almost two months ago. Let’s stabilize?

@rfcbot fcp merge

You are correct that some users will want native Endianness. IMO it's wrong to make the easiest option platform-dependent, but I don't see a good, concise alternative.

You are also right about these not being special with regards to integer operations. Having traits is very useful for generic code, so an Int trait is a good idea IMO. Not sure why the current design has so many duplicated methods, but never mind.

Then I have no more concerns regarding this proposal. 👍

(The Int trait was removed in #23549 and moved to https://crates.io/crates/num / https://crates.io/crates/num-traits.)

Stabilize to_bytes and from_bytes for integers.

Auto merge of #51835 - tmccombs:stable-int-to-from-bytes, r=joshtriplett

Reopening for now because of #52850 and #51919.

Rollup merge of rust-lang#52850 - SimonSapin:unstablize, r=alexcrichton

Auto merge of #52850 - SimonSapin:unstablize, r=alexcrichton

It looks like unstabilization #52850 made it into beta 1.29. Moving to #52963, as I don’t know if rfcbot will enjoy doing a second FCP in the same issue.

Change tracking issue from rust-lang#49792 to rust-lang#51919

Navigation Menu

Repository navigation

Tracking issue for the to_bytes and from_bytes methods of integers #49792

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue for the to_bytes and from_bytes methods of integers #49792

Description

Activity

sfackler commented on Apr 9, 2018

scottmcm commented on Apr 9, 2018

SimonSapin commented on Apr 9, 2018

SimonSapin commented on Apr 9, 2018

sfackler commented on Apr 10, 2018

SimonSapin commented on Jun 6, 2018

8 remaining items

dhardy commented on Jun 21, 2018

SimonSapin commented on Jun 21, 2018

SimonSapin commented on Jul 30, 2018

SimonSapin commented on Aug 1, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions