Representation of structs

[nikomatsakis]

Discussion topic about how structs are represented in Rust. Some things to work out:

• Do we ever say anything about how a #[repr(rust)] struct is laid out
  (and/or treated by the ABI)?
  - e.g., what about different structs with same definition
  - across executions of the same program?
• For example, rkruppe writes that we might "want to guarantee (some subset of) newtype unpacking and relegate #[repr(transparent)] to being the way to guarantee to other crates that a type with private fields is and will remain a newtype?"
• When is interop with #[rust(C)] guaranteed and what can we say there?

[nikomatsakis]

I for one would like someone to talk a bit about what #[repr(C)] really means anyway. It seems to play a number of roles for us, at once defining a kind of fixed layout strategy but maybe also having other effects?

Is it worth maybe trying to tease those out into distinct repr flags (any lang changes would require an RFC, of course).

There may also be lesser variants that are useful: for example, I think it would be useful to be able to say that "the first field appears at offset 0" but say nothing about the positions or layouts of other fields within a struct.

It'd be good -- while we're thinking about this stuff -- to think about the kinds of things we might want to say but currently cannot.

[joshtriplett]

The C11 spec provides some fairly detailed guidance on structure layout, which we could apply to repr(C) structs.

[gnzlbg]

The C17 standard is here: http://www.open-std.org/jtc1/sc22/wg14/www/abq/c17_updated_proposed_fdis.pdf

The struct layout is specified in 6.7.2.1 Structure and union specifiers (starting at paragraph 3). Ignoring bitfields and other things that do not apply to Rust, things that might be relevant are:

3. A structure or union shall not contain a member with incomplete or function type (hence, a structure shall not contain an instance of itself, but may contain a pointer to an instance of itself), except that the last member of a structure with more than one named member may have incomplete array type; such a structure (and any union containing, possibly recursively, a member that is such a structure) shall not be a member of a structure or an element of an array.

4. As discussed in 6.2.5, a structure is a type consisting of a sequence of members, whose storage is allocated in an ordered sequence.

5. Within a structure object, the non-bit-field members and the units in which bit-fields reside have
   addresses that increase in the order in which they are declared. A pointer to a structure object,
   suitably converted, points to its initial member (or if that member is a bit-field, then to the unit in
   which it resides), and vice versa. There may be unnamed padding within a structure object, but not
   at its beginning.

6. There may be unnamed padding at the end of a structure or union.

7. As a special case, the last element of a structure with more than one named member may have an
   incomplete array type; this is called a flexible array member. In most situations, the flexible array
   member is ignored. In particular, the size of the structure is as if the flexible array member were
   omitted except that it may have more trailing padding than the omission would imply. However,
   when a . (or-> ) operator has a left operand that is (a pointer to) a structure with a flexible array
   member and the right operand names that member, it behaves as if that member were replaced with the longest array (with the same element type) that would not make the structure larger than the object being accessed; the offset of the array shall remain that of the flexible array member, even if this would differ from that of the replacement array. If this array would have no elements, it behaves as if it had one element but the behavior is undefined if any attempt is made to access that element or to generate a pointer one past it.

---

I think that we should just say that #[repr(C)] Rust's structs have the same layout as C structs as specified in some C standard version and call it a day. Newer C standards are (often) backwards compatible with older ones, so as long as the changes are backwards compatible, we could upgrade the spec as new standards are released to point to these without issues.

[the8472]

Let's say the compiler keeps the freedom to choose arbitrary layouts at compile time, would punning be valid after checking that the layout just happens to be identical (via some offset_of construct)?

E.g. as an optimization to just cast between repr(C) and repr(rust) where things match and otherwise shoveling data from one type the other field by field?

[joshtriplett]

On Fri, Aug 31, 2018 at 12:14:02PM +0000, gnzlbg wrote: Ignoring bitfields and other things that do not apply to Rust,

...yet. :)

[nikomatsakis]

I'm going to stake out a tentative position:

• I think we may well want the freedom to reorder distinct #[repr(Rust)] structs independently, for example based on PGO information.
  • The only real thing this seems to rule out is people using transmute to gain "unauthorized" access to private fields of data structures, right?
  • That is definitely a real thing, but also a pretty sketchy one.
• I think that the following repr options might be useful:
  • #[repr(prefix)], which would force the first field to be at offset 0, but not constrain the rest
    • useful for guaranteeing "prefix" properties
  • #[repr(in_order)], whihc would force fields to be laid out in order, but not otherwise require full #[repr(C)] compatibility
    • though maybe not: what else does #[repr(C)] even do?

[hanna-kruppe]

My quick two cents on easy questions before I tackle the big issue of what to do about repr(Rust):

@the8472

Let's say the compiler keeps the freedom to choose arbitrary layouts at compile time, would punning be valid after checking that the layout just happens to be identical (via some offset_of construct)?

Yes, this should definitely be valid if done properly. The checks can get quite involved though, if we truly guarantee nothing about repr(Rust) layout, one would have to check at minimum:

• alignment of the whole struct (since the compiler would be allowed to raise the alignment requirements of the repr(Rust) struct)
• offsets of all fields, as you say
• size of the whole struct (even if the offsets of all fields are identical, tail padding would be a possibility)

@nikomatsakis

• #[repr(in_order)], whihc would force fields to be laid out in order, but not otherwise require full #[repr(C)] compatibility
  • though maybe not: what else does #[repr(C)] even do?

repr(C) also prescribes the amount and locations of padding, but I don't think this makes repr(in_order) useful: we can't lower the alignment requirements of the fields, and raising it (edit: or alignment of the struct itself) is basically useless. It can theoretically facilitate the use of instructions with higher alignment requirements (e.g., SIMD) or prevent false sharing, but both of those are very tricky and niche even when done by a programmer, and automatically detecting when these opportunities are present and justifying them against the increased padding (and the cache misses, memory bandwidth, peak rss, etc. implied by it) seems infeasible to me.

[joshtriplett]

I personally feel that we shouldn't add repr(inorder), or any other "some but not all of what repr(Rust) is free to do", until we have a specific, concrete use case for it.

I also hope we don't need repr(prefix) to ensure that initial fields stay in place.

Personally, I would greatly prefer that we handle structure ordering issues by helping developers reorder the actual structure order in the source, rather than deciding to reorder it at compile time. If you write a struct with a u8, u32, and u8, in that order, and it isn't repr(C), why don't we emit a warning rather than just reordering? (If you use repr(C) then you might need that exact layout for compatibility.)

[nikomatsakis]

I also hope we don't need repr(prefix) to ensure that initial fields stay in place.

Just to make sure I understand you: you would prefer that we respect the field ordering that the user gave? Or you hope that users never care about prefix?

Personally, I would greatly prefer that we handle structure ordering issues by helping developers reorder the actual structure order in the source, rather than deciding to reorder it at compile time.

Interesting. I feel the opposite. If the developer wants field order to be preserved, I would rather they say so, as I think it's an exceptional case and I'd like my attention to be drawn to it. Otherwise, I would prefer that I have the freedom to order my fields in a way that maximizes readability of my source, and without regard for "bitpacking" or other things.

[gnzlbg]

I think that the following repr options might be useful:

I think we already talked about this on IRC, but for the record, I do find these ideas for new reprs interesting, but think we should try to avoid introducing new reprs and features in the unsafe code guidelines if possible. There is so much to do already, that if we try to also specify features that Rust does not have yet, we might end up achieving nothing.

Once we have the guidelines closer to a finished state, if someone wants to fill an issue in the rfc repo to discuss these, or fill in a rfc, then we can discuss these and how to incorporate them into the guidelines there. Maybe the issues could be filled there already so that we don't forget?

[the8472]

Otherwise, I would prefer that I have the freedom to order my fields in a way that maximizes readability of my source, and without regard for "bitpacking" or other things.

Preserving order by default does make even less sense when generic fields are present.