Suboptimal layout for nested enums

[moulins]

I tried this code:

pub enum Outer {
    A(Inner),
    B(u32, u8),
}

pub enum Inner {
    A(u32),
    B(u32),
}

fn main() {
  dbg!(std::mem::size_of::<Outer>());
}

I expected size_of::<Outer>() to be 8, indicating that rustc was able to "merge" the two enums:

• Outer::A(Inner::A) gets discriminant 0;
• Outer::B(Inner::B) gets discriminant 1;
• Outer::B gets discriminant 2;

Instead, size_of::<Outer>() is 12, indicating that no layout optimization was done.

Meta

$ rustc +nightly --version
rustc 1.92.0-nightly (fa3155a64 2025-09-30)

[zachs18]

In this case, it will not be possible to optimize Outer to 8 bytes without changing Inner's layout, because Inner has a 4-byte tag¹, so even if the niche-filling layout were used for Outer, Outer::B would have 9 bytes of data, which cannot fit in 8 bytes. The layout of repr(Rust) enums is not stable, so we could change Inner's layout to have a smaller tag, but IIUC it is intentional that both: 1. the largest tag that does not increase the size/align of the enum is chosen, to maximize available niches, and 2. that layout calculation does not depend on usage so that the layout can be computed without knowing how the type will be used.

---

For a slightly different example (that you gave on discord), with Inner as follows:

pub enum Inner {
    A(bool, u32),
    B(u32),
}

In this case, Inner's tag is the bool niche, so Outer could be laid out in 8 bytes, but is not currently.

(Or as in OP, but #[repr(u8)] enum Inner)

I believe this is because, when computing the niche-filling layout, we compute the layout for each variant individually (with no tag, as though it were a struct), choose the largest variant, take it's niche field, then try to fit each other variant's layout around the niche field as a whole, i.e. we do not try to move fields around in the other variants' layouts to fit around the niche field.

rust/compiler/rustc_abi/src/layout.rs

Lines 656 to 675 in 99b9a88

	if layout.size <= niche_offset {
	// This variant will fit before the niche.
	return true;
	}
	// Determine if it'll fit after the niche.
	let this_align = layout.align.abi;
	let this_offset = (niche_offset + niche_size).align_to(this_align);
	if this_offset + layout.size > size {
	return false;
	}
	// It'll fit, but we need to make some adjustments.
	match layout.fields {
	FieldsShape::Arbitrary { ref mut offsets, .. } => {
	for offset in offsets.iter_mut() {
	*offset += this_offset;
	}
	}

In this example, Outer::A has a 1-byte niche at offset 0 and has size 8, and Outer::B is given layout [4B u32. 1B u8, 3B padding], which cannot be moved as a whole in any way to fit "around" the niche while still being within 8 bytes.

An optimization to make Outer::B fit around the niche would need to consider its fields individually.

Footnotes

1. playground link with #[rustc_layout(debug)] on Inner ↩