layout: Store inverse memory index in `FieldsShape::Arbitrary`#150116

moulins · 2025-12-17T23:37:18Z

All usages of memory_index start by calling invert_bijective_mapping, so storing the inverted mapping directly saves some work and simplifies the code.

rustbot · 2025-12-17T23:37:22Z

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

rustbot · 2025-12-17T23:37:25Z

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

workingjubilee · 2025-12-18T00:40:33Z

huh. I wonder what it's inverted relative to?

workingjubilee · 2025-12-18T01:06:25Z

@moulins I have no objection to the substance of the change but "inverse" suggests a relativeness this doesn't capture... IMO this is still just a "memory index" until you do something like specify the other half of the pair, whether it's the key or value, at which point they might adopt a logical ordering.

mem_src_field_index maybe? idk.

workingjubilee · 2025-12-18T01:08:58Z

kinda curious if this has any effect on perf or if it's just logical cleanup

@bors try @rust-timer queue

layout: Store inverse memory index in `FieldsShape::Arbitrary`

moulins · 2025-12-18T01:31:06Z

I have no objection to the substance of the change but "inverse" suggests a relativeness this doesn't capture... IMO this is still just a "memory index" until you do something like specify the other half of the pair, whether it's the key or value, at which point they might adopt a logical ordering.

mem_src_field_index maybe? idk

Hm, good point... what about in_memory_order?

workingjubilee · 2025-12-18T01:58:13Z

something like that sounds good to me!

rust-bors · 2025-12-18T03:24:22Z

☀️ Try build successful (CI)
Build commit: ed52dc8 (ed52dc8f27781eef22c18d699984ef94c758cc92, parent: f794a08738958bbef01c3dcab34ec21d3da1e51a)

rust-timer · 2025-12-18T04:19:17Z

Finished benchmarking commit (ed52dc8): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.1%	[-0.2%, -0.0%]	2
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary 1.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.8%	[1.8%, 1.8%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.8%	[1.8%, 1.8%]	1

Cycles

Results (primary 2.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.3%, 2.3%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[2.3%, 2.3%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 481.694s -> 485.432s (0.78%)
Artifact size: 390.59 MiB -> 390.53 MiB (-0.01%)

RalfJung · 2025-12-18T06:57:18Z

compiler/rustc_const_eval/src/interpret/visitor.rs

-            FieldsShape::Arbitrary { memory_index, .. } => {
-                for idx in Self::aggregate_field_iter(memory_index) {
+            FieldsShape::Arbitrary { inverse_memory_index, .. } => {
+                for idx in Self::aggregate_field_iter(inverse_memory_index) {


If iterating in memory order is now cheap, maybe we should just always do that... in fact that could unlock a bunch of follow-up cleanup here. Probably best to make all that a follow-up PR, but a good indication that this PR moves us in the right direction. :)

So you'd propose removing aggregate_field_iter entirely and and simply iterate the fields directly?

Yes. But I would suggest doing it in a separate PR so we can separately benchmark that change.

moulins · 2025-12-18T14:12:36Z

Renamed inverse_memory_index to in_memory_order, and updated some UI tests that I missed previously.

EDIT: I just noticed that rust-analyzer has its own version of FieldsShape, should I perform the same change here too?

workingjubilee · 2025-12-18T18:07:26Z

@moulins I don't think this type is related in a meaningful way? https://github.com/rust-lang/rust/blob/main/src/tools/rust-analyzer/crates/hir-def/src/item_tree.rs#L519

Do you mean something else?

workingjubilee · 2025-12-18T20:31:49Z

To be clear, I am 100% certain that rust-analyzer uses rustc_abi::FieldsShape also but have no idea what uses need to be changed if any.

moulins · 2025-12-18T21:04:38Z

You're totally right; from what I can see, the uses are here in Layout::{tail_padding, largest_padding}, but for some reason my ./x check still succeeds and trying to rename these uses causes an error.

All usages of `memory_index` start by calling `invert_bijective_mapping`, so storing the inverted mapping directly saves some work and simplifies the code.

rustbot · 2025-12-18T21:26:59Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

rustbot · 2025-12-18T21:27:01Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

workingjubilee · 2025-12-19T15:34:53Z

Cool!

r? workingjubilee
@bors r+ rollup

( I don't believe this is truly performance sensitive by itself, I only asked the bot because it was truly just idle curiosity and I'm trying to get a sense of where the hotter spots in rustc_abi actually are (since they're effectively only realized at usage sites). )

bors · 2025-12-19T15:34:56Z

📌 Commit b31ee3a has been approved by workingjubilee

It is now in the queue for this repository.

…=workingjubilee layout: Store inverse memory index in `FieldsShape::Arbitrary` All usages of `memory_index` start by calling `invert_bijective_mapping`, so storing the inverted mapping directly saves some work and simplifies the code.

Rollup of 7 pull requests Successful merges: - #149633 (Enable `outline-atomics` by default on AArch64 FreeBSD) - #149788 (Move shared offload globals and define per-kernel globals once) - #149989 (Improve filenames encoding and misc) - #150012 (rustc_target: Add `efiapi` ABI support for LoongArch) - #150116 (layout: Store inverse memory index in `FieldsShape::Arbitrary`) - #150151 (Destabilise `target-spec-json`) - #150159 (Split eii macro expansion code) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 6 pull requests Successful merges: - #149633 (Enable `outline-atomics` by default on AArch64 FreeBSD) - #149788 (Move shared offload globals and define per-kernel globals once) - #149989 (Improve filenames encoding and misc) - #150012 (rustc_target: Add `efiapi` ABI support for LoongArch) - #150116 (layout: Store inverse memory index in `FieldsShape::Arbitrary`) - #150159 (Split eii macro expansion code) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of #150116 - moulins:layout-inv-memory-index, r=workingjubilee layout: Store inverse memory index in `FieldsShape::Arbitrary` All usages of `memory_index` start by calling `invert_bijective_mapping`, so storing the inverted mapping directly saves some work and simplifies the code.

…utine layout This commit fixes a critical bug introduced in PR rust-lang#150116 that caused "offset is not a multiple of 16" errors when compiling for x86_64-unknown-uefi. Problem: -------- PR rust-lang#150116 optimized memory layout by inverting the storage representation from `memory_index` (source→memory) to `in_memory_order` (memory→source). However, in compiler/rustc_abi/src/layout/coroutine.rs (lines 244-276), the code was building `combined_in_memory_order` directly instead of: 1. Building `combined_memory_index` (source→memory) first 2. Inverting it to get `in_memory_order` (memory→source) This caused fields to be placed at incorrect offsets, violating PE/COFF alignment requirements (16-byte multiples), triggering linker errors. Solution: --------- Changed the coroutine layout code to: - Build `combined_memory_index` as an intermediate representation - Use correct indexing: `combined_memory_index[i] = memory_index` - Invert at the end: `combined_in_memory_order = combined_memory_index.invert_bijective_mapping()` Impact: ------- - Fixes UEFI target compilation (x86_64-unknown-uefi) - May affect other PE/COFF targets - Enables SOBEX OS development to continue Files changed: -------------- - compiler/rustc_abi/src/layout/coroutine.rs: Applied fix - BUG_FIX_REPORT.md: Detailed investigation report - fix_pr150116_coroutine_layout.patch: Patch for upstream submission - .gitignore: Added Claude Code settings Upstream: --------- This fix should be submitted to rust-lang/rust as it affects the latest nightly (2025-12-19) and potentially other PE/COFF targets. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Rollup of 6 pull requests Successful merges: - rust-lang/rust#149633 (Enable `outline-atomics` by default on AArch64 FreeBSD) - rust-lang/rust#149788 (Move shared offload globals and define per-kernel globals once) - rust-lang/rust#149989 (Improve filenames encoding and misc) - rust-lang/rust#150012 (rustc_target: Add `efiapi` ABI support for LoongArch) - rust-lang/rust#150116 (layout: Store inverse memory index in `FieldsShape::Arbitrary`) - rust-lang/rust#150159 (Split eii macro expansion code) r? `@ghost` `@rustbot` modify labels: rollup

PR rust-lang#150116 introduced a bug in coroutine variant layout by incorrectly building the `in_memory_order` mapping directly instead of building an intermediate `memory_index` and inverting it. The bug caused fields to be placed at incorrect memory offsets, violating alignment requirements. This manifested as linker errors on PE/COFF targets (e.g., x86_64-unknown-uefi): error: offset is not a multiple of 16 Fix by correctly building `combined_memory_index` (source→memory) as an intermediate representation, then inverting it to get `combined_in_memory_order` (memory→source).

PR rust-lang#150116 introduced a bug in coroutine variant layout by incorrectly keeping the direct construction of `in_memory_order` without adapting it to the new representation. Before PR rust-lang#150116, the code correctly built an inverse mapping indexed by memory position, then inverted it to get the memory_index. After PR rust-lang#150116, the code still built the mapping indexed by memory position (as if it were an inverse mapping), but treated it directly as `in_memory_order` without inversion. This caused incorrect field ordering. The bug manifested as linker errors on PE/COFF targets (e.g., x86_64-unknown-uefi): error: offset is not a multiple of 16 And as ICEs with non-bijective mapping assertions in other cases. Fix by correctly keeping the inverse memory index construction (indexed by memory position), then using it directly as `in_memory_order` since that's what the new representation expects.

Rollup of 6 pull requests Successful merges: - rust-lang/rust#149633 (Enable `outline-atomics` by default on AArch64 FreeBSD) - rust-lang/rust#149788 (Move shared offload globals and define per-kernel globals once) - rust-lang/rust#149989 (Improve filenames encoding and misc) - rust-lang/rust#150012 (rustc_target: Add `efiapi` ABI support for LoongArch) - rust-lang/rust#150116 (layout: Store inverse memory index in `FieldsShape::Arbitrary`) - rust-lang/rust#150159 (Split eii macro expansion code) r? `@ghost` `@rustbot` modify labels: rollup

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 17, 2025

rustbot assigned nnethercote Dec 17, 2025

This comment has been minimized.

Sign in to view

moulins force-pushed the layout-inv-memory-index branch from c3ca7dc to 8db435e Compare December 18, 2025 00:12

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Dec 18, 2025

Auto merge of #150116 - moulins:layout-inv-memory-index, r=<try>

ed52dc8

layout: Store inverse memory index in `FieldsShape::Arbitrary`

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 18, 2025

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 18, 2025

RalfJung reviewed Dec 18, 2025

View reviewed changes

moulins force-pushed the layout-inv-memory-index branch from 8db435e to e0bc269 Compare December 18, 2025 14:11

This comment has been minimized.

Sign in to view

layout: Store inverse memory index in FieldsShape::Arbitrary

b31ee3a

All usages of `memory_index` start by calling `invert_bijective_mapping`, so storing the inverted mapping directly saves some work and simplifies the code.

moulins force-pushed the layout-inv-memory-index branch from e0bc269 to b31ee3a Compare December 18, 2025 21:26

rustbot assigned workingjubilee and unassigned nnethercote Dec 19, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 19, 2025

matthiaskrgr mentioned this pull request Dec 19, 2025

Rollup of 6 pull requests #150167

Closed

matthiaskrgr mentioned this pull request Dec 19, 2025

Rollup of 7 pull requests #150168

Closed

matthiaskrgr mentioned this pull request Dec 19, 2025

Rollup of 6 pull requests #150177

Merged

bors merged commit 22440fd into rust-lang:main Dec 20, 2025
11 checks passed

rustbot added this to the 1.94.0 milestone Dec 20, 2025

moulins deleted the layout-inv-memory-index branch December 20, 2025 13:33

leonardocxavier mentioned this pull request Dec 22, 2025

Fix PR #150116 bug: incorrect memory index inversion in coroutine layout #150266

Closed

leonardocxavier mentioned this pull request Dec 22, 2025

Fix coroutine layout bug from PR #150116 #150275

Closed

github-actions bot mentioned this pull request Dec 23, 2025

Toolchain upgrade to nightly-2026-01-30 flux-rs/flux#1345

Draft

Uh oh!

Conversation

moulins commented Dec 17, 2025

Uh oh!

rustbot commented Dec 17, 2025

Uh oh!

rustbot commented Dec 17, 2025

Uh oh!

This comment has been minimized.

workingjubilee commented Dec 18, 2025

Uh oh!

This comment has been minimized.

workingjubilee commented Dec 18, 2025

Uh oh!

workingjubilee commented Dec 18, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

moulins commented Dec 18, 2025

Uh oh!

workingjubilee commented Dec 18, 2025

Uh oh!

rust-bors bot commented Dec 18, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Dec 18, 2025

Overall result: ✅ improvements - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

RalfJung Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

moulins Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

RalfJung Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

moulins commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workingjubilee commented Dec 18, 2025

Uh oh!

workingjubilee commented Dec 18, 2025

Uh oh!

moulins commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Dec 18, 2025

Uh oh!

rustbot commented Dec 18, 2025

Uh oh!

workingjubilee commented Dec 19, 2025

Uh oh!

bors commented Dec 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

moulins commented Dec 18, 2025 •

edited

Loading

moulins commented Dec 18, 2025 •

edited

Loading