Implement parameter validation for functions using x86-interrupt abi.

[Siech0]

Unsized parameters or zero sized parameters provided to an function with the x86-interrupt abi could emit ICE. Such parameters are conceptually invalid. This commit implements parameter validation logic for the x86-iterrupt ABI to prevent these error classes and to improve user diagnostics.

Relevant unstable feature: #40180
Fixes #124806
Fixes #126418
(May also close this, a reviewer's input would be optimal: #63018. It depends on what the desired resolution is, because current rules don't bound the size requirements of frame though they could be trivially added. The bug itself seems to be resolved at this point.)

---

LLM assistance disclosure: An LLM was used in the development of this PR, specifically for the following tasks:

• Querying about the repository's structure, common types, and existing patterns
• Preliminary code-review and style checks

There exists no LLM generated code, comments, documentation, or tests in this PR.

[rustbot]

HIR ty lowering was modified

cc @fmease

[rustbot]

Thanks for the pull request, and welcome! The Rust Project has assigned @TaKO8Ki (or someone else) to review your changes, you should hear from them (or someone else) within the next two weeks.

Please see the contribution instructions and our LLM policy for more information.

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 75 candidates
• Random selection from 20 candidates

[Siech0]

Converting to draft, while it passed unit tests locally I was able to cause an issue with the validator when running against code in the wild. I will reopen once resolved.

Specifically, this case fails:

struct Addr(u64);

#[repr(C)]
struct ComplexFrame { ip: Addr, cs: u64, flags: u64, sp: Addr, ss: u64 }

struct Idt { function_pointer_member_should_work: extern "x86-interrupt" fn(ComplexFrame) }

I'm adding it to my tests now as a regression test, and I'm going to try to figure out why this is happening. Getting this error:
ERROR: cycle detected when computing the inferred outlives-clauses for items in this crate [E0391]

My original design was based on the cmse.rs impl inside of hir_ty_lowering, I'm rebuilding my LLVM to see if I can find any odd behaviors with their implementation too.

[Siech0]

With a fairly minimal refactor I moved it into check (I'm still not confident on where to integrate it, so advice would be welcome) and the cycle error is resolved against my tests, however, the validation no longer runs against function pointers.

I'm still a bit uphappy about that, but, for now, since I after running more tests on existing code (including the x86_64 crate) I found no regressions, I'm reopening.

[Siech0]

A quick reference for what shapes GCC/Clang/rustc allows / rejects (pre / post changes)

Charts

Assume 64-bits except if noted otherwise.

Rust types used:

#[repr(C)] pub struct Frame { ip: u64, cs: u64, flags: u64, sp: u64, ss: u64 }
#[repr(C)] pub struct FrameBool { ip: u64, b: bool }
#[repr(transparent)] pub struct Newtype(u64);
#[repr(C)] pub struct NonScalar { a: u8, b: u32 }
#[repr(u64)] pub enum Ec { A }
#[repr(u32)] pub enum EcInt { B }
pub enum Opt<T> { None, Some(T) }

C types used:

typedef unsigned long uword_t;
struct frame { uword_t ip, cs, flags, sp, ss; };
struct empty {};
struct nt { uword_t x; };
struct ns { unsigned char a; unsigned int b; };
struct fb { uword_t ip; _Bool b; };
enum ec : uword_t { EC_A };
enum ec_int { EC_B };
#define ISR __attribute__((interrupt))

Frame parameter

C signature	Rust signature	Clang	GCC	rustc nightly (before)	rustc + PR
void ISR f(void *fr)	_: *const u8	ok	ok	wrong code	reject
void ISR f(struct frame *fr)	_: &Frame	ok	ok	wrong code	reject
-	_: *const [u8]	n/a	n/a	wrong code	reject
void ISR f(struct frame *fr)	_: Opt<&Frame>	ok	ok	wrong code	reject
void ISR f(struct frame fr)	_: Frame	reject	reject	ok	ok
void ISR f(unsigned char fr)	_: u8	reject	reject	ok	ok
void ISR f(uword_t fr)	_: u64	reject	reject	ok	ok
void ISR f(_Bool fr)	_: bool	reject	reject	accepted	reject
void ISR f(struct fb fr)	_: FrameBool	reject	reject	accepted	reject
void ISR f(struct empty fr)	_: ()	reject	reject	ICE	reject
void ISR f(void)	(none)	reject	reject	reject	reject

Error code parameter

C signature	Rust signature	Clang	GCC	rustc nightly (before)	rustc + PR
void ISR f(struct frame *fr, uword_t c)	_: Frame, _: u64	ok	ok	ok	ok
void ISR f(struct frame *fr, unsigned int c)	_: Frame, _: u32	reject	reject	accepted	reject
void ISR f(struct frame *fr, long c)	_: Frame, _: i64	reject	ok	ok	ok
void ISR f(struct frame *fr, unsigned char c)	_: Frame, _: u8	reject	reject	accepted	reject
void ISR f(struct frame *fr, unsigned short c)	_: Frame, _: u16	reject	reject	accepted	reject
void ISR f(struct frame *fr, __uint128_t c)	_: Frame, _: u128	reject	reject	LLVM fatal	reject
void ISR f(struct frame *fr, _Bool c)	_: Frame, _: bool	reject	reject	accepted	reject
void ISR f(struct frame *fr, enum ec c)	_: Frame, _: Ec	ok	reject	accepted	reject
void ISR f(struct frame *fr, enum ec_int c)	_: Frame, _: EcInt	reject	reject	accepted	reject
void ISR f(struct frame *fr, struct nt c)	_: Frame, _: Newtype	reject	reject	ok	ok
void ISR f(struct frame *fr, struct ns c)	_: Frame, _: NonScalar	reject	reject	LLVM fatal	reject
void ISR f(struct frame *fr, unsigned char c[3])	_: Frame, _: [u8; 3]	reject	reject	wrong code	reject
void ISR f(struct frame *fr, void *c)	_: Frame, _: *const u8	reject	reject	accepted	reject
void ISR f(struct frame *fr, struct empty c)	_: Frame, _: ()	reject	reject	wrong code	reject
void ISR f(struct frame *fr, double c)	_: Frame, _: f64	reject	reject	accepted	reject

Whole-signature shape

C signature	Rust signature	Clang	GCC	rustc nightly (before)	rustc + PR
void ISR f(struct frame *fr, uword_t c, uword_t d)	_: Frame, _: u64, _: u64	reject	reject	reject	reject
void ISR f(struct frame *fr, ...)	_: Frame, _: ...	ok	ok	reject	reject
int ISR f(struct frame *fr)	_: Frame) -> u8	reject	reject	reject	reject

32-bit variants

C signature	Rust signature	Clang	GCC	rustc nightly (before)	rustc + PR
void ISR f(struct frame fr)	_: Frame	reject	reject	ok	ok
void ISR f(struct frame *fr, unsigned int c)	_: Frame, _: u32	ok	ok	ok	ok
void ISR f(struct frame *fr, unsigned long long c)	_: Frame, _: u64	reject	reject	LLVM fatal	reject
void ISR f(struct frame *fr, unsigned short c)	_: Frame, _: u16	reject	reject	accepted	reject

---

Notes

Most significant things I note is:

• Error code table line 12, produced invalid code but emits an FFI safety warning anyway. Change promotes to rejection.
• two instances where esotetic inputs could generate invalid assembly

Wrong Code (Array)

arr_code:
        push    rax
        push    rax
        push    rcx
        mov     rax, qword ptr [rsp + 24]  ; load the error into rax
        movzx   ecx, byte ptr [rax]            ; ERR: use the error code as address, read c[0] from it
        add     rcx, qword ptr [rsp + 32]   
        movzx   eax, byte ptr [rax + 2]     ; BAD: same address plus 2, read c[2] from it
        add     rax, rcx
        mov     rcx, qword ptr [rip + SINK@GOTPCREL]
        mov     qword ptr [rcx], rax
        pop     rcx
        pop     rax
        add     rsp, 16
        iretq

Basically, error code resides at [rsp + 24], we load it as a pointer into rax and then dereference it twice for accesses into c[0] and c[2]. let err_code == 2, we then dereference 0x2, causing a fault.

Wrong code (ZST Code)

no_code: ; zst code aliases this
        push    rax
        push    rcx
        mov     rax, qword ptr [rsp + 16] ; code thinks frame.ip (2*8 + 0)
                                                            ; however, if installed on a vector that uses code
                                                            ; we would actually be pointing at code rather than frame.ip.                        
        mov     rcx, qword ptr [rip + SINK@GOTPCREL]
        mov     qword ptr [rcx], rax
        pop     rcx
        pop     rax
        ; We don't add rsp, 16 here, so we don't drop the error code before iretq
        ; meaning the iretq here will actually use the error code instead of frame.ip
        iretq
        
zst_code = no_code

If we installed zst_code as a vector for a function that expected a code, the handler would fail because the CPU would still allocate the code onto the stack, but our code would still believe that rsp + 16 is the frame.ip, though it would actually be the inserted code. Additionally, we're missing add rsp, 16 before iretq, so iretq` pops the error code as frame.ip.

[Siech0]

I've implemented generics support and validated against most patterns. Not sure I got all patterns possible since the breadth of possible generic structures is quite high, but I tried to be thorough.

[Siech0]

Did some research on this (and nightly) against pointers. I assumed they were valid frame parameters because abi tests elsewhere used them as the reference and because the underlying LLVM abi itself in fact provides you a pointer, however, I don't believe this is actually the case. I believe the pointer is essentially being lowered as [i8;8] so the parameter is the actual value of the frame rather than the address of it (meaning the pointer itself is invalid).

I've gone and updated my code to check for pointers now, and also did a retest of my build against community crates & projects. No code I tested seems to be broken by this design, which makes sense because if they used pointers as a frame param it would have broken. (Meanwhile, every single test we wrong seems to use *const u8, which is invalid and painful to update.)

[Siech0]

Note: I've considered adding a rule for restricting Frame size, but I chose not to because the frame size is actually variable on 32-bit, and can be between 12-36 bytes. The C abi's avoid having to care about this because they all operate on pointers, so I think we should also leave this to largely the user's discretion.

In contrast, llvm & gcc are both strict on what the code parameter may be, so that one I believe we can be strict on size requirements for.

View changes since the review