can't call (diverging) foreign function: __rust_alloc_error_handler

[RalfJung]

We don't support whatever magic links in the alloc error handler, it seems:

#![feature(alloc_error_hook, allocator_api)]

use std::alloc::*;
use std::ptr::NonNull;

struct BadAlloc;

unsafe impl Allocator for BadAlloc {
    fn allocate(&self, _l: Layout) -> Result<NonNull<[u8]>, AllocError> {
        Err(AllocError)
    }
    
    unsafe fn deallocate(&self, _ptr: NonNull<u8>, _layout: Layout) {
        unreachable!();
    }
}

fn main() {
    set_alloc_error_hook(|_| panic!("alloc error"));
    let _b = Box::new_in(0, BadAlloc);
}

error: unsupported operation: can't call (diverging) foreign function: __rust_alloc_error_handler
   --> /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:385:13
    |
385 |             __rust_alloc_error_handler(layout.size(), layout.align());
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't call (diverging) foreign function: __rust_alloc_error_handler
    |
    = help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support
    = note: BACKTRACE:
    = note: inside `std::alloc::handle_alloc_error::rt_error` at /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:385:13: 385:70
    = note: inside `std::alloc::handle_alloc_error` at /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:391:9: 391:75
    = note: inside `std::boxed::Box::<i32, BadAlloc>::new_uninit_in` at /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:451:23: 451:49
    = note: inside `std::boxed::Box::<i32, BadAlloc>::new_in` at /playground/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:383:25: 383:51
note: inside `main`
   --> src/main.rs:20:14
    |
20  |     let _b = Box::new_in(0, BadAlloc);
    |              ^^^^^^^^^^^^^^^^^^^^^^^^

[bjorn3]

__rust_alloc_error_handler will need similar handling to __rust_alloc to call the function returned by rustc_ast::expand::allocator::alloc_error_handler_name(tcx.alloc_error_handler_kind(()).unwrap()).

https://github.com/rust-lang/rust/blob/029cb1b13b6388b95e64e8996ec8b41a9f3cf16e/compiler/rustc_codegen_llvm/src/allocator.rs#L64-L74

[workingjubilee]

Does Miri support defining a finite heap?

[RalfJung]

No. Evaluation will just abort when Miri fails to allocate memory on the host.

[RalfJung]

__rust_alloc_error_handler will need similar handling to __rust_alloc to call the function returned by rustc_ast::expand::allocator::alloc_error_handler_name(tcx.alloc_error_handler_kind(()).unwrap()).

There's a strange difference though; for the allocator, the attribute directly generates symbols with the name __rust_alloc but for the handler there's an indirection and it generates __rg_oom instead.