OS-7059

Users testing the latest 1.27 rustlang on SmartOS reported that it would abort.
Installing it via pkgsrc trunk (in a 18.1 zone) confirms this:

[root@8a309eb2-4315-6521-acb2-ceb9a72cfd97 ~]# rustc
thread 'main' panicked at 'failed to allocate a guard page', libstd/sys/unix/thread.rs:337:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.
[root@8a309eb2-4315-6521-acb2-ceb9a72cfd97 ~]# RUST_BACKTRACE=full rustc
thread 'main' panicked at 'failed to allocate a guard page', libstd/sys/unix/thread.rs:337:17
stack backtrace:
   0: 0xfffffd7fde402f3e - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::ha22b267de1618f10
   1: 0xfffffd7fde3cdee6 - std::sys_common::backtrace::print::h9be6459ca953ea86
   2: 0xfffffd7fde410a1b - std::panicking::default_hook::{{closure}}::h11d519068135b02a
   3: 0xfffffd7fde4106e1 - std::panicking::default_hook::he7d6fa94ad994305
   4: 0xfffffd7fde41110c - std::panicking::rust_panic_with_hook::hef4aa8f9edd49f7f
   5: 0xfffffd7fde410ef6 - std::panicking::begin_panic::h350ed1a97f2676c6
   6: 0xfffffd7fde3e7b24 - std::sys::unix::thread::guard::init::h66e847be7b85328f
   7: 0xfffffd7fde3e44fd - std::rt::update_stack_guard::h3749f1240dd6decc
   8: 0xfffffd7fde786185 - rustc_driver::run::hf99f9e1bb314cfb8
   9: 0xfffffd7fde7954da - rustc_driver::main::he45a01066d380276
  10:           0x401842 - std::rt::lang_start::{{closure}}::h50ce2fc38c241613
  11: 0xfffffd7fde410be2 - std::panicking::try::do_call::h260c23fa696aee03
  12: 0xfffffd7fde417919 - __rust_maybe_catch_panic
  13: 0xfffffd7fde3e4325 - std::rt::lang_start_internal::hef1606e7d03cbe2a
  14:           0x4018a3 - main
  15:           0x40167b - _start

Recalling that our mitigation for stack-clash in OS-6323 also employs a stack guard, let's trace rustc around the time of the abort:

getrlimit(RLIMIT_STACK, 0xFFFFFD7FFFDFF8D0)     = 0
        cur = 10485760  max = RLIM64_INFINITY
mprotect(0xFFFFFD7FFF400000, 4096, PROT_READ|PROT_WRITE) = 0
setrlimit(RLIMIT_STACK, 0xFFFFFD7FFFDFF8D0)     = 0
        cur = 16777216  max = RLIM64_INFINITY
mmap(0xFFFFFD7FFEE00000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANON, 4294967295, 0) Err#12 ENOMEM

It appears to be querying the max stack size and then likely attempts to map a page immediately following the would-be max top of the stack.

Tracing the related function from OS-6323 confirms this:

seghole_unmap({base:fffffd7fef400000 len:10000000}, fffffd7ffee00000, 1000) = 22
        p_usrstack:fffffd7fffe00000
        p_stksize:3000
        p_stk_ctl:1000000
        p_stkg_start:fffffd7fef400000
        p_stkg_end:fffffd7fff400000

              genunix`as_unmap+0x1b0
              genunix`choose_addr+0xd0
              genunix`zmap+0xfe
              genunix`smmap_common+0x35a
              genunix`smmap64+0xa1
              unix`sys_syscall+0x19f

seghole_unmap explicitly forbids unmap operations which would split the mapping into two. Either the rust runtime must exclude its own userspace stackguard usage on illumos platforms, or seghole_unmap must be made more lenient to cases like this.

rust runtime runs afoul of stack guard

Description

Comments (1)