Repository navigation
force-frame-pointers flag isn't honored always unless std is recompiled as such #103711
Description
The compiler flag -C force-frame-pointers=yes isn't always honored, clobbering the %rbp register and resulting in the unrecoverable stackframe for perf --call-graph fp.
It seems that if I compile rustc under -C force-frame-pointers=yes, it works as shown below:
setup
$ cargo new clobbered-frame-pointer && cd clobbered-frame-pointer
pre-built stock binary rustc
$ rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -A50 'std::sys::unix::futex::futex_wait>:'
Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
Finished release [optimized] target(s) in 0.33s
000000000001fe60 <std::sys::unix::futex::futex_wait>:
1fe60: 55 push %rbp
1fe61: 41 57 push %r15
1fe63: 41 56 push %r14
1fe65: 41 55 push %r13
1fe67: 41 54 push %r12
1fe69: 53 push %rbx
1fe6a: 48 83 ec 28 sub $0x28,%rsp
1fe6e: 41 89 f6 mov %esi,%r14d
1fe71: 48 89 fb mov %rdi,%rbx
1fe74: 48 83 3a 00 cmpq $0x0,(%rdx)
1fe78: 74 4d je 1fec7 <std::sys::unix::futex::futex_wait+0x67>
1fe7a: 48 8b 6a 08 mov 0x8(%rdx),%rbp
1fe7e: 44 8b 7a 10 mov 0x10(%rdx),%r15d
1fe82: 48 8d 74 24 10 lea 0x10(%rsp),%rsi
1fe87: bf 01 00 00 00 mov $0x1,%edi
1fe8c: ff 15 a6 cb 02 00 callq *0x2cba6(%rip) # 4ca38 <clock_gettime@GLIBC_2.17>
1fe92: 83 f8 ff cmp $0xffffffff,%eax
1fe95: 0f 84 c1 00 00 00 je 1ff5c <std::sys::unix::futex::futex_wait+0xfc>
1fe9b: 48 8b 44 24 10 mov 0x10(%rsp),%rax
1fea0: 48 01 e8 add %rbp,%rax
1fea3: 0f 90 c2 seto %dl
1fea6: 31 c9 xor %ecx,%ecx
1fea8: 48 85 ed test %rbp,%rbp
1feab: 78 37 js 1fee4 <std::sys::unix::futex::futex_wait+0x84>
1fead: 84 d2 test %dl,%dl
1feaf: 75 33 jne 1fee4 <std::sys::unix::futex::futex_wait+0x84>
1feb1: 48 8b 4c 24 18 mov 0x18(%rsp),%rcx
1feb6: 41 01 cf add %ecx,%r15d
1feb9: 41 81 ff 00 ca 9a 3b cmp $0x3b9aca00,%r15d
1fec0: 72 10 jb 1fed2 <std::sys::unix::futex::futex_wait+0x72>
1fec2: 48 ff c0 inc %rax
1fec5: 71 04 jno 1fecb <std::sys::unix::futex::futex_wait+0x6b>
1fec7: 31 c9 xor %ecx,%ecx
1fec9: eb 19 jmp 1fee4 <std::sys::unix::futex::futex_wait+0x84>
1fecb: 41 81 c7 00 36 65 c4 add $0xc4653600,%r15d
1fed2: 44 89 f9 mov %r15d,%ecx
1fed5: 48 89 44 24 18 mov %rax,0x18(%rsp)
1feda: 48 89 4c 24 20 mov %rcx,0x20(%rsp)
1fedf: b9 01 00 00 00 mov $0x1,%ecx
1fee4: 48 89 4c 24 10 mov %rcx,0x10(%rsp)
1fee9: 4c 8d 7c 24 18 lea 0x18(%rsp),%r15
1feee: 4c 8b 25 a3 cd 02 00 mov 0x2cda3(%rip),%r12 # 4cc98 <syscall@GLIBC_2.2.5>
1fef5: 4c 8b 2d ac ca 02 00 mov 0x2caac(%rip),%r13 # 4c9a8 <__errno_location@GLIBC_2.2.5>
1fefc: 0f 1f 40 00 nopl 0x0(%rax)
1ff00: 8b 03 mov (%rbx),%eax
1ff02: 40 b5 01 mov $0x1,%bpl # !!!! %RBP IS CLOBBERED
1ff05: 44 39 f0 cmp %r14d,%eax
1ff08: 75 41 jne 1ff4b <std::sys::unix::futex::futex_wait+0xeb>
1ff0a: 4c 8b 44 24 10 mov 0x10(%rsp),%r8
1ff0f: 4d 85 c0 test %r8,%r8
re-built rustc under -C force-frame-pointer=rustc
rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo +kustom-for-frame-pointer build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -A50 'std::sys::unix::futex::futex_wait>:'
Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
Finished release [optimized] target(s) in 0.16s
000000000001e480 <std::sys::unix::futex::futex_wait>:
1e480: 55 push %rbp
1e481: 48 89 e5 mov %rsp,%rbp
1e484: 41 57 push %r15
1e486: 41 56 push %r14
1e488: 41 55 push %r13
1e48a: 41 54 push %r12
1e48c: 53 push %rbx
1e48d: 48 83 ec 28 sub $0x28,%rsp
1e491: 48 83 3a 00 cmpq $0x0,(%rdx)
1e495: 41 89 f6 mov %esi,%r14d
1e498: 49 89 ff mov %rdi,%r15
1e49b: 74 48 je 1e4e5 <std::sys::unix::futex::futex_wait+0x65>
1e49d: 44 8b 62 10 mov 0x10(%rdx),%r12d
1e4a1: 48 8b 5a 08 mov 0x8(%rdx),%rbx
1e4a5: 48 8d 75 c0 lea -0x40(%rbp),%rsi
1e4a9: bf 01 00 00 00 mov $0x1,%edi
1e4ae: ff 15 b4 25 03 00 callq *0x325b4(%rip) # 50a68 <clock_gettime@GLIBC_2.17>
1e4b4: 83 f8 ff cmp $0xffffffff,%eax
1e4b7: 0f 84 c2 00 00 00 je 1e57f <std::sys::unix::futex::futex_wait+0xff>
1e4bd: 48 8b 45 c0 mov -0x40(%rbp),%rax
1e4c1: 48 01 d8 add %rbx,%rax
1e4c4: 0f 90 c1 seto %cl
1e4c7: 48 85 db test %rbx,%rbx
1e4ca: 78 19 js 1e4e5 <std::sys::unix::futex::futex_wait+0x65>
1e4cc: 84 c9 test %cl,%cl
1e4ce: 75 15 jne 1e4e5 <std::sys::unix::futex::futex_wait+0x65>
1e4d0: 48 8b 4d c8 mov -0x38(%rbp),%rcx
1e4d4: 41 01 cc add %ecx,%r12d
1e4d7: 41 81 fc 00 ca 9a 3b cmp $0x3b9aca00,%r12d
1e4de: 72 16 jb 1e4f6 <std::sys::unix::futex::futex_wait+0x76>
1e4e0: 48 ff c0 inc %rax
1e4e3: 71 0a jno 1e4ef <std::sys::unix::futex::futex_wait+0x6f>
1e4e5: 48 c7 45 c0 00 00 00 movq $0x0,-0x40(%rbp)
1e4ec: 00
1e4ed: eb 1a jmp 1e509 <std::sys::unix::futex::futex_wait+0x89>
1e4ef: 41 81 c4 00 36 65 c4 add $0xc4653600,%r12d
1e4f6: 44 89 e1 mov %r12d,%ecx
1e4f9: 48 89 45 c8 mov %rax,-0x38(%rbp)
1e4fd: 48 89 4d d0 mov %rcx,-0x30(%rbp)
1e501: 48 c7 45 c0 01 00 00 movq $0x1,-0x40(%rbp)
1e508: 00
1e509: 4c 8b 2d 70 27 03 00 mov 0x32770(%rip),%r13 # 50c80 <syscall@GLIBC_2.2.5>
1e510: 4c 8b 25 e9 28 03 00 mov 0x328e9(%rip),%r12 # 50e00 <__errno_location@GLIBC_2.2.5>
1e517: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
1e51e: 00 00
1e520: 41 8b 07 mov (%r15),%eax
1e523: b3 01 mov $0x1,%bl # OKAY!
1e525: 44 39 f0 cmp %r14d,%eax
1e528: 75 44 jne 1e56e <std::sys::unix::futex::futex_wait+0xee>
1e52a: 4c 8b 45 c0 mov -0x40(%rbp),%r8
I picked std::sys::unix::futex::futex_wait for the above example. But it seems that there's many:
$ rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -E '>:$|\$0x1,%bpl' | grep -B1 bpl | head -n 50
Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
Finished release [optimized] target(s) in 0.33s
000000000000f7a0 <addr2line::ResUnit<R>::parse_lines>:
11d14: 40 b5 01 mov $0x1,%bpl
11d47: 40 b5 01 mov $0x1,%bpl
11d5c: 40 b5 01 mov $0x1,%bpl
11d6c: 40 b5 01 mov $0x1,%bpl
11d7e: 40 b5 01 mov $0x1,%bpl
--
00000000000121e0 <addr2line::ResDwarf<R>::parse>:
1469d: 40 b5 01 mov $0x1,%bpl
16e6b: 40 b5 01 mov $0x1,%bpl
16e7d: 40 b5 01 mov $0x1,%bpl
--
000000000001dcc0 <std::panicking::default_hook>:
1dfc6: 40 80 f5 01 xor $0x1,%bpl
1e064: 40 b5 01 mov $0x1,%bpl
1e076: 40 b5 01 mov $0x1,%bpl
1e0a6: 40 b5 01 mov $0x1,%bpl
--
000000000001fe60 <std::sys::unix::futex::futex_wait>:
1ff02: 40 b5 01 mov $0x1,%bpl
--
0000000000020e30 <std::backtrace_rs::symbolize::gimli::Context::new>:
218a4: 40 b5 01 mov $0x1,%bpl
219a7: 40 b5 01 mov $0x1,%bpl
21d71: 40 b5 01 mov $0x1,%bpl
--
0000000000022020 <std::backtrace_rs::symbolize::gimli::resolve::{{closure}}>:
22d20: 40 80 fd 01 cmp $0x1,%bpl
240b1: 40 80 fd 01 cmp $0x1,%bpl
253bf: 40 b5 01 mov $0x1,%bpl
257fc: 40 f6 c5 01 test $0x1,%bpl
266ed: 40 b5 01 mov $0x1,%bpl
267d2: 40 b5 01 mov $0x1,%bpl
--
0000000000029a50 <gimli::read::abbrev::Abbreviations::insert>:
29b1b: 40 80 fd 01 cmp $0x1,%bpl
--
000000000002ce30 <<rustc_demangle::v0::Ident as core::fmt::Display>::fmt>:
2d243: 40 b5 01 mov $0x1,%bpl
2d2f3: 40 b5 01 mov $0x1,%bpl
--
000000000002da20 <rustc_demangle::v0::Printer::print_backref>:
2da99: 40 b5 01 mov $0x1,%bpl
000000000002dbe0 <rustc_demangle::v0::Printer::print_backref>:
2dc59: 40 b5 01 mov $0x1,%bpl
000000000002dda0 <rustc_demangle::v0::Printer::print_backref>:
2de19: 40 b5 01 mov $0x1,%bpl
--
000000000002f9d0 <rustc_demangle::v0::Printer::print_type::{{closure}}>:
2facd: 40 b5 01 mov $0x1,%bpl
I suspect this is caused by pre-built fn machine code is statically linked into the binary, skipping compilation and not considering RUSTFLAGS.
$ rustc --version --verbose
rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52
commit-date: 2022-09-19
host: x86_64-unknown-linux-gnu
release: 1.64.0
LLVM version: 14.0.6
Reacted by Ivan Molodetskikh
Reacted by tlambertz
Activity
[-]`force-frame-pointers` flag isn't honored always unless rustc is recompiled[/-][+]`force-frame-pointers` flag isn't honored always unless rustc is recompiled as such[/+]the8472 commentedon Oct 29, 2022
That's probably prebuilt code shipped with the standard library, have you tried using
-Zbuild-std(requires nightly)?ryoqun commentedon Oct 29, 2022
hey, thanks for the tip. yep, i confirmed
-Zbuild-stdfixes the problem.ryoqun commentedon Oct 29, 2022
so, this is a known issue? hopefully, a warn or something can be printed without being too noisy?
the8472 commentedon Oct 29, 2022
Since this also applies to other codegen options (e.g. optimization levels) almost everyone setting RUSTFLAGS would get that warning. Maybe it could be documented in the cargo book.
alatiera commentedon Feb 15, 2023
What would be the appropriate place to raise this issue regarding building the precompiled rustc binaries that rustup distributes with frame-pointers enabled?
workingjubilee commentedon Jul 15, 2023
I am assuming the final comment by @alatiera is essentially the "real issue" here and leaving
needs-triagebecause it's not clear to me what the answer is, however it is of some concern to the compiler and library team, since it has to do with how std is built and shipped. Unless-Zbuild-stdis accepted as the canonical option, of course, but it doesn't seem like it would be clear to users why they would need to rebuildstdto support frame pointers, rather than that being done implicitly.alatiera commentedon Jul 17, 2023
I guess there are 2 or maybe 3 different issues here probably:
force-frame-pointersand rebuilding the std if the existing install doesn't matchAll kinda related, but also can be tackled independently I think
saethlin commentedon Aug 1, 2023
I think there are 3 ways to address this question, with varying tradeoffs:
Enable frame pointers by default in distributed stdlib. This may have overhead, but we have some benchmarking so we should be able to provide some answer. This is by far the easiest option: Enable frame pointers by default #114323
Distribute the standard library as MIR-only rlibs, which is a theoretical idea that nobody has implemented. The now-dead tracking issue was Tracking issue for MIR-only RLIBs #38913 I am interested in this for other reasons, so I might make progress here, but pulling this off would probably be very messy. For example, just trying to blindly enable
-Zalways-encode-mirproduces multiple definition errors when bootstrapping, and undefined symbol errors when combined with-Zbuild-std.Make
-Zbuild-stdwork on a stable compiler. This effort has its own repo for tracking issues: https://github.com/rust-lang/wg-cargo-std-aware/issues as well as a tag here: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3A-Zbuild-std. There seems to be some activity on that repo this year which is nice, but I'm not holding my breath. There's a lot of work to do still.alatiera commentedon Aug 2, 2023
Enabling frame-pointers I think it's the easiest option and developers will thank you for it. There is gonna be backlash from all the people with an opinion on the internet but I think the issue is pretty clear for any person that ever profiled an application. CPython is also making frame-pointers mandatory for 3.12 last I heard.
workingjubilee commentedon Sep 9, 2023
Even with still allowing frame pointer elision for leaf functions, it induced a pretty hard compiler performance hit (test results in #115521 (comment) if you're wondering). I'm not sure if I could sell the compiler performance WG on it, much less the entire Rust community, despite frame pointers usually helping profiling.
scottmcm commentedon Nov 18, 2023
Hmm, this is definitely a place where using instructions is sub-optimal, though, because it'll clearly be more instructions -- it has to be, to maintain them -- but that means the "they're a fine proxy for speed because it'll be about the same run-to-run" argument for looking at them most of the time doesn't apply.
If you instead look at its wall-time results and include everything (not just the regressions/improvements), then it's:
That's not nothing, but I think it's a better summary than the "+1.5%" that the instruction-count-changes summary shows. I still don't know that it's small enough to say "yeah, of course just take it", but it doesn't seem entirely unreasonable either.
Maybe we could find the functions where most of the impact is coming from and
#[inline]them so they'll be compiled in the caller's context -- without frame pointers for the people who don't want them -- and then it'd be fine to have them in the rest of the places?Like having frame pointers in
panicking::default_hookall the time seems fine, since anything where that's the perf bottleneck has bigger problems.graydon commentedon Dec 14, 2023
(Just found my way here through random thread-hopping and want to add whatever persuasive/rhetorical weight I can: having frame pointers everywhere by default is so, so worth the sprinkling of instructions it costs. Please do try to turn them on!)
tiberiusferreira commentedon Jan 8, 2024
Hey, I also recently got hit by this issue while using https://github.com/tikv/pprof-rs on MacOS because of tikv/pprof-rs#131 (comment).
I also want to point out that increased ability to profile code will unlock much greater optimizations than the raw perf loss, specially at ~1% or less. It will also get closer to perf tools "just working".
saethlin commentedon Jan 8, 2024
The perf numbers being discussed here are about the performance of the compiler, not of user code. When hacking on the compiler we don't use a precompiled standard library, so limitations like this issue which are caused by the build configuration used for the precompiled standard library do not affect the compiler dev workflow.
I agree with the plain "the small icount regression is worth the improved behavior" argument, but not "we'll claw back that 1% via optimizations people find because profiling is easier".
jonhoo commentedon Mar 17, 2024
Worth pointing out that Fedora and Ubuntu are switching to having frame pointers everywhere as well. There's a good write-up about the what and why here: https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
I wonder if we could update the Rust default to simply always enable them?
[-]`force-frame-pointers` flag isn't honored always unless rustc is recompiled as such[/-][+]`force-frame-pointers` flag isn't honored always unless std is recompiled as such[/+]workingjubilee commentedon Mar 18, 2024
As-of #122646 we now have frame pointers enabled in the stdlib! This does not "enable frame pointers by default", but it resolves the primary concern here and the one the issue title is actually about. For those wanting to recommend we enable frame pointers by default, it may be best to propose it to the compiler team and perhaps first get answers to some questions like "wait, doesn't the aarch64 call ABI use frame-pointers anyways? so the perf regressions, such as they are, don't necessarily materialize on all platforms? has anyone checked the assembly to verify this?"