release builds using rustc 1.86.0 on macOS Ventura (intel) SDK exhibit incorrect behaviour

[spikegrobstein]

I’d like to report what appears to be a bug in the rust compiler which can be reproduced by using the reqwest crate, where it fails to include the http version string in the request and it only affects binaries that were:

• built targeting x86_64-apple-darwin
• using release mode
• using a macOS Ventura (13.x) SDK

The bug itself has nothing to do with the reqwest crate (looking at the code, it appears that this behaviour should be impossible) and specifically related to the built binary.

this bug does exist in release binaries cross-compiled from linux to macOS when using the macOS 13 (Ventura) SDK with osxcross.

the bug does not exist when creating a debug build or when cross-compiling from an AppleSilicon machine → intel and does not exist when performing a release build on intel → intel when the OS is not Ventura.

We tracked this down to being triggered by opt-level = 2

this bug does not exist in rustc 1.85.0 and appears to only affect 1.86.0 (and also occurs in nightly rust).

I have a reduction:

add reqwest with --features=blocking and this main.rs:

use reqwest::{blocking::Client, Method};

fn main() {
    Client::new().request(Method::GET, "http://localhost:8888/hello").send().unwrap();
}

in another window, start a nc server with: nc -l localhost 8888

then cargo run and you should see:

GET /hello HTTP/1.1
accept: */*
host: localhost:8888

If you then start a new nc server and cargo run --release you will see:

GET /hello
accept: */*
host: localhost:8888

(note the missing HTTP/1.1 on the first line)

when running this code in a debugger, I can see that the slice that’s supposed to add the version number to the request buffer has a length of 0, so it contains no data:

Process 79729 stopped
* thread #2, name = 'reqwest-internal-sync-runtime', stop reason = step in
    frame #0: 0x00000001000b380f reqwest-test`_$LT$hyper..proto..h1..role..Client$u20$as$u20$hyper..proto..h1..Http1Transaction$GT$::encode::h33cc17a167a456d1 at spec_extend.rs:61:18 [opt]
   58       #[track_caller]
   59       fn spec_extend(&mut self, iterator: slice::Iter<'a, T>) {
   60           let slice = iterator.as_slice();
-> 61           unsafe { self.append_elements(slice) };
   62       }
   63   }
Target 0: (reqwest-test) stopped.
(lldb) p self
(alloc::vec::Vec<unsigned char, alloc::alloc::Global> *) $0 = 0x0000000100504b90
(lldb) p self[0]
(alloc::vec::Vec<unsigned char, alloc::alloc::Global>) $1 = size=11 {
  [0] = 'G'
  [1] = 'E'
  [2] = 'T'
  [3] = ' '
  [4] = '/'
  [5] = 'h'
  [6] = 'e'
  [7] = 'l'
  [8] = 'l'
  [9] = 'o'
  [10] = ' '
}
(lldb) p slice
(*const [u8]) $2 = {
  data_ptr = 0x0000000000000000
  length = 0
}

a note from someone on my team who continued to look into it more deeply:

“it looks like it's computing the correct string HTTP/1.1 but for some odd reason it's computing a length of 0. as best I can tell it uses one lookup table of string pointers to find the string, and it uses another lookup table of 4-byte offsets that get added to the lookup table's base address to produce a new pointer, which looks like it's supposed to be the end of the string. Unfortunately that second lookup table has 3 identical values in it, meaning it will produce the correct end pointer for HTTP/1.0 but it produces the start pointer for HTTP/1.1, and so it ends up calculating a length of 0”

This always happens no matter what version of reqwest is used, and it seems to be caused by the rustc version.

I hope this is enough information.

[moxian]

Can you try running cargo bisect-rustc to narrow down what triggered the change? The command line would probably look like cargo bisect-rustc --start 1.85.0 --end 1.86.0 --prompt -- run

Edit: .. ah, you might need to replace the -- run with a more involved --script setup (I missed the fact that the issue requires cross-compilation)..

[jieyouxu]

this bug does exist in release binaries cross-compiled from linux to macOS when using the macOS 13 (Ventura) SDK with osxcross.

We may have to rely on your bisection to pinpoint a commit (range), because that setup is quite complicated...

[jieyouxu]

To minimize the variables:

• Another thing to try: does this also reproduce in the latest beta (I'd assume so)?
• Can you provide the exact reqwest version?
  • And transitively, which http version?

[spikegrobstein]

@moxian Sorry, I should have been more clear; this isn't only on cross compile. it happens if the macOS 13 SDK is used with the x86_64-apple-darwin target. both native and cross compiled.

I ran the bisect script and came up with this being offending commit: d88ffcd

output:

searched nightlies: from nightly-2025-01-03 to nightly-2025-02-15
regressed nightly: nightly-2025-02-15
searched commit range: a567209...d8810e3
regressed commit: d88ffcd

bisected with cargo-bisect-rustc v0.6.9

Host triple: x86_64-apple-darwin
Reproduce with:

cargo bisect-rustc --start 1.85.0 --end 1.86.0 --prompt -- run --release

@jieyouxu

confirmed that this bug occurs in the beta, as well.

I ran this with the following versions of reqwest and it occurred on each: 0.12.3, 0.12.5, 0.12.15

http crate version 1.3.1 is the one in my reduction project.

[jieyouxu]

cc @scottmcm