Linking to private symbols from frameworks between compilation units

This used to work on Rust nightly-2021-11-24:

// Cargo.toml
[package]
name = "foo"
version = "0.1.0"
edition = "2021"

// lib.rs
#[link(name = "AppKit", kind = "framework")]
extern "C" {}

// main.rs
use foo as _;

#[link(name = "CoreGraphics", kind = "framework")]
extern "C" {}

extern "C" {
    fn CGDisplayCreateUUIDFromDisplayID(display: u32) -> *const std::os::raw::c_void;
}

fn main() {
    unsafe { CGDisplayCreateUUIDFromDisplayID(0) };
}

CGDisplayCreateUUIDFromDisplayID is present in the ColorSync framework (on macOS +10.13; on lower versions, in the CoreGraphics framework). The AppKit framework uses ColorSync internally, so the symbol is available through that (I think).

Linking like this worked previously, but since Rust nightly-2021-11-26 it started failing.

This is affecting winit's CI.

Meta

OS: macOS Mojave 10.14.6.

rustc --version --verbose:

rustc 1.58.0-nightly (dd549dcab 2021-11-25)
binary: rustc
commit-hash: dd549dcab404ec4c7d07b5a83aca5bdd7171138f
commit-date: 2021-11-25
host: x86_64-apple-darwin
release: 1.58.0-nightly
LLVM version: 13.0.0

Backtrace

   Compiling link_error v0.1.0 (./link_error)
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "-arch" "x86_64" "./link_error/target/debug/deps/link_error-df545b853490a663.1684psdcp1c8t8w.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.3mg4gkj495esxxn3.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.4lqpqqd6m4wsa5ac.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.4q6v301re75li0ws.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.4spajtpt70opf5uj.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.5gpk52ejgwv2r4xx.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.rk5agyyp0moj2ik.rcgu.o" "./link_error/target/debug/deps/link_error-df545b853490a663.aueco4pi09j3onj.rcgu.o" "-L" "./link_error/target/debug/deps" "-L" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "-framework" "CoreGraphics" "./link_error/target/debug/deps/liblink_error-9c8bbc70359362ee.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-12c85425d8bdabd2.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-c69df909f9b81081.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libobject-f0c16355db7307b7.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libmemchr-d69ae56aea540e36.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-3fd4756c45c77fb7.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libgimli-1d2aab3cb0cd976a.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-1491074c150e2040.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-5a7d603578c76924.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-a5bcb345953a0f7b.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-3ffc0cbe23a29292.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-ee442041a1b0c90d.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-e0c705be41bf34ff.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-23f0461cd27b5763.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-7db0f39aca3e5046.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-7454d006639b0b9c.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-07b1a016408f5808.rlib" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-852eb1ab762cbb0b.rlib" "-framework" "AppKit" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "~/.rustup/toolchains/nightly-2021-11-26-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "-o" "./link_error/target/debug/deps/link_error-df545b853490a663" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: Undefined symbols for architecture x86_64:
            "_CGDisplayCreateUUIDFromDisplayID", referenced from:
                link_error::main::h68dbd3e6b69f3cec in link_error-df545b853490a663.5gpk52ejgwv2r4xx.rcgu.o
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: could not compile `link_error` due to previous error

Sorry if the issue is not very well explained, I'm very inexperienced with how linking works

Fix CGDisplayCreateUUIDFromDisplayID linking (again)

This is affecting coreaudio as well from what I can tell.

@hkratz

This appears to have regressed via #90499, cc @hkratz.

As a temporary workaround, you can try setting the environment variable MACOSX_DEPLOYMENT_TARGET to something like 11.0.

With #90499 Rust sets MACOSX_DEPLOYMENT_TARGET to 10.7 for the linker, if not set, to match what we are passing to LLVM. Setting it to 10.8 or above get rids of the linker failure.

I will look into this more.

@madsmtm

This is affecting coreaudio as well from what I can tell.

@madsmtm Can you elaborate on that? Building it locally with cargo +nightly build --target x86_64-apple-darwin worked fine for me on an M1.

@madsmtm

I can confirm that using MACOSX_DEPLOYMENT_TARGET=10.8 gets rid of the linker failure in winit.
For coreaudio, MACOSX_DEPLOYMENT_TARGET=10.11 is required.

This is affecting coreaudio as well from what I can tell.

@madsmtm Can you elaborate on that? Building it locally with cargo +nightly build --target x86_64-apple-darwin worked fine for me on an M1.

You need to build a binary, so that the linker is actually invoked. Try building an example (e.g. cargo +nightly build --target x86_64-apple-darwin --example feedback).

I can clarify this a little more: If you were building a Rust program for an Apple platform that linked to APIs with availability somewhere between 10.7+ and 12.0+, and you weren't setting your own deployment target, #90499 has revealed a bug in your build process. Your binaries were only ever going to run on recent OS releases, but you would have been unaware without testing.

For coreaudio, if it really requires APIs from 11.0+, that seems a bit recent/limiting for a library that's existed since forever on macOS. If the authors were unaware of this, #90499 has revealed that they need to do more work to attain backwards compatibility. There will be more breakage like it, but it is the good kind, because anyone using newer Apple APIs should be aware of their minimum platform requirement, just as Xcode forces you to be.

Hmm, this change had more impact than I thought.

(Open-source) clang does a dance to determine which minimum Macos version to pass as part of the target triple to LLVM and to ld via -platform_version by looking at the various command line options and environment variables that have accumulated over the years. If nothing is specified, it apparently defaults to the minimum of the ~~DefaultDeploymentTarget~~Version specified in the SDKROOTs SDKSettings.json and the host Macos version here.

I think ideally we should probably try to mimic this behavior somewhat so we don't confront users with having to set MACOSX_DEPLOYMENT_TARGET, while they don't have to do it using clang. Not sure how we can easily accomplish this though.

Fix CGDisplayCreateUUIDFromDisplayID linking (again)

I can clarify this a little more: If you were building a Rust program for an Apple platform that linked to APIs with availability somewhere between 10.7+ and 12.0+, and you weren't setting your own deployment target, #90499 has revealed a bug in your build process. Your binaries were only ever going to run on recent OS releases, but you would have been unaware without testing.

For coreaudio, if it really requires APIs from 11.0+, that seems a bit recent/limiting for a library that's existed since forever on macOS. If the authors were unaware of this, #90499 has revealed that they need to do more work to attain backwards compatibility. There will be more breakage like it, but it is the good kind, because anyone using newer Apple APIs should be aware of their minimum platform requirement, just as Xcode forces you to be.

Actually, that makes a lot of sense, thanks! I fully agree that it's better to catch these issues when developing (instead of later, when users complain that the application doesn't work on their older system)!

I've opened rust-windowing/winit#2078 to fix the linking in winit, and RustAudio/coreaudio-sys#49 would fix the linking in coreaudio.

Note that this is a breaking change for macOS users, before, they didn't have to specify MACOSX_DEPLOYMENT_TARGET, now they do. Furthermore, they have no easy way to find out that they need to (they just get a confusing linker error)!

Additionally, the affected crates (that we know about) are quite widely used (at least half of all Rust games depend on winit and/or coreaudio-sys), and have not yet fixed the issue, nor yet released updated versions which includes these fixes. And even when this has been done, it will take time before users have run cargo update and received working versions (can probably be helped along by yanking the affected versions).

I don't really know the procedure for this, but I would like to propose reverting #90499 (at least from the current beta) to give the affected crates more time to update.

Again, I really do agree with the change, I just also want to minimize the breakage macOS users will experience from upgrading Rust.

Rollup merge of rust-lang#91870 - rusticstuff:macosx_min_version_reve…

reopening as this is a beta-regression, and we keep those open until beta fix lands (or we explicitly decline to fix).

Revert "Set MACOSX_DEPLOYMENT_TARGET env var to default for linking i…

Backport merged.

Just for reference, the linking in the noted crates have since been fixed:

winit v0.26.1, Fix CGDisplayCreateUUIDFromDisplayID linking (again) rust-windowing/winit#2078
coreaudio-sys v0.2.9, Fix AudioUnit linking RustAudio/coreaudio-sys#51

Help: Why is this WIP Bevy platform failing?

[HELP] Why is this WIP Bevy platform failing?

I root-caused this a little bit more today, sharing my findings:

The problem can be reproduced with MACOSX_DEPLOYMENT_TARGET=10.7 rustc --target=x86_64-apple-darwin and the following file:

#[link(name = "AppKit", kind = "framework")]
#[link(name = "CoreGraphics", kind = "framework")]
extern "C" {
    pub fn CGDisplayCreateUUIDFromDisplayID(display: u32) -> *const std::os::raw::c_void;
}

fn main() {
    unsafe { CGDisplayCreateUUIDFromDisplayID(0) };
}

Weirdly enough, if you remove the linking to CoreGraphics, everything links fine.

I think this is actually a bug in ld64, it seems like:

The linker is reading the symbol tables from AppKit.framework/AppKit.tbd and CoreGraphics.framework/CoreGraphics.tbd to try to figure out where CGDisplayCreateUUIDFromDisplayID comes from.
When reading AppKit.tbd, it sees that it has a few "reexported-libraries", which includes /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices, which is where the symbol actually comes from (well, nowadays from ColorSync, but that is re-exported from ApplicationServices).
But when reading CoreGraphics.tbd , it finds the symbol $ld$install_name$os10.7$/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices, which is intended to indicate that "when linking CoreGraphics on older platforms, use ApplicationServices instead"
But instead, it ends up making it seem like CoreGraphics is actually the real ApplicationServices, and CoreGraphics does not expose _CGDisplayCreateUUIDFromDisplayID, so the linking fails.

Again, I believe this to be a linker bug because the problem does not happen with lld (try passing -Clinker=rust-lld).

Navigation Menu

Repository navigation

Linking to private symbols from frameworks between compilation units #91372

Meta

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Linking to private symbols from frameworks between compilation units #91372

Description

Meta

Activity

madsmtm commented on Nov 29, 2021

madsmtm commented on Nov 29, 2021

ehuss commented on Nov 30, 2021

hkratz commented on Nov 30, 2021

hkratz commented on Nov 30, 2021

madsmtm commented on Nov 30, 2021

cormacrelf commented on Dec 1, 2021

hkratz commented on Dec 1, 2021

madsmtm commented on Dec 1, 2021

madsmtm commented on Dec 13, 2021

10 remaining items

pnkfelix commented on Dec 16, 2021

emilyalbini commented on Jan 8, 2022

madsmtm commented on Jan 27, 2022

madsmtm commented on Sep 9, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions