Support Apple app store bitcode

[brson]

Bitcode is the future of Apple's app distribution and we don't support it now. Doing so is tricky because Apple choses whatever rev of LLVM they want and upgrade it per their whims. We can't couple our LLVM to theirs because we have our own needs and can't be forced to upgrade whenever Apple decides.

Here are some options:

• We ship an entirely separate toolchain for targetting Apple bitcode. This is quite ugly.
• We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.
• We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly
• We create a build of rustc called rustc-apple and ship it in an optional rustc-apple package. It is exactly like rustc except instead of linking to rustc_llvm it links to rustc_llvm_apple. When rustc gets a request to emit bitcode it just defers completely to the rustc-apple binary. I actually fancy this solution quite a bit.
• We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

I think the LLVM dynamic loading and defer-to-alternate-rustc solutions are most promising.

cc https://users.rust-lang.org/t/ios-rust-integration/6928/4

cc @bluejekyll

[nagisa]

We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

---

I feel like I heard all the way back in 2013-ish that using LLVM bitcode as distribution format is stupid. I certainly agree; does distributing native binary libraries not work anymore for iOS?

[nagisa]

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

[bluejekyll]

I experimented briefly with this, i.e. output bitcode during cargo builds, then tried to create a static library with the bitcode included. When I tried to llvm-link against the bitcode library, I got an incompatible LLVM version from the bitcode.

I don't have an easy test to reproduce this, but my guess is there is just a dumb version check that denies linking between different LLVM versions? Or, I just did something entirely wrong. I will try to come up with a test case when I have some time to look at this again.

[sanxiyn]

It would be useful to have exact error message from llvm-link.

[nagisa]

@bluejekyll LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

[ricky26]

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

This is somewhat easier said than done - as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode. This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

I think they avoid this by only allowing the latest Xcode to submit apps (and I'm pretty sure it's burned into the output image which LLVM version you used).

IIRC, the bitcode is packaged for each architecture, since cross-architecture bitcode isn't generated by clang (I think that's an anti-goal of clang and bitcode in general). It's stored in a section for each object file so at the very least it's duplicated. That could be part of why someone mentioned that bitcode might not be the best way of going about this.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

Since this bug mentions the App Store, is it worth talking about the exception handling story here? (i.e. panic=abort is strictly required at the moment.)

[bluejekyll]

LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

@nagisa thanks for letting me know this. It gives me hope that there might still be a solution here, and that I was probably doing something wrong.

[nagisa]

@ricky26 good points.

as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode.

Isn’t Xcode LLVM on apple the same thing as system LLVM? I meant the Xcode LLVM then. We would have to make sure xcode is always the most recent version when shipping rustc trains.

Of course the way apple does things rules out us producing a valid apple bitcode with an old version of rustc and essentially forces us throw all the benefits our stability story provides out the window, and I don’t see any way this could be fixed.

This also potentially means maintaining the LLVM bindings for two LLVM versions

We already maintain¹ support for LLVM versions 3.7 through 3.9 (and potentially trunk). As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard. If Xcode LLVM is really some ancient/custom/etc version, then I don’t see us being able to support this feature at all, then. Especially since we have no option of sending patches to that LLVM in order to add the features we need. I also wouldn’t want to lock rustc into supporting 3.7 forever in case Apple decided to not update Xcode LLVM until 2038.

¹: however, if rustc was built against LLVM x.y, it must be linked to LLVM x.y exactly.