Python doesn't find Xcode 7 SDK stub libraries

[tdsmith]

BPO	25136
Nosy	@brettcannon, @ronaldoussoren, @ned-deily, @merwok, @dstufft, @tdsmith
Files	xcode-stubs.diff: Diff against 3.5.0 release xcode-stubs-2.7.patch: Patch against 2.7 release branch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2016-02-24.14:44:52.495>
created_at = <Date 2015-09-16.01:03:02.962>
labels = ['OS-mac', 'build', 'library']
title = "Python doesn't find Xcode 7 SDK stub libraries"
updated_at = <Date 2016-02-24.14:44:52.493>
user = 'https://github.com/tdsmith'

bugs.python.org fields:

activity = <Date 2016-02-24.14:44:52.493>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2016-02-24.14:44:52.495>
closer = 'ned.deily'
components = ['Build', 'Distutils', 'macOS']
creation = <Date 2015-09-16.01:03:02.962>
creator = 'tdsmith'
dependencies = []
files = ['40478', '40479']
hgrepos = []
issue_num = 25136
keywords = ['patch']
message_count = 6.0
messages = ['250813', '250894', '260694', '260803', '260804', '260805']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ned.deily', 'eric.araujo', 'afoglia', 'python-dev', 'dstufft', 'tdsmith']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue25136'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[tdsmith]

In Xcode 7, Apple is replacing many of the .dylibs in SDKROOT with textual stubs. 1 These files exist on disk with filenames like:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib/libz.tbd

They are short YAML documents that look like this: 2

The same linker invocation that has always worked will continue to work with Xcode 7 (i.e. you still pass -lz to the linker), but this disrupts the checks that cpython's setup.py uses to determine if it can build extension modules. The dylibs physically exist on disk in /usr/lib, but since we've set -isysroot to the appropriate SDKROOT in CPPFLAGS, distutils searches for the dylibs in the sysroot path, and does not find them (since they have been replaced with .tbd stubs). Since distutils cannot find the libraries, setup.py declines to attempt to build any of the extension modules that depend on libraries in the OS X SDK, even though it would have succeeded if it had tried. Several Homebrew users have reported this while trialling Xcode 7 3.

distutils should treat the .tbd files as a "real" library so that compiler.find_library_file succeeds and setup.py will proceed to attempt to build the extension modules.

The attached diff applies against the 3.5.0 release and allows extension modules to be built against Xcode 7 without installing the Command-Line Tools package.

If anyone is experiencing this issue, a workaround is to install the Xcode Command Line Tools package with xcode-select --install (which, among other things, installs headers to /usr/include), ensure the CLT is active with xcode-select -s /Library/Developer/CommandLineTools, and do not include -isysroot in CPPFLAGS when building python.

[ned-deily]

Thanks for the analysis and the patches, Tim. Now that Xcode 7 is released, we'll need to ensure we fully support the new stubs. But, as you note, the workaround as always is to make sure that the current Command Line Tools are installed, the configuration that we test with.

[brettcannon]

We should update the devguide to specify that the command-line tools need to be installed and either explain or point to documentation on how to install the tools.

[python-dev]

New changeset 59d797915dca by Ned Deily in branch '2.7':
Issue bpo-25136: Support Apple Xcode 7's new textual SDK stub libraries.
https://hg.python.org/cpython/rev/59d797915dca

New changeset b0490b8af7aa by Ned Deily in branch '3.5':
Issue bpo-25136: Support Apple Xcode 7's new textual SDK stub libraries.
https://hg.python.org/cpython/rev/b0490b8af7aa

New changeset db6c1308eb31 by Ned Deily in branch 'default':
Issue bpo-25136: merge from 3.5
https://hg.python.org/cpython/rev/db6c1308eb31

[python-dev]

New changeset 157eb9d40bf9 by Ned Deily in branch '2.7':
Issue bpo-25136: Add reference to 'xcode-select --install' to Mac README.
https://hg.python.org/cpython/rev/157eb9d40bf9

New changeset 61752c7ea9c7 by Ned Deily in branch '3.5':
Issue bpo-25136: Add reference to 'xcode-select --install' to Mac README.
https://hg.python.org/cpython/rev/61752c7ea9c7

New changeset 2c4448bbed1f by Ned Deily in branch 'default':
Issue bpo-25136: merge from 3.5
https://hg.python.org/cpython/rev/2c4448bbed1f

[ned-deily]

I've committed Tim's patches with some additional comments for release in 2.7.12, 3.5,2, and 3.6.0 and have added a note about using 'xcode-select --install' to the Mac/README files. (The Developer's Guide already documents this.) Thanks again, Tim.