gh-114099: Add configure/Makefile changes to support iOS builds.

[freakboy3742]

Part of the PEP 730 work to add iOS support.

Adds:

• the configure/Makefile changes need build an iOS/tvOS/watchOS framework,
• a testbed Xcode project
• resources used by the framework builds.

Apologies in advance for the large PR; this is by far the largest single patch needed for PEP 730. Unfortunately, all these changes are interrelated - the resources are used by the build, and the makefile references the testbed. I couldn't work out a way to make the patch smaller. By way of partial defence - once you remove the autogenerated files (such as configure), and the new XCode project and stub binaries, the patch is a lot smaller 😅

Why tvOS and watchOS?

PEP 730 only proposes adding Tier 3 support for iOS. The tvOS and watchOS portions of this patch are strictly optional. I've included those portions of the patch because they're mostly symmetrical with the iOS changes, and it saves me maintaining these patches externally. However, I'm definitely aware of the project-level optics of a tvOS and watchOS folder appearing in the CPython repo. Unfortunately, the platforms are just different enough that I felt they needed their own standalone folders.

If the existence of the tvOS and watchOS folders (and/or the references in the configure script) are an impediment to merging, I'll gladly remove them.

Differences between macOS and iOS Frameworks

iOS frameworks are slightly different to macOS frameworks.

• macOS frameworks are "multi-version", as they are designed to support having multiple framework versions in a single Framework folder. iOS frameworks can't be shared between apps, so they don't support "multi-version" format.
• The metadata required by Info.plist is slightly different.
• The install name for the library in the framework must be based on an @rpath, rather than an absolute path, as the install path for the framework will be app-specific.
• iOS frameworks can only include binary .dylib artefacts and headers; and the headers won't be copied into the final distribution binary. As a result, the standard library and support binaries cannot be distributed inside the framework - they must be distributed parallel to the Framework folder. To accomodate this, the make install target for iOS will generate a file structure like the following:

--enable-framework location

• bin
• include symlink to -> Python.framework/Headers
• lib
  • python3.X
    • ...
• Python.framework
  • Headers
    • ...
  • Python (the dylib binary)
  • Info.plist

The task of copying the standard library into an appropriate location in the final app bundle is left as a task for the developer using the framework.

Summary of changes

configure changes

• Adds a RESSRCDIR variable to allow sharing of macOS and iOS Makefile steps
• Adds an INSTALLTARGETS variable so that iOS framework builds don't install binaries, manfiles, etc.
• Adds a PYTHONFRAMEWORKINSTALLNAMEPREFIX variable; this is used as the install name for the library, allowing iOS frameworks to specify an @rpath-based name
• Evaluates MACHDEP earlier in the configure process so that ac_sys_system is available
• Modifies _PYTHON_HOST_PLATFORM evaluation so that it includes the OS, ABI and architecture (e.g., ios-iphoneos-arm64 or ios-iphonesimulator-x86_64). This is used as the sysconfig identifier,
• Adds a IOS_DEPLOYMENT_TARGET variable, tracking the iOS minimum API version.
• Differentiates between SOABI_PLATFORM and PLATFORM_TRIPLET. SOABI_PLATFORM is used in binary module names, and includes the ABI, but not the OS or CPU architecture (e.g., math.cpython-313-iphonesimulator.dylib). PLATFORM_TRIPLET is used as the multiarch value, and contains the ABI and architecture (e.g., iphoneos-arm64)
• Skips the automated detection of a couple of system functions where autoconf gets the answer wrong - autoconf only checks whether a symbol can be linked, and doesn't use header files; iOS headers raise warnings when some symbols are used. There are also methods that exist and can be compiled, but raise errors if used at runtime.
• Automatically skips the configuration of ac_cv_file__dev_ptmx and ac_cv_file__dev_ptc, which are usually mandatory command line arguments for cross-platform builds. iOS is a reliable cross-build target, so we know these aren't available.
• Presumptively adds the 3 PEP 730 platforms to the PY_SUPPORT_TIER=3 list.
• Disables the standard lib modules that aren't available on iOS

Makefile changes

• Removes the use of the deprecated -Wl,-single_module flag on macOS
• Modifies the macOS framework builds to allow for iOS frameworks as well, using the RESSRCDIR, INSTALLTARGETS, and PYTHONFRAMEWORKINSTALLNAMEPREFIX variables.
• Adds a testiOS target to build and run the XCode testbed project, and extract the test results.

config.sub changes

The config.sub script has been updated to the 2024-01-01 version; additional patches have been applied to support the -simulator suffix, and the arm64_32 CPU architecture.

The additional patches were submitted upstream to autoconf in August last year. The official 2024-01-01 update includes the portions of that patch that added support for tvOS and watchOS, but didn't include updates adding the -simulator suffix or the arm64_32 CPU architecture. Unfortunately, the autoconf submission process is a bit of a black hole - you email them patches and get no further correspondence, even when the patch is merged. I've re-submitted the suffix and arm64_32 architecture patches; in the meantime, they've been manually applied here.

Support files

• Adds folders to store iOS/tvOS/watchOS build resources and documentation.

• Adds template plist files for iOS/tvOS/watchOS frameworks.

• Adds stub binaries that wrap compiler tooling. Xcode doesn't expose explicit compilers for iOS/tvOS/watchOS; instead, it uses an xcrun script that resolves to a full compiler path (e.g., xcrun --sdk iphoneos clang to get the clang for an iPhone device). However, using this script poses 2 problems:

  • The output of xcrun includes paths that are then machine specific, resulting in a sysconfig module that cannot be shared between users
  • It results in CC/CPP/LD/AR definitions that include spaces. There is a lot of C ecosystem tooling (including distutils and setuptools) that assumes that you can split a command line at the first space to get the path to the compiler executable; this isn't the case when using xcrun.

  So - to avoid these problems, the Resources/bin folders for each platform includes shell script wrappers around the tools needed by configure, named using the scheme that autoconf expects by default. These scripts are relocatable, and will always resolve to the appropriate local system paths. By including these scripts in the "installed" bin folder, the contents of the sysconfig module becomes useful for end-users to compile their own modules.

Other changes

• Adds an iOS testbed Xcode project. This test project uses the XCTest framework to expose a single test - that test instantiates a embedded Python interpreter, and runs the equivalent of python -m test. If the CPython test suite passes, the single XCTest passes; otherwise, the XCTest fails. The test suite is configured to run on an "iPhone SE (3rd gen)" simulator - this is a stable device identifier that isn't subject to annual device refresh cycles.
• There's no testbed project for tvOS or watchOS at this time, as they're not being targeted for Tier 3.
• The iOS/tvOS/watchOS folders will eventually contain READMEs detailing how to use these changes. I've omitted the README from this PR to keep the patch smaller and focussed; also the build still won't actually work until there's a few more patches in place.

• Issue: Add support for iOS as a target platform #114099

[erlend-aasland]

My gut feeling says we want to keep the watchOS and tvOS changes outside for now¹, if that's ok with you :) cc. @ned-deily, who might have opinions on this.

There are some configure changes that might be split out into separate changes (introducing the INSTALLTARGETS and PYTHONFRAMEWORKINSTALLNAMEPREFIX build variables, yak-shave like moving the MACHDEP switch in configure.ac).

Footnotes

1. if only to bring the size of the PR down ↩

[ned-deily]

My gut feeling says we want to keep the watchOS and tvOS changes outside for now1, if that's ok with you :)

We did have some discussions about this at the Brno sprint. After a quick look through, I don't have a strong feeling about it. But, assuming the end goal is to support all of the platforms (at some level) and since the work is already done in the PR, it might be best to bite the bullet now and get it all in, rather than making extra work later. Perhaps @ronaldoussoren has an opinion on this.

[erlend-aasland]

But, assuming the end goal is to support all of the platforms (at some level) and since the work is already done in the PR, it might be best to bite the bullet now and get it all in, rather than making extra work later.

Yep. I would still like to consider to keep tvOS/watchOS out for this PR, for the sake of the review (not a hard requirement from me though, so feel free to give this suggestion a thumbs down 😄)

[freakboy3742]

I'll take a closer look in the morning, but I think it might be possible to separate out:

• Initial reordering and variable renaming (with no new iOS material)
• Adding iOS-specific cases and resources, plus Makefile targets for framework building
• Adding the testbed project and Makefile changes to run the tests
• Adding tvOS and watchOS cases and resources

The downside to this sort of split is that it won't necessarily be obvious why some of the refactoring from an early step is needed until later step lands. However, the individual patches will definitely be smaller, so maybe with some explanatory notes (and reference to this PR as a "full" version), it can work; and it would let the tvOS/watchOS portions be their own entity.

Alternatively, I could just purge the watchOS and tvOS portions of this patch (and submit them separately). That will make this PR smaller; my counterargument would be that everywhere this PR adds tvOS/watchOS, it's essentially 2 near-identical clauses in the same case statement where iOS appears, which is context you lose (or is harder to see) is a separate PR.

@ned-deily @erlend-aasland Do either of these sound worthwhile alternative approaches?

[mhsmith]

also the build still won't actually work until there's a few more patches in place.

Does this mean "the build won't complete", or "the build completes but the outputs can't be run"?

And is the answer to these questions different between iOS, watchOS and tvOS?

[freakboy3742]

also the build still won't actually work until there's a few more patches in place.

Does this mean "the build won't complete", or "the build completes but the outputs can't be run"?

And is the answer to these questions different between iOS, watchOS and tvOS?

The build will complete, but it won't be usable/runnable. The situation will be the same on all three new platforms. Apologies for the ambiguity.

[erlend-aasland]

• Initial reordering and variable renaming (with no new iOS material)

Yes, I would tear this refactoring out and submit it as a separate PR, of course coupled with a message that it is a yak-shave in preparation for the larger PR. It would definitely make it easier to see the exact changes that are being done in configure.ac in the following PR(s).

• Adding iOS-specific cases and resources, plus Makefile targets for framework building
• Adding the testbed project and Makefile changes to run the tests

IIUC, these two must be submitted together in order for the build to complete and be usable?

[freakboy3742]

IIUC, these two must be submitted together in order for the build to complete and be usable?

The build will complete with just the first of these two, but won't be testable without the testbed project and makefile targets from the second.

However, even then, there's a couple more patches needed before the test suite will start successfully.

[erlend-aasland]

However, even then, there's a couple more patches needed before the test suite will start successfully.

Sure, but those are not part of this PR anyway (at least not as far as I can see) :)

[freakboy3742]

Closing in favor of a more granular refactor.