Build failure for `python@3.12.1` on mac with non-system version of ncurses

[s-sajid-ali]

Bug report

Bug description:

Using the spack package manager to install python@3.12.1 on darwin-sonoma-m1 with the following command:

sasyed@MAC-140753 ~> spack install --keep-stage python@3.12.1+optimizations%clang@17.0.6 ^libffi@3.4.6

results in the following build error:

Build error

...
==> Installing python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il [21/21]
==> No binary for python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il found: installing from source
==> Fetching https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tgz
==> Ran patch() for python
==> python: Executing phase: 'configure'
==> python: Executing phase: 'build'
==> python: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j1' 'install'
...
     1763    ld: warning: reexported library with install name '@rpath/libunwind.1.dylib' found at '/Users/sasyed/Documents/packages/spack/opt/spa
             ck/darwin-sonoma-m1/apple-clang-15.0.0/llvm-17.0.6-uzrfikeehxbzxp3x3twl6wjnh53hpvmr/lib/libunwind.1.0.dylib' couldn't be matched with
              any parent library and will be linked directly
     1764    DYLD_LIBRARY_PATH=/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-python-3.12.1-frh24vuhc6b5e
             yy346wng65mlwm3h3il/spack-src ./python.exe -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(
             ), *sys.version_info[:2]))' >platform
  >> 1765    [ERROR] _curses failed to import: dlopen(/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-pyth
             on-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il/spack-src/build/lib.macosx-14.0-arm64-3.12/_curses.cpython-312-darwin.so, 0x0002): symbol 
             not found in flat namespace '_mvwadd_wch'
  >> 1766    [ERROR] _curses_panel failed to import: PyCapsule_Import could not import module "_curses"
  ...

The associated log files with this build error are:

spack-build-01-configure-out.txt
spack-build-02-build-out.txt
spack-build-03-install-out.txt

No build error when building python@3.11.7. The associated log files are:
spack-build-01-configure-out.txt
spack-build-02-build-out.txt
spack-build-03-install-out.txt

Note that the following flags are added by spack when optimizations are turned on:

        if "+optimizations" in spec:
            config_args.append("--enable-optimizations")
            # Prefer thin LTO for faster compilation times.
            if "@3.11.0: %clang@3.9:" in spec or "@3.11.0: %apple-clang@8:" in spec:
                config_args.append("--with-lto=thin")
            else:
                config_args.append("--with-lto")
            config_args.append("--with-computed-gotos")

https://github.com/spack/spack/blob/b79761b7eb5eb2e2d6b8174c3043524462772bf6/var/spack/repos/builtin/packages/python/package.py#L574-L581

CPython versions tested on:

3.12

Operating systems tested on:

macOS

[ned-deily]

You should take this issue up first with the spack project. I'm not familiar with spack but a quick glance at the logs you provided shows a rather complicated and non-standard build process. A web search shows that at least one other person has seen this particular build failure using spack. The failure seems to be that that the built _curses module fails to import:

[ERROR] _curses failed to import: dlopen(/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il/spack-src/build/lib.macosx-14.0-arm64-3.12/_curses.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_mvwadd_wch'

It could be due to a mismatch between ncurses headers and libs when building.

Also, be aware that Python 3.12.3 is current so 3.12.1 is no longer supported.

[s-sajid-ali]

Thanks for the response @ned-deily!

The underlying issue seems to be from this bit of configure.ac:

cpython/configure.ac

Lines 6572 to 6588 in a019347

	AS_VAR_IF([ac_cv_header_ncurses_h], [yes], [
	if test "$ac_sys_system" != "Darwin"; then
	dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw.
	PKG_CHECK_MODULES([CURSES], [ncursesw], [
	AC_DEFINE([HAVE_NCURSESW], [1])
	have_curses=ncursesw
	], [
	WITH_SAVE_ENV([
	AC_CHECK_LIB([ncursesw], [initscr], [
	AC_DEFINE([HAVE_NCURSESW], [1])
	have_curses=ncursesw
	CURSES_CFLAGS=${CURSES_CFLAGS-""}
	CURSES_LIBS=${CURSES_LIBS-"-lncursesw"}
	])
	])
	])
	fi

which makes the following assumption:

 dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw.

However, for the build environment I'm using, with a custom ncurses the above assumption breaks:

sasyed@MAC-140753 ~> diff (pkg-config --libs ncurses | psub) (pkg-config --libs ncursesw | psub)
1c1
< -L/PATH_TO_NCURSES_6.5/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
 -Wl,-search_paths_first -lncurses -ltinfo
---
> -L/PATH_TO_NCURSES_6.5/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
 -Wl,-search_paths_first -lncursesw -ltinfow
sasyed@MAC-140753 ~ [1]> 

Would it be possible to add the -lncursesw via CURSES_LIBS in spack or does the configure.ac need to be patched with something like the following?

--- a/configure.ac	2024-05-10 23:12:33
+++ b/configure.ac	2024-05-10 23:25:10
@@ -6418,8 +6418,8 @@
 AC_CHECK_HEADERS([curses.h ncurses.h])
 
 AS_VAR_IF([ac_cv_header_ncurses_h], [yes], [
-  if test "$ac_sys_system" != "Darwin"; then
-    dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw.
+  dnl man diff: Exit status is 0 if inputs are the same, 1 if different!
+  AS_IF([test "diff <($PKG_CONFIG --libs ncurses) <($PKG_CONFIG --libs ncursesw)" != 0 ], [
     PKG_CHECK_MODULES([CURSES], [ncursesw], [
       AC_DEFINE([HAVE_NCURSESW], [1])
       have_curses=ncursesw
@@ -6433,7 +6433,7 @@
         ])
       ])
     ])
-  fi
+  ])
 
   AS_VAR_IF([have_curses], [no], [
     PKG_CHECK_MODULES([CURSES], [ncurses], [

[ned-deily]

@erlend-aasland, what do you think? This might also be related to #113565.

[erlend-aasland]

Could you check if #119816 works for you?

[s-sajid-ali]

@erlend-aasland : The current spack recipe for Python does not support building from git branches, though it is possible to do so. I'm not sure if it is worth the effort to add that to test this PR.

On the other hand, if I use system compilers with the newer ncurses in the path, configure seems to pick the system built ncurses as in #113565.

For the build that failed, I'd used spack to build Python and harnessed spack built llvm. Note that spack comes with compiler wrappers that were used during the build that failed. Using the spack built compilers directly leads to unrelated errors:

checking for long double... yes
checking size of long double... configure: error: in '/Users/sasyed/Documents/packages/temp/bugfix/cpython-head':
configure: error: cannot compute sizeof (long double)
See 'config.log' for more details

[erlend-aasland]

@s-sajid-ali, got it. Looks like it should be fairly easy to hack that script to download a 3.13 beta release, though. If we decide to backport #119816 to 3.13, you'd should be able to test it relatively soon.

[s-sajid-ali]

@erlend-aasland : Yes, building the 3.13-beta2 release is straightforward. I'll be on the lookout for the beta2 release and let you know if the patch fixes the issue.

[erlend-aasland]

I think you'll have to wait for beta3; it's a large PR, and I don't want to rush it. Feel free to take a stab at reviewing it, if you feel like it :)

[s-sajid-ali]

Thanks for the update and asking me to review it, but I don't really know enough about autotools/autoconf to give useful feedback. It does look good on a high level and I'd be happy to test it out when the beta3 release comes out.

Thanks again for fixing this!

[erlend-aasland]

@s-sajid-ali, you'll find the 3.13 backport of #119816, #121202, in the upcoming 3.13-beta4 release, scheduled to be released in a couple of weeks time. There has been no decision regarding a 3.12 backport yet.