Building sharedmods fails with any dangling symlink in include path

[jmroot]

Bug report

I am the maintainer of the Python ports in MacPorts. Ports generally install their headers to /opt/local/include, and that is the case for several of Python's dependencies, so we configure the Python build to use it. It appears that Python will fail to build if any dangling symlink exists in that include directory, even if it's not a header that Python uses at all. The build output ends like this:

DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/<…>_ports_lang_python311/python311/work/Python-3.11.2 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/<…>_ports_lang_python311/python311/work/Python-3.11.2 ./python.exe -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/<…>_ports_lang_python311/python311/work/Python-3.11.2 CC='/usr/bin/clang' LDSHARED='/usr/bin/clang -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -arch x86_64  -flto -Wl,-export_dynamic -g -fprofile-instr-generate' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	./python.exe -E ./setup.py --no-user-cfg  build
running build
running build_ext
error: [Errno 2] No such file or directory: '/opt/local/include/lapacke_utils.h'
make[3]: *** [sharedmods] Error 1

There is no reference to lapacke_utils.h anywhere in the configured sources, so I can only assume something must be scanning the entire include path. Of course our OpenBLAS port should not be installing dangling symlinks, but I think the Python build system should be able to tolerate them when they are not something that is needed or used at all.

I think this was the first downstream report of this issue, and has a full log attached: https://trac.macports.org/ticket/66242

Your environment

• CPython versions tested on: 3.11.2
• Operating system and architecture: macOS 13.2 x86_64

Linked PRs

• [3.11] gh-101778: Fix build error when there's a dangling symlink in the directory containing "ffi.h" #113466

[FFY00]

I tried reproducing this on my Linux machine but wasn't able to 😕

[jmroot]

@FFY00 Thanks for trying. Can you share what the exact test you tried was?

[FFY00]

@jmroot sorry for the delay. I don't remember now, but if you can give me reproduction instructions, I can try on a macOS 13.3 machine.

[jmroot]

@FFY00 If you have MacPorts installed, this will do it:

1. sudo ln -s bar.h /opt/local/include/foo.h (assumes bar.h does not exist, so foo.h will be a dangling symlink)
2. sudo port -v build python311

I haven't tried a manual build, but can if you need me to. It's possible that implicit include dirs like /usr/local/include might be handled differently to ones specified via a -I option in CPPFLAGS, which is what MacPorts does with /opt/local/include.

[ronaldoussoren]

To reproduce this:

• Make sure you have libffi and pkg-config installed
• Run configure and make sure LIBFFI_INCLUDEDIR is set in Makefile
• Add a dangling symlink to $LIBFFI_INCLUDEDIR (which should be the directory containing ffi.h)
• Run make

The problem is that grep_headers_for in setup.py is not robust when one of the files it needs to check is a dangling symlink. Fixing this is easy enough (on the 3.11 branch):

diff --git a/setup.py b/setup.py
index 4f122b62e0..ad8fb81b21 100644
--- a/setup.py
+++ b/setup.py
@@ -224,6 +224,7 @@ def is_macosx_sdk_path(path):
 
 def grep_headers_for(function, headers):
     for header in headers:
+        if not os.path.exists(header): continue
         with open(header, 'r', errors='surrogateescape') as f:
             if function in f.read():
                 return True

I haven't checked yet if a fix is needed on main an 3.12, where setup.py no longer exists and this functionality is moved to the configure script.

[jmroot]

Thanks for tracking this down @ronaldoussoren! Confirming that your patch fixes the issue for me.

[ronaldoussoren]

Thanks for tracking this down @ronaldoussoren! Confirming that your patch fixes the issue for me.

Thanks for the confirmation.