Difficulties building tip of tree Python for emscripten

[hoodmane]

BPO	47232
Nosy	@tiran, @pmp-p, @hoodmane
Files	Makefile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-04-05.22:33:35.926>
created_at = <Date 2022-04-05.19:27:14.891>
labels = []
title = 'Difficulties building tip of tree Python for emscripten'
updated_at = <Date 2022-04-05.22:33:35.925>
user = 'https://github.com/hoodmane'

bugs.python.org fields:

activity = <Date 2022-04-05.22:33:35.925>
actor = 'hoodchatham'
assignee = 'none'
closed = True
closed_date = <Date 2022-04-05.22:33:35.926>
closer = 'hoodchatham'
components = []
creation = <Date 2022-04-05.19:27:14.891>
creator = 'hoodchatham'
dependencies = []
files = ['50723']
hgrepos = []
issue_num = 47232
keywords = []
message_count = 8.0
messages = ['416807', '416812', '416815', '416816', '416819', '416820', '416821', '416824']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'hoodchatham']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue47232'
versions = []

[hoodmane]

I am trying to build tot Python for Emscripten to test recent changes and having trouble. The problem is caused by recent changes since 3.11.0a6 to sre. I build a build Python 3.11 from tot, and provided it to ./configure --with-build-python=$(HOSTBUILDDIR)/python. However, I still see invocations to the system Python 3.11 at /usr/local/bin/python3.11.

The problem seems to be that the generated Makefile has a bunch of stuff collapsed into one line:

PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) /src/cpython/build/Python-3.11.0dev0/builddir/build-host/python

manually fixing this to:

PYTHON_FOR_BUILD=/src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) 
PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)

fixes the problem.

My configure invocation is as follows:

	( \
		cd $(BUILDDIR); \
		CONFIG_SITE=./config.site-wasm32-emscripten emconfigure \
		  ../../configure \
		  	  --cache-file=$(ROOT)/python.config.cache \
			  CFLAGS="${PYTHON_CFLAGS}" \
			  CPPFLAGS="-I$(SQLITEBUILD) -I$(BZIP2BUILD) -I$(ZLIBBUILD)" \
			  PLATFORM_TRIPLET="$(PLATFORM_TRIPLET)" \
			  --with-build-python=$(HOSTBUILDDIR)/python \
			  --without-pymalloc \
			  --disable-shared \
			  --disable-ipv6 \
			  --enable-big-digits=30 \
			  --enable-optimizations \
			  --host=wasm32-unknown-emscripten\
			  --build=$(shell $(EXTRACTDIR)/config.guess) \
			  --prefix=$(INSTALL)  \
	)

[hoodmane]

Okay I found the problem. Elsewhere in my Makefiles PYTHON_FOR_BUILD was getting overwritten.

[hoodmane]

I take it back, I am still having this problem.

[tiran]

Please attach config.log and Makefile from the builddir.

[hoodmane]

config.log is apparently 1.7 MB and when I try to upload I get "Error 413: Entity Too Large". I've attached the Makefile.

[hoodmane]

There is still a pretty good change I am making some dumb mistake.

[tiran]

You are building a cross-build with "--with-build-python=/usr/local/bin/python3.11". Is the interpreter up to date?

During alpha and beta phase, the build Python interpreter should be built from the exact same git commit as the sources in "../..". If the interpreter deviates from the sources in your srcdir, then Python byte code (.pyc), frozen byte code, and re engine can have mismatching magic.

I recommend that you build inside an environment that does not have Python 3.11 installed. Instead build a build Python interpreter from the same checkout that you later use for cross-compilation. "./configure && make" is sufficient. You don't have to install the build interpreter. "--with-build-python=$(pwd)/../build/python" works fine.

We use this approach in python-wasm:

mkdir -p cpython/builddir/build
pushd cpython/builddir/build
../../configure -C
make -j$(nproc)
popd

mkdir -p cpython/builddir/emscripten-browser
pushd cpython/builddir/emscripten-browser
CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
emconfigure ../../configure -C \
--host=wasm32-unknown-emscripten \
--build=$(../../config.guess) \
--with-build-python=$(pwd)/../build/python
emmake make -j$(nproc)

It's going to get easier and less painful as soon as we reach beta phase. During beta the byte code will only change when it is required for a bug fix.

[hoodmane]

Okay I got it working. I was making several mistakes that each caused the same symptoms so it was hard to track them all down.

Thanks again for your help!