Missing extensions modules when cross compiling python 3.5.2 for arm on Linux

[bennykj]

BPO	28444
Nosy	@akuchling, @doko42, @xdegaye, @vadmium, @zware, @moreati, @yan12125
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	python_x86_host_arm_target_config.log: config.log for the cross compilation python3.5.2_x86_host_arm_target.log: Build output from cross compiling python 3.5.2 for ARM on x86_64 Linux setup_x86_arm: modules/setup file for cross compiling on x86_64 for ARM setup_x86_x86: modules/set-up for natively building on x86_64 removed_modules.patch python_3_5_2_remove_modules.build.log: Build output (with removed_modules.patch) from cross compiling python 3.5.2 for ARM on x86_64 Linux python_3_5_2_remove_modules.config.log: config.log (with removed_modules.patch) from cross compiling python 3.5.2 for ARM on x86_64 Linux python_3_6_0b2_remove_modules.build.log: Build output (with removed_modules.patch) from cross compiling python 3.6.0b2 for ARM on x86_64 Linux python_3_6_0b2_remove_modules.config.log: config.log (with removed_modules.patch) from cross compiling python 3.6.0b2 for ARM on x86_64 Linux removed_modules_2.patch removed_modules_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-10-29.15:05:12.472>
created_at = <Date 2016-10-14.15:34:43.601>
labels = ['extension-modules', 'build', '3.7']
title = 'Missing extensions modules when cross compiling python 3.5.2 for arm on Linux'
updated_at = <Date 2017-03-31.16:36:22.144>
user = 'https://bugs.python.org/bennykj'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:22.144>
actor = 'dstufft'
assignee = 'none'
closed = True
closed_date = <Date 2016-10-29.15:05:12.472>
closer = 'xdegaye'
components = ['Extension Modules', 'Cross-Build']
creation = <Date 2016-10-14.15:34:43.601>
creator = 'bennykj'
dependencies = []
files = ['45093', '45098', '45099', '45100', '45103', '45109', '45110', '45111', '45112', '45143', '45239']
hgrepos = []
issue_num = 28444
keywords = ['patch']
message_count = 31.0
messages = ['278654', '278662', '278695', '278697', '278700', '278702', '278703', '278707', '278709', '278710', '278713', '278714', '278717', '278718', '278719', '278720', '278722', '278725', '278730', '278740', '278741', '278751', '278832', '278833', '278990', '278999', '279533', '279649', '279677', '282957', '282967']
nosy_count = 10.0
nosy_names = ['akuchling', 'doko', 'xdegaye', 'python-dev', 'martin.panter', 'zach.ware', 'Alex.Willmer', 'yan12125', 'bennykj', 'John Huang']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue28444'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[bennykj]

When cross compiling Python for ARM many of the extension modules are not build

However when compiling for the native platform the extension modules are properly build.

Cross Compilation Steps
=======================

CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3

make

sudo PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin:$PATH make install

Extension Modules Built when cross compiled
===========================================

building '_ctypes_test' extension
building 'cmath' extension
building '_json' extension
building '_testcapi' extension
building '_testbuffer' extension
building '_testimportmultiple' extension
building '_testmultiphase' extension
building '_lsprof' extension
building '_opcode' extension
building 'parser' extension
building 'mmap' extension
building 'audioop' extension
building '_crypt' extension
building '_csv' extension
building 'termios' extension
building 'resource' extension
building 'nis' extension
building '_multibytecodec' extension
building '_codecs_kr' extension
building '_codecs_jp' extension
building '_codecs_cn' extension
building '_codecs_tw' extension
building '_codecs_hk' extension
building '_codecs_iso2022' extension
building '_decimal' extension
building '_multiprocessing' extension
building 'ossaudiodev' extension
building 'xxlimited' extension
building '_ctypes' extension

Compilation Steps on x86 Machine
================================

CONFIG_SITE=config.site ./configure --enable-shared --disable-ipv6 --prefix=/opt/python3

make

sudo make install

Extension Modules Built when natively compiled
===========================================
building '_struct' extension
building '_ctypes_test' extension
building 'array' extension
building 'cmath' extension
building 'math' extension
building '_datetime' extension
building '_random' extension
building '_bisect' extension
building '_heapq' extension
building '_pickle' extension
building '_json' extension
building '_testcapi' extension
building '_testbuffer' extension
building '_testimportmultiple' extension
building '_testmultiphase' extension
building '_lsprof' extension
building 'unicodedata' extension
building '_opcode' extension
building 'fcntl' extension
building 'grp' extension
building 'spwd' extension
building 'select' extension
building 'parser' extension
building 'mmap' extension
building 'syslog' extension
building 'audioop' extension
building 'readline' extension
building '_crypt' extension
building '_csv' extension
building '_posixsubprocess' extension
building '_socket' extension
building '_sha256' extension
building '_sha512' extension
building '_md5' extension
building '_sha1' extension
building 'termios' extension
building 'resource' extension
building 'nis' extension
building 'binascii' extension
building 'pyexpat' extension
building '_elementtree' extension
building '_multibytecodec' extension
building '_codecs_kr' extension
building '_codecs_jp' extension
building '_codecs_cn' extension
building '_codecs_tw' extension
building '_codecs_hk' extension
building '_codecs_iso2022' extension
building '_decimal' extension
building '_multiprocessing' extension
building 'ossaudiodev' extension
building 'xxlimited' extension
building '_ctypes' extension

I've further tried building for ARM natively on ARM machine and the extensions was build successfully.

Tool chain used for cross compilation
=======================================

Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.4/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: /home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/snapshots/gcc-linaro-4.9-2016.02/configure SHELL=/bin/bash --with-bugurl=https://bugs.linaro.org --with-mpc=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-mpfr=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gmp=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gnu-as --with-gnu-ld --disable-libstdcxx-pch --disable-libmudflap --with-cloog=no --with-ppl=no --with-isl=no --disable-nls --enable-c99 --with-tune=cortex-a9 --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-multilib --enable-multiarch --with-build-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/sysroots/arm-linux-gnueabihf --enable-lto --enable-linker-build-id --enable-long-long --enable-shared --with-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu/arm-linux-gnueabihf/libc --enable-languages=c,c++,fortran,lto --enable-checking=release --disable-bootstrap --with-bugurl=https://bugs.linaro.org --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=arm-linux-gnueabihf --prefix=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu
Thread model: posix
gcc version 4.9.4 20151028 (prerelease) (Linaro GCC 4.9-2016.02)

Host Machine
=============
Ubuntu 16.04.1 LTS

Linux whachamacallit 4.4.0-42-generic #62-Ubuntu SMP Fri Oct 7 23:11:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[xdegaye]

Please upload the output of make, this is where the error messages are printed when extension modules fail to build.
Where is set the root directory for arm headers and libraries in the configure command line ?

[bennykj]

Thank you for the response.

I'm not too sure how to set the root directory for arm headers and libraries in the configure command line.

So I've tried the below, but still not all of the extensions modules was not build (e.g math, socket, select)

export CROSS_PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc

CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf CFLAGS="-I${CROSS_PATH}/usr/include" LDFLAGS="-L${CROSS_PATH}/usr/lib" CPPFLAGS="-I${CROSS_PATH}/usr/include" ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3

Attached please find the build log

[bennykj]

Could you please tell me if the decision to make the extension modules like math, socket, select etc taken in the configure step or is it left to the Makefile.

Also could you please give me some hints on where to look for on how this decision is made (e.g file name / function name.)

[yan12125]

Wierd. Modules like 'math', 'select' or '_socket' are added unconditionally in detect_modules() of setup.py. Did you disable modules in Modules/Setup.local or Modules/Setup or via the global variable disabled_module_list in setup.py?

[bennykj]

Thank you for your response.

Please note that I haven't touched the Modules/Setup.local or Modules/Setup or the global variable disabled_module_list in setup.py

However I'm noticing that the math is commented in the Modules/Setup on both cross compiling on x86_64 for ARM as well as natively compiling on x86_64 for x86_64. But on the native build for x86_64 the math, socket, select are all build properly.

Attached please find the modules/setup file

[bennykj]

Attaching the modules/set-up for natively building on x86_64

[xdegaye]

Your Setup file matches the one distributed with Python 3.5.2.
Did you get the Python source from https://www.python.org/ ?

[bennykj]

Thank you for the response.

Yes I've downloaded the XZ compressed source from python.org

Based on your comment I've double checked the MD5 with

https://www.python.org/downloads/release/python-352/

benny@whachamacallit:~/Downloads$ md5sum Python-3.5.2.tar.xz
8906efbacfcdc7c3c9198aeefafd159e Python-3.5.2.tar.xz

[bennykj]

please let me know if there is any additional steps I need to follow when building from the source downloaded from python.org

[xdegaye]

I assume from your logs that a native (not the cross-built one) python3.5 already exists and is on your PATH.
What is the value of sys.builtin_module_names as given by this native python3.5 interpreter ?

[bennykj]

Thank you for your response

Yes. There is a python 3.5 that is installed on the system (from apt-get)

benny@whachamacallit:~$ which python3
/usr/bin/python3

benny@whachamacallit:~$ python3
Python 3.5.2 (default, Sep 10 2016, 08:21:44) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> import sys
>>> 
>>> sys.builtin_module_names
('_ast', '_bisect', '_codecs', '_collections', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha512', '_signal', '_socket', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
>>>