configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc

[vstinner]

BPO	26662
Nosy	@vstinner, @xdegaye, @vadmium, @koobs, @yan12125
Files	delete-on-error.patch py_for_gen_26662.patch py_for_gen_26662_2.patch py_for_gen_26662_3.patch: patch with a new error message

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-07-26.11:00:00.993>
created_at = <Date 2016-03-29.10:38:22.939>
labels = ['build']
title = 'configure/Makefile doesn\'t check if "python" command works, needed to build Objects/typeslots.inc'
updated_at = <Date 2016-07-26.11:34:34.112>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2016-07-26.11:34:34.112>
actor = 'koobs'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-26.11:00:00.993>
closer = 'xdegaye'
components = ['Build']
creation = <Date 2016-03-29.10:38:22.939>
creator = 'vstinner'
dependencies = []
files = ['43683', '43767', '43792', '43829']
hgrepos = []
issue_num = 26662
keywords = ['patch']
message_count = 20.0
messages = ['262600', '262602', '262634', '262636', '262639', '269926', '270159', '270160', '270161', '270162', '270652', '270653', '270667', '270824', '270904', '270913', '270915', '270968', '270980', '271356']
nosy_count = 6.0
nosy_names = ['vstinner', 'xdegaye', 'python-dev', 'martin.panter', 'koobs', 'yan12125']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26662'
versions = ['Python 3.5', 'Python 3.6']

[vstinner]

I built Python 3.6 (default) on FreeBSD using:

./configure --cache-file=config_debug.cache --with-pydebug CFLAGS=-O0
make

I'm using FreeBSD current, uname -a:

FreeBSD freebsd 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r296485: Tue Mar 8 07:04:36 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

It looks like "cc" is clang LLVM compiler 3.8.0.

Extract of the make output:
---
[haypo@freebsd ~/prog/python/default]$ make
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
*** WARNING: importing extension "_testmultiphase" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_ssl" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_curses_panel" failed with <class 'RuntimeError'>: invalid slot offset
---

[vstinner]

Oh ok, I understood the issue.

[haypo@freebsd ~/prog/python/default]$ rm Objects/typeslots.inc
[haypo@freebsd ~/prog/python/default]$ make
python ./Objects/typeslots.py < ./Include/typeslots.h > Objects/typeslots.inc
/bin/sh: python: not found
*** Error code 127

Stop.
make: stopped in /home/haypo/prog/python/default
---

I have "python2" and "python3" but no "python" on FreeBSD CURRENT. The configure script is smart enough to select "python3" as the ASDL generator, but it uses $(PYTHON) ("python") to build Objects/typeslots.inc.

We should use a smarter code to select the external "python" program to build Objects/typeslots.inc.

Moreover, when "make Objects/typeslots.inc" fails, it creates an empty Objects/typeslots.inc file:
---
Objects/typeobject.o: Objects/typeslots.inc
Objects/typeslots.inc: $(srcdir)/Include/typeslots.h $(srcdir)/Objects/typeslots.py
$(PYTHON) $(srcdir)/Objects/typeslots.py < $(srcdir)/Include/typeslots.h > Objects/typeslots.inc
---

The Objects/typeslots.inc should be written directly by Objects/typeslots.py.

[vadmium]

Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

I noticed there are other boostrap scripts that require a “python” command via various mechanisms, e.g. Python/makeopcodetargets.py is run via “#! /usr/bin/env python” by default. It would at least be nice to make everything uniform, i.e. always use a /usr/bin/env hashbang, always use $(PYTHON), or always use this smarter external Python command.

[vstinner]

Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

My initial issue is that "import ssl" fails with <class 'RuntimeError'>: invalid slot offset. This error is really strange and Google doesn't know it.

I probably missed something when I compiled Python, but I would prefer that Python build system detects such issues for me, rather than having to use workarounds.

[vadmium]

Maybe the configure.ac code for generating opcode.h is a better alternative. It looks like it prints a warning but would continue with the build:

if test "$PYTHON" = not-found; then
    OPCODEHGEN="@echo python: $PYTHON! cannot run Tools/scripts/generate_opcode_h.py"
else
    OPCODEHGEN="$PYTHON"
fi

[yan12125]

Hit by missing python command from Python/makeopcodetargets.py, too. Is make touch the correct way?

[vadmium]

Victor: Gnu Make has a special “.DELETE_ON_ERROR” target that will cause Objects/typeslots.inc to be removed if the “python” command fails, which is slightly better than truncating it. See delete-on-error.patch. It looks like you were not using Gnu Make, but maybe there is an equivalent for BSD’s Make.

Chi: Yes I think “make touch” may help you work around the problem. It should be safe to try anyway :)

[yan12125]

Thanks! It's now working find with make touch :)

[vstinner]

Right, I built Python on FreeBSD, it's likely that make was not GNU make.

[vadmium]

Survey of different mechanisms for running Python to generate files:

Tools/scripts/generate_opcode_h.py and Parser/asdl_c.py: configure.ac tries python3.6, python3, python, etc commands. It either runs the scripts with what it finds, or substitutes an “echo” command that emits an explanation but does not fail. One disadvantage of this was it needed a script to be made compatible with Python 2.5 <https://bugs.python.org/issue17861#msg217409\>.

Python/makeopcodetargets.py: Always run as a Unix command with shebang “#! /usr/bin/env python”.

Objects/typeslots.py: Always run via $(PYTHON) command, which defaults to “python$(EXE)”. No magic configure stuff, but could be manually adjusted by using “make PYTHON=[. . .]”. This is the only remaining use of the $(PYTHON) makefile variable in 3.6, but it is shared with other stuff in 2.7.

An idea I had a while ago was to only run these scripts with a special Make command, eliminating the need for “make touch”. Going over a discussion earlier this year, Nick Coghlan <http://article.gmane.org/gmane.comp.python.devel/156682\> said a disadvantage of this is that you have to remember the special Make command to use in the rare event that your work affects a generated. Maybe this could be overcome with documentation, warning messages, etc. Argument Clinic code and the configure script are not automatically regenerated.

Another potential idea is a flag (either to configure, or to Make) to manually disable or enable all running of Python scripts. When disabled, it would use the existing generated files, and perhaps warn if they are out of date. Or by default perhaps the warning could be an error, with the resolution to either tell it to use a pre-installed Python, or explicitly run the build with the existing generated files.

[xdegaye]

This patch follows the first mechanism listed by Martin. The change in configure.ac fixes the problem described by Victor, it does not generate an empty Objects/typeslots.inc file when python is not found that would cause the cryptic " <class 'RuntimeError'>: invalid slot offset" errors on subsequent builds.

The change in Objects/typeslots.py is not strictly necessary to fix the problem but prevents typeslots.py to create an invalid typeslots.inc file through the previous stdout redirection mechanism when, for example, typeslots.py is modified inadvertently with a change that is not python2 compatible and typeslots.py is run by python2 and fails.

[xdegaye]

Forgot to say that with the patch and no python in $PATH, the following error message is output:
$ touch Include/typeslots.h
$ make
Cannot generate Objects/typeslots.inc, python not found !
Re-run configure with python in PATH.
make: *** [Makefile:908: Objects/typeslots.inc] Error 1

[vadmium]

The PYTHON_FOR_GEN scheme seems reasonable to me, as is changing the way typeslots script is run.

I do wonder about the advice in the message. If I ran into the problem, I would probably either override PYTHON_FOR_GEN when running Make, or try something like “make touch” instead. But that’t not a big deal.