The coverage job is broken: distutils build_ext fails on None

[vstinner]

BPO	30273
Nosy	@brettcannon, @pitrou, @vstinner, @jkloth
PRs	bpo-30273: Update sysconfig #1464 bpo-23404: make touch becomes make regen-all (#1405) (#1461) #1465 bpo-30273: update distutils.sysconfig #1515 [3.6] bpo-30273: update distutils.sysconfig for venv's created from Python (#1515) #1625 [3.5] bpo-30273: update distutils.sysconfig for venv's created from Python (#1515) #1626

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2017-06-26.09:58:56.498>
created_at = <Date 2017-05-04.20:51:05.020>
labels = ['3.7', 'build', 'tests']
title = 'The coverage job is broken: distutils build_ext fails on None'
updated_at = <Date 2018-05-18.23:42:24.996>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-05-18.23:42:24.996>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2017-06-26.09:58:56.498>
closer = 'vstinner'
components = ['Build', 'Tests']
creation = <Date 2017-05-04.20:51:05.020>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30273
keywords = []
message_count = 21.0
messages = ['293014', '293015', '293016', '293017', '293027', '293028', '293029', '293031', '293035', '293300', '293304', '293306', '293308', '293320', '293321', '293832', '293834', '294270', '294283', '296868', '317067']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'pitrou', 'vstinner', 'jkloth']
pr_nums = ['1464', '1465', '1515', '1625', '1626']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30273'
versions = ['Python 3.7']

[vstinner]

Bug reported by Antoine Pitrou on the python-committers mailing list.

According to Brett Canon, the first failure was:
https://travis-ci.org/python/cpython/jobs/228409786

And the regression is related to my commit a5c62a8: bpo-23404.

Collecting coverage
Downloading coverage-4.3.4.tar.gz (361kB)
� Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/coverage.egg-info
writing pip-egg-info/coverage.egg-info/PKG-INFO
writing dependency_links to
pip-egg-info/coverage.egg-info/dependency_links.txt
writing entry points to pip-egg-info/coverage.egg-info/entry_points.txt
writing top-level names to pip-egg-info/coverage.egg-info/top_level.txt
writing manifest file 'pip-egg-info/coverage.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-ewtgqc2r/coverage/setup.py", line 204, in
<module>
        main()
      File "/tmp/pip-build-ewtgqc2r/coverage/setup.py", line 194, in main
        setup(**setup_args)
      File "/home/travis/build/python/cpython/Lib/distutils/core.py",
line 148, in setup
        dist.run_commands()
      File "/home/travis/build/python/cpython/Lib/distutils/dist.py",
line 955, in run_commands
        self.run_command(cmd)
      File "/home/travis/build/python/cpython/Lib/distutils/dist.py",
line 974, in run_command
        cmd_obj.run()
      File
"/home/travis/build/python/cpython/venv/lib/python3.7/site-packages/setuptools/command/egg_info.py",
line 279, in run
        self.find_sources()
      File
"/home/travis/build/python/cpython/venv/lib/python3.7/site-packages/setuptools/command/egg_info.py",
line 306, in find_sources
        mm.run()
      File
"/home/travis/build/python/cpython/venv/lib/python3.7/site-packages/setuptools/command/egg_info.py",
line 533, in run
        self.add_defaults()
      File
"/home/travis/build/python/cpython/venv/lib/python3.7/site-packages/setuptools/command/egg_info.py",
line 562, in add_defaults
        sdist.add_defaults(self)
      File
"/home/travis/build/python/cpython/Lib/distutils/command/sdist.py", line
228, in add_defaults
        self._add_defaults_ext()
      File
"/home/travis/build/python/cpython/Lib/distutils/command/sdist.py", line
311, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/home/travis/build/python/cpython/Lib/distutils/cmd.py",
line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/travis/build/python/cpython/Lib/distutils/cmd.py",
line 107, in ensure_finalized
        self.finalize_options()
      File
"/home/travis/build/python/cpython/Lib/distutils/command/build_ext.py",
line 150, in finalize_options
        py_include = sysconfig.get_python_inc()
      File
"/home/travis/build/python/cpython/Lib/distutils/sysconfig.py", line
100, in get_python_inc
        incdir = os.path.join(_sys_home, get_config_var('AST_H_DIR'))
      File "/home/travis/build/python/cpython/Lib/posixpath.py", line
92, in join
        genericpath._check_arg_types('join', a, *p)
      File "/home/travis/build/python/cpython/Lib/genericpath.py", line
149, in _check_arg_types
        (funcname, s.__class__.__name__)) from None
    TypeError: join() argument must be str or bytes, not 'NoneType'

\----------------------------------------

�Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-build-ewtgqc2r/coverage/�

[vstinner]

To reproduce the bug, build the master branch of Python and then run:

./python -m venv venv
./venv/bin/python -m pip install -U coverage

Or just:

$ ./venv/bin/python -c 'from distutils import sysconfig; print(sysconfig.get_python_inc())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/haypo/prog/python/master/Lib/distutils/sysconfig.py", line 100, in get_python_inc
    incdir = os.path.join(_sys_home, get_config_var('AST_H_DIR'))
  File "/home/haypo/prog/python/master/Lib/posixpath.py", line 92, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/haypo/prog/python/master/Lib/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

[vstinner]

get_config_var('AST_H_DIR')

Oh, I see. My commit a5c62a8 removed multiple variables from Makefile.pre.in to simplify it, but I didn't notice that some of them (only AST_H_DIR?) were used outside Makefile.pre.in. I completely forgot the evil sysconfig (evil in term of backward compatibility...).

I removed the following variables from Makefile.pre.in:

• @GENERATED_COMMENT@
• ASDLGEN
• ASDLGEN_FILES
• AST_ASDL
• AST_C
• AST_C_DIR
• AST_H
• AST_H_DIR
• GRAMMAR_C
• GRAMMAR_H
• GRAMMAR_INPUT
• OPCODETARGETGEN
• OPCODETARGETGEN_FILES
• OPCODETARGETS_H
• OPCODE_H
• OPCODE_H_DIR
• OPCODE_H_GEN
• OPCODE_H_SCRIPT
• PYTHON_FOR_GEN (renamed to PYTHON_FOR_REGEN)

I didn't see the point of the AST_H_DIR variable since its value was hardcoded to "Include" and it was only used to regenerate generated files related to AST.

I don't understand why sysconfig chose this specific variable to get the "Include" subdirectory!?

I'm writing a fix to replace AST_H_DIR variable with "Include".

[vstinner]

New changeset b109a1d by Victor Stinner in branch 'master':
bpo-30273: Update sysconfig (bpo-1464)
b109a1d

[vstinner]

New changeset 9d02f56 by Victor Stinner in branch '3.6':
[3.6] bpo-23404: make touch becomes make regen-all (bpo-1405) (bpo-1461)
9d02f56

[jkloth]

The change to sysconfig *may* be a regression wrt bpo-15366.

I'm not in the position to be able to check this possibility, ATM, but it is possible that other things have also changed since this was commited 5 years ago.

[vstinner]

The change to sysconfig *may* be a regression wrt bpo-15366.

Hum, I'm still able to create a venv and install coverage in the venv after my commit b109a1d, whereas this commit basically reverts the bpo-15366 fix.

[vstinner]

New changeset ab6b962 by Victor Stinner in branch '3.5':
bpo-23404: make touch becomes make regen-all (bpo-1405) (bpo-1461) (bpo-1465)
ab6b962

[vstinner]

Jeremy Kloth: "I'm not in the position to be able to check this possibility, ATM, but it is possible that other things have also changed since this was commited 5 years ago."

I expect that many bugs have been fixed in virtualenv and venv modules, so maybe bpo-15366 was fixed indirectly?

It would be nice if you can tests on your side, since I'm not confident in my own tests :-)

I modified the build system of Python 2.7, 3.5, 3.6 and master (3.7). So you can pick any of these branches to run your test :-) I'm not sure that the bug affected or can affect Python 2.7.

[vstinner]

The coverage job has been fixed and I'm unable to reproduce bpo-15366 bug (my change doesn't seem to have introduced a regression), so I close the issue.

I sent an email to python-dev to warn about this change, so maybe others will double check.

@jeremy: please double check my change to make sure that I didn't introduce a regression.