Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x

[vstinner]

BPO	33614
Nosy	@pfmoore, @vstinner, @tjguk, @ned-deily, @zware, @serhiy-storchaka, @zooba, @miss-islington
PRs	bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated #7165 [3.7] bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165) #7167 [3.6] bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165) #7168 bpo-33614: Override exit code in find_msbuild.bat #7169 [3.7] bpo-33614: Override exit code in find_msbuild.bat (GH-7169) #7176 [3.6] bpo-33614: Override exit code in find_msbuild.bat (GH-7169) #7177

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2018-05-28.23:17:58.099>
created_at = <Date 2018-05-23.12:20:55.236>
labels = ['build', '3.8', 'OS-windows']
title = 'Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x'
updated_at = <Date 2018-05-28.23:17:58.097>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-05-28.23:17:58.097>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2018-05-28.23:17:58.099>
closer = 'steve.dower'
components = ['Build', 'Windows']
creation = <Date 2018-05-23.12:20:55.236>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33614
keywords = ['patch']
message_count = 22.0
messages = ['317392', '317398', '317406', '317420', '317426', '317457', '317459', '317463', '317464', '317465', '317518', '317594', '317597', '317598', '317903', '317904', '317908', '317909', '317913', '317939', '317950', '317954']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'miss-islington']
pr_nums = ['7165', '7167', '7168', '7169', '7176', '7177']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33614'
versions = ['Python 3.8']

[vstinner]

http://buildbot.python.org/all/#/builders/80/builds/244

   (Link target) -> 
     python3_d.def : error LNK2001: unresolved external symbol PyImport_GetModule [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
     python3_d.def : error LNK2001: unresolved external symbol Py_UTF8Mode [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
     D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\amd64\python3_d.lib : fatal error LNK1120: 2 unresolved externals [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]

Maybe it's related to this recent change:

commit 4e29f56
Author: Serhiy Storchaka <storchaka@gmail.com>
Date: Tue May 22 20:59:42 2018 +0300

Add missed details of the C API introduced in 3.7. (GH-7047)

* Set the limited API version for PyImport_GetModule and PyOS_*Fork
  functions.
* Add PyImport_GetModule and Py_UTF8Mode in [PC/python3.def](https://github.com/python/cpython/blob/main/PC/python3.def).
* Add several functions in [Doc/data/refcounts.dat](https://github.com/python/cpython/blob/main/Doc/data/refcounts.dat).

The two missing symbols are defined by:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_FUNC(PyObject *) PyImport_GetModule(PyObject *name);
#endif

and

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_DATA(int) Py_UTF8Mode;
#endif

They are not part of the stable ABI.

[serhiy-storchaka]

My bad, I didn't test that change on Windows myself, and CI seems was broken at that moment.

But I have no ideas why the build is failed. These names are part of stable ABI since version 3.7. I don't see any difference between PyImport_GetModule and say PyInterpreterState_GetID.

[serhiy-storchaka]

It succeeds after removing directories amd64 and obj in PCbuild. Seems they are left from old builds and the clean up script did nor remove them.

This may be a cause of other weird bugs on Windows.

[zooba]

I guess that means there is an invalid date comparison going on to detect when the file has changed. This file doesn't change often, so it's easy to miss :)

All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.

[serhiy-storchaka]

If "PCbuild\build.bat -p x64" builds Python, "PCbuild\clean.bat -p x64" removes all files in PCbuild\amd64 but python3stub.exp and python3stub.lib.

[ned-deily]

All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.

It's more than a *bit* urgent since it is causing one of the 3.7 stable buildbots to fail to build and not run tests and we normally would not allow a release with stable buildbot failures.

[ned-deily]

(for example, http://buildbot.python.org/all/#/builders/133/builds/251 )

[zooba]

Okay, I'll take a look now.

[ned-deily]

Steve: just a minute ago, zware out the win8.1 bot's repo and restarted a 3.7 build.