Missing 'extern "C"' for _Py_ctype_table

[andrewvaughanj]

BPO	43816
Nosy	@scoder, @vstinner, @ericvsmith, @matrixise, @miss-islington, @andrewvaughanj
PRs	bpo-43816: add 'extern "C"' to pyctype.h #25365 [3.8] bpo-43816: Add extern "C" to Include/cpython/pyctype.h (GH-25365) #25386 [3.9] bpo-43816: Add extern "C" to Include/cpython/pyctype.h (GH-25365) #25387

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-13.12:21:07.917>
created_at = <Date 2021-04-12.15:16:23.621>
labels = ['expert-C-API', 'build', '3.8', '3.9', '3.10']
title = 'Missing \'extern "C"\' for _Py_ctype_table'
updated_at = <Date 2021-04-13.13:31:26.617>
user = 'https://github.com/andrewvaughanj'

bugs.python.org fields:

activity = <Date 2021-04-13.13:31:26.617>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-13.12:21:07.917>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-04-12.15:16:23.621>
creator = 'andrewvaughanj'
dependencies = []
files = []
hgrepos = []
issue_num = 43816
keywords = ['patch']
message_count = 16.0
messages = ['390855', '390856', '390858', '390860', '390868', '390890', '390896', '390900', '390910', '390949', '390954', '390957', '390958', '390959', '390960', '390963']
nosy_count = 6.0
nosy_names = ['scoder', 'vstinner', 'eric.smith', 'matrixise', 'miss-islington', 'andrewvaughanj']
pr_nums = ['25365', '25386', '25387']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue43816'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[andrewvaughanj]

With Python 3.9.4, and when compiling with Visual Studio 2019, we have noticed that the variable _Py_ctype_table is not scoped with in an extern "C" block, and where the Python library (python39.lib) has been compiled with a C compiler.

This causes an issue when trying to refer to _Py_ctype_table from a C++ file, as the compiler tries to name-mangle the use of _Py_ctype_table, but the linker cannot then tie the mangled name to non-mangled named from python39.lib.

Example:

#include "Python.h"
int main() { return _Py_ctype_table[0]; }

Compilation:

cl.exe /Fe:test.exe /TP /I include test.cpp /link libs/python39.lib
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29336 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

test.cpp
Microsoft (R) Incremental Linker Version 14.28.29336.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.exe
libs/python39.lib
test.obj
test.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) unsigned int const * const _Py_ctype_table" (__imp_?_Py_ctype_table@@3QBIB) referenced in function main
test.exe : fatal error LNK1120: 1 unresolved externals

With cl.exe:

cl.exe /Bv  
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29336 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

Compiler Passes:
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\cl.exe:        Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1.dll:        Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1xx.dll:      Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c2.dll:        Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1xx.dll:      Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\link.exe:      Version 14.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\mspdb140.dll:  Version 14.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\1033\clui.dll: Version 19.28.29336.0

A naïve check of Python.h (e126547) seems to suggest that:

* There are 82 includes

* 64 of these contain `extern "C"`

* 8 do not contain `extern "C"`

* The remaining 10 are either system includes or pyconfig.h

For the 8 that do not contain extern "C", none of these use PyAPI_DATA. This leads me to believe that it is an oversight that pyctype.h does not have extern "C"

[matrixise]

In fact, _Py_ctype_table is limited to the internal parts of the interpreter. So in this case, this one could not be used in an external tool.

You can read: https://docs.python.org/3/c-api/stable.html

I am not sure that you correctly use the API.

[andrewvaughanj]

In fact, _Py_ctype_table is limited to the internal parts of the interpreter. So in this case, this one could not be used in an external tool.

Hmm, so why is this "exposed" by the "world-facing" Python.h file?

I should say: we found this bug via Cython; and it was Cython that was accessing/referring to _Py_ctype_table -- our Cythonated code pulls in C++ headers, so we need to compile these files as C++.

I am happy to re-assign this as a Cython bug, but the fact it is fixed with an extern "C" in Python.h, really makes it feel like it is a Python-proper issue and not a "user" issue.

[andrewvaughanj]

I am happy to re-assign this as a Cython bug, but the fact it is fixed with an extern "C" in Python.h, really makes it feel like it is a Python-proper issue and not a "user" issue.

Just to extend on this:

1) The Cython-generated code uses `Py_ISSPACE` (and not `_Py_ctype_table`), but the expansion of the macro `Py_ISSPACE` then adds `_Py_ctype_table` to the user's code

2) The "user-fix" is to wrap `#include <Python.h>` in `extern "C"` -- however, given other parts of Python.h already do this, it seems extraneous to expect a C++ user to wrap Python.h in this way

[andrewvaughanj]

1. The Cython-generated code uses Py_ISSPACE (and not _Py_ctype_table), but the expansion of the macro Py_ISSPACE then adds _Py_ctype_table to the user's code

I wrote this up as a Cython bug here (just to see if the Cython team consider this "their" bug): cython/cython#4111

[scoder]

These macros are a sort-of documented part of the C-API. At least, they were mentioned in a What's New document:

https://docs.python.org/3/whatsnew/2.7.html?highlight=py_isspace#build-and-c-api-changes

They were added here, for the Py2.7 release:

8374981

It looks to me like the header file should use "extern C".
@eric, do you agree?

[ericvsmith]

These files were probably added as part of str.format() or short float repr. I think the fact that they've been moved to Include/cpython means that user code shouldn't be using them. See https://bugs.python.org/issue35134 and https://vstinner.github.io/split-include-directory-python38.html. Probably best to ask Victor.