[C API] Test that the Python C API is compatible with C++

[vstinner]

BPO	47165
Nosy	@gpshead, @vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-30.14:14:35.817>
labels = ['expert-C-API', 'type-feature', 'tests', '3.11']
title = '[C API] Test that the Python C API is compatible with C++'
updated_at = <Date 2022-04-05.06:49:28.827>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-04-05.06:49:28.827>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests', 'C API']
creation = <Date 2022-03-30.14:14:35.817>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 47165
keywords = []
message_count = 2.0
messages = ['416359', '416756']
nosy_count = 2.0
nosy_names = ['gregory.p.smith', 'vstinner']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47165'
versions = ['Python 3.11']

[vstinner]

There are more and more popular projects using the Python C API. The first big player is pybind11:
"Seamless operability between C++11 and Python"
https://pybind11.readthedocs.io/

Recently, I proposed a PR to add Python 3.11 support to the datatable project:
h2oai/datatable#3231

My PR uses pythoncapi_compat.h header file which provides recent C API functions to old Python functions. The header file implements these functions as static inline function.

Problem: a static inline function implemented in a C header file used in a C++ code file can emit C++ compiler warnings.

In datatable, I got two kinds of C++ compiler warnings:

• Usage of the C NULL constant: C++ prefers nullptr
• "Old-style" cast like (PyObject*)obj: C++ prefers static_cast, reinterpret_cast, etc.

It seems like these compiler warnings are not enabled by default. The datatable project seems enabling them in its CI and I was asked to fix these warnings.

In the pythoncapi-compat project (*), I chose to use nullptr and reinterpret_cast if the "__cplusplus" macro is defined. Example:

// C++ compatibility
#ifdef __cplusplus
#  define PYCAPI_COMPAT_CAST(TYPE, EXPR) reinterpret_cast<TYPE>(EXPR)
#  define PYCAPI_COMPAT_NULL nullptr
#else
#  define PYCAPI_COMPAT_CAST(TYPE, EXPR) ((TYPE)(EXPR))
#  define PYCAPI_COMPAT_NULL NULL
#endif

// Cast argument to PyObject* type.
#ifndef _PyObject_CAST
#  define _PyObject_CAST(op) PYCAPI_COMPAT_CAST(PyObject*, op)
#endif

(*) https://github.com/python/pythoncapi_compat

It's unclear to me if the Python C API has or has not the same issue than pythoncapi_compat.h.

Last years, some old macros of the Python C API have been converted to static inline functions, like Py_INCREF(). It's unclear to me if these compiler warnings happen on Py_INCREF(). I don't understand why, but static inline macros from Python.h didn't emit compiler warnings in datatable, whereas similar static inline functions of pythoncapi_compat.h emitted compiler warnings.

Maybe there is a difference between <Python.h> and "Python.h". Or maybe it depends if the header file is a "local" file, or a "system" header file (ex: installed in /usr/include/ on Linux).

A first step would be to build a C++ extension as part of the Python test suite and check that there is no compiler warning. My #76356 PR is a proof-of-concept of that.

I don't know which C++ version we should target. pybind11 targets C++11. See bpo-39355 for a discussion about C++20: usage of the C++20 "module" keyword... which is a "contextual keyword" in practice.

[gpshead]

If we can conditionally test new things based on C++XX version, accumulating modern issue regression tests seems useful. Otherwise 11 at minimum.

As for why some things trigger this and others don't, my wild _guess_ would be whether the statements appear within an extern "C" block. Though that's not really what that is for so it isn't clear to me.

[vstinner]

@erlend-aasland: it seems like PEP 670 made this issue worse. It seems like C and C++ compilers emit more warnings when code comes from a static inline function, rather than from a macro.

cc @corona10

[erlend-aasland]

That depends on how the macro was written in the first place (argument casts?, "return" cast?), so yeah, for some macros you'll get more warnings if you convert them to static inlined functions because of the strict typing.

[vstinner]

An example of (fixed) C++ issue with the Python C API: #87347