C API: Implement singletons as functions calls in the stable ABI 3.13

[vstinner]

Feature or enhancement

Currently, the C API implements many constants as macros. Examples:

#define Py_False _PyObject_CAST(&_Py_FalseStruct)
#define Py_None (&_Py_NoneStruct)

I propose to export important constants as symbols in the stable ABI and the limited C API, but still implement them as macros in the non-limited C API.

Linked PRs

• gh-115754: Export Py_None, Py_False, Py_True as symbols #115755
• gh-115754: Get singletons via function calls #116572
• gh-115754: Add Py_GetConstantRef() function #116883
• gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_STR) #125194
• gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) #125195
• gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) #125581
• gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_STR) #125583

[vstinner]

cc @encukou @davidhewitt @da-woods who might be interested by the topic.

[vstinner]

I also tried a different approach, use static const:

diff --git a/Include/boolobject.h b/Include/boolobject.h
index 19aef5b1b8..a841bf1587 100644
--- a/Include/boolobject.h
+++ b/Include/boolobject.h
@@ -18,8 +18,8 @@ PyAPI_DATA(PyLongObject) _Py_FalseStruct;
 PyAPI_DATA(PyLongObject) _Py_TrueStruct;
 
 /* Use these macros */
-#define Py_False _PyObject_CAST(&_Py_FalseStruct)
-#define Py_True _PyObject_CAST(&_Py_TrueStruct)
+static PyObject * const Py_False = _PyObject_CAST(&_Py_FalseStruct);
+static PyObject * const Py_True = _PyObject_CAST(&_Py_TrueStruct);
 
 // Test if an object is the True singleton, the same as "x is True" in Python.
 PyAPI_FUNC(int) Py_IsTrue(PyObject *x);

But it changes the constant type (make them constant in the limited C API), and it doesn't work with dlopen/dlsym.

cc @pitrou

[vstinner]

I'm not sure how to categorize "important" vs "non-important" constants.

Maybe only "pointers" to variables are interesting?

Example of constants:

#define PY_VECTORCALL_ARGUMENTS_OFFSET \
    (_Py_STATIC_CAST(size_t, 1) << (8 * sizeof(size_t) - 1))

#define Py_single_input 256   // compile.h

#define _PyDateTime_DATE_DATASIZE 4
#define PyDateTime_CAPSULE_NAME "datetime.datetime_CAPI"

#define Py_T_DOUBLE    4

#define S_IFMT 0170000   // fileutils.h

#define METH_VARARGS  0x0001

#define SEP L'\\'

#define Py_MATH_PI 3.14159265358979323846

#define Py_NAN ((double)NAN)

In vim, I looked for :grep '\# *define [A-Za-z0-9_]\+ \+[^ \\]' Include/*.h.

[zooba]

Can we export them as functions rather than symbols? Symbols are the reason we've struggled so hard to get the isolation we want, because we have to invent tricky solutions that wouldn't have been necessary had they been functions.

[zooba]

And if functions, why not Py_GetConstant(int) and an enum to select which one is wanted? That's way more easily manageable from our POV (we'd want to document that the constant comes with an incremented refcount and is only valid in the active interpreter).

---

I'm not sure how to categorize "important" vs "non-important" constants.

My vote is for "important ones are PyObject *"

[vstinner]

Can we export them as functions rather than symbols?

Sure, it's possible. But it would be slower and require C extensions to have a different API than the non-limited C API, no?

I have a working implementation where Py_None and other constants are exported as symbols ("variables") and implemented as macros: best of both world.

[zooba]

Well no, they could all use the same API if they want, it'd just be the function one ;-)

And the stable ABI can be Py_GetConstant(PY_NONE) while the limited API includes #define Py_None Py_GetConstant(PY_NONE). It adds some overhead, obviously, but if we'd done it this way from the start then we wouldn't have the overhead on all the INCREF/DECREF calls instead (because we wouldn't have needed to immortalise static types and constants).

Maybe we've run out of reasons that a function would be better than a data reference? I don't see any reason to assume that, and so I think having a function would be a safer ABI.