Allowing heaptypes to have a token for superclass identification

[neonene]

This is a continuation proposal of PEP 489 and later PEPs that aim to add the possibility of C-API modules and classes behaving more like Python equivalents. I would appreciate any feedback. Thanks.

Abstract

Superclass objects are currently not so easy to access depending on per-module state in multi-phase init extensions, which affects common subclass-checking such as Py*_Check(). They are unreachable in some cases.

This proposes a new Py_tp_token type slot ID for storing a key into heaptypes, so that a desired superclass can be found through the given type’s tp_mro or tp_bases slot.

Background

Superclasses are frequently referenced in slot methods (e.g. nb_add) for subclass checking. When implementing a multi-phase init module, a superclass heaptype object is supposed to be placed on a module state, which is not always good for the check:

• Unsafe during the module finalization.

  When a module is finalized, its state can be cleared by the GC before the associated objects and heaptypes are freed. Even if the module outlives the heaptypes, the types’ MRO and the module reference can also be cleared (gh-115874). Currently, there is no secure way to get a particular superclass at the final phase.

• Possibly redundant, which can affect at least micro benchmarks.

  module = PyType_GetModuleByDef(type, module_def);     // 1st MRO walk
  module_state = PyModule_GetState(module);
  PyType_IsSubtype(type, module_state->My_Super_Type);  // 2nd MRO walk
  

Proposal

The heap types will contain an additional pointer member as a kind of token, as long as the module author confirms:

• The pointer outlives the class^[1].
• It is “owned” by the module where the class lives, so it can keep alive and won’t clash with other modules.

The new Py_tp_token type slot will be available to store it as below:

PyType_Slot foo_slots[] = {
    {Py_tp_token, (pointer)},
    ...
};
PyType_Spec foo_spec = { ..., .slots = foo_slots};
PyType_Spec bar_spec = { ..., .slots = foo_slots};

• {Py_tp_token, NULL}:
  Equivalent to the absence of the slot.

• {Py_tp_token, foo_slots}:
  Token will be the pointer not to the foo_slots but to the assosiated type spec. The spec can be used at least to identify the memory layout of the given type.

• {Py_tp_token, &pointee_in_the_module}:
  The spec’s address can be specified explicitly, which will need a forward declaration. For another example, an extension modules that automatically wraps C++ classes could use the typeid operator for a token.

After the existing PyType_From*(spec, ...) function call, the token in the created type can be verified with PyType_GetSlot(type, Py_tp_token).

Specification

The PyHeapTypeObject struct will have a new member, the ht_token void pointer (NULL by default), which will not be inherited by subclasses.

The existing PyType_FromMetaclass(..., spec, ...) function will do the following, when the proposed type slot ID, Py_tp_token, is detected in spec->slots:

• ht_token = spec if PyType_Slot.pfunc == spec->slots else PyType_Slot.pfunc

Helpers

No public function is planned to be added.

Another subclass check would be:

1. Walk the subtype’s tp_mro from the last, or walk the tp_bases recursively.
2. Look for the token of the desired superclass.

Backwards Compatibility

One new pointer is added to the PyHeapTypeObject struct:

• ht_token member, which will not be documented.

One type slot ID is added:

• Py_tp_token to set PyHeapTypeObject.ht_token, which will be documented together with the usage of a public helper function for subclass checking.

Alternative

Another effective approach will be:

PyType_Slot foo_slots[] = {
    {Py_tp_token, Py_USE_SPEC},  // Py_USE_SPEC is NULL

• {Py_tp_token, NULL}:
  Token will be the pointer to the assosiated type spec rather than NULL.

• Absence of the slot:
  Token will not be stored into a heap type.

Pervious discussions

Better support for type checking

---

1. Comparing a live pointer to a dangling pointer is implementation defined. Also, an invalid pointer can become valid, reused by another object. ↩︎

[encukou (Petr Viktorin)]

Thanks for taking this on!

For context, this is a solution to one of the 4 issues left open in PEP 630 (Isolating Extension Modules); 2 of the others are now solved^[1].
Of course that there’s another possible solution to issues like this, along the lines of immortal per-interpreter singleton modules (#119663), and perhaps making static types more usable, but the mechanism proposed here would still be useful for modules that don’t want to be singletons.

The issue is that to check if an instance of a heap type ^[2] has a particular memory layout – something known at compile time – you currently need to go through dynamically allocated objects (module and types), which is cumbersome and makes finalization tricky.

The first solution that came to mind was to link the class to its PyType_Spec. But we can only do that if the spec is static (or otherwise outlives the class). If you’re defining a memory layout you probably do have a static spec, but it’s possible to make classes dynamically from throw-away specs.
If we have to ask the extension author for info (at least a confirmation flag), and we can’t assume the pointer points to a spec, we can just ask for a generic “token”.
Such a token could, with caveats, be used for other things, like the mentioned type-id when wrapping a foreign type system. To be honest I don’t think that’s an very important use case: once you can verify the memory layout you can also put that info in the type or instance.

Regarding how to specify that the spec should be used, I’d prefer {Py_tp_token, Py_TP_USE_SPEC}: if we silently switch a user-supplied pointer to another one, we should do it with NULL (or another special value).

IMO we do need to define helper functions: to get the token, and to do the check – e.g. get the first superclass with the given token.
(The proposal as posted is inconsistent: “Specification” says “No public function is planned to be added.” but “ Backwards Compatibility” says “ Py_tp_token […] will be documented together with the usage of a public helper function for subclass checking.”)

---

1. with PyType_FromMetaclass and PEP 697) ↩︎

2. or the type itself ↩︎

[neonene]

I’d prefer {Py_tp_token, Py_TP_USE_SPEC}: if we silently switch a user-supplied pointer to another one, we should do it with NULL (or another special value).

I agree to non-NULL Py_TP_USE_SPEC. I would not type the identifier for the NULL, and my NULL might be a bit confusing at first glance especially for others who are used to static types so far.

[neonene]

Alternatively, new prefix to the slot name:

• Py_optional_tp_token: pointer is optional

• Py_ht_token: heap type specific convention

• …

[neonene]

The document has a sentence:

Slots other than Py_tp_doc may not be NULL.

I think the rule should not have more exceptions on the Py_tp prefix. We could use a type slot instead of a luxury Py_TPFLAGS_*.

[encukou (Petr Viktorin)]

I’d be fine with more exceptions, e.g. we could have used {Py_tp_new, NULL} instead of Py_TPFLAGS_DISALLOW_INSTANTIATION.

[neonene]

I understand that {Py_tp_*, NULL} could have overridden the inheritance.

Regarding the doc, the update would be (?):

---

The following slots accept (the equivalent of) NULL when enabling the feature:

• {Py_tp_token, Py_TP_USE_SPEC}: See …

The other slots except Py_tp_doc may not be NULL.

[neonene]

The following is my current image of helper functions:

(expand):

// Check the MRO (or tp_bases) from the first:
int
PyType_GetBaseByToken(PyTypeObject *type, void *token,
                      PyTypeObject **result)
{
    // Find the first superclass whose token (memory layout ID)
    // matches the given one except `NULL`.
    //
    //   * If found, set *result to the new reference to
    //     the type and return 1.
    //   * If not found, set *result to `NULL` and return 0.
    //   * On error, raise an exception and return -1.
}

// MRO checks could be optimized:
int
PyType_CheckLayout(PyTypeObject *type, void *token)
{
    // Return true if the type or its superclass has the memory
    // layout ID that matches the given token except `NULL`.
    // This function always succeeds.
}

// Equivalent to `PyType_GetSlot(type, Py_tp_token) == token`:
int
PyType_CheckLayoutExact(PyTypeObject *type, void *token)
{
    // Return true if the type has the memory layout ID that
    // matches the given token except `NULL`.
    // This function always succeeds.
}

// Equivalent to `PyType_GetSlot(type, Py_tp_token)`:
void *
PyType_GetToken(PyTypeObject *type)
{
    // Return the token (memory layout ID) if the heap type
    // contains, 'NULL' otherwise.
}

[encukou (Petr Viktorin)]

I don’t know what you mean by “matches the given token except NULL”.

PyType_CheckLayoutExact looks redundant; you can do PyType_GetToken(type) == token instead.

Similarly, PyType_CheckLayout(type, token) would be equivalent to PyType_GetBaseByToken(type, token, NULL), if we allow calling it with NULL.

[neonene]

Thanks for the suggestions. Reduced to two helper functions. Also, reconsidered the Py_tp_token slot. Specifically, it will not have a particular offset, which PyType_GetSlot() will not support, raising an error.

Objects/typeslots.inc
// old proposal
{-1, offsetof(PyHeapTypeObject, ht_token)},
// new
{-2, Py_tp_token},  // use only the left

---

void *PyType_GetToken(PyTypeObject *type)

Return a token if the heap type contains, NULL otherwise.

• The token is a memory layout ID (void pointer) to identify the class. You can store the preferred one through PyType_FromMetaclass(), if you know that:

  • The pointer outlives the class, so it’s not reused for something else while the class exists.
  • It “belongs” to the extension module where the class lives, so it won’t clash with other extensions.

  You can specify the pointer by enabling the Py_tp_token slot:

  PyType_Slot foo_slots[] = {
      ...
      {Py_tp_token, &pointee_in_the_module},
  }
  

  As a default option, the Py_TP_USE_SPEC identifier (NULL) is available with which a heap type holds the spec pointer passed to PyType_FromMetaclass().

  // Be careful when the spec is dynamically created
  {Py_tp_token, Py_TP_USE_SPEC},
  

  To disable the feature, remove the slot.

  NOTE: PyType_GetSlot() does not recognize Py_tp_token.

---

int PyType_GetBaseByToken(PyTypeObject *type, void *token,
                          PyTypeObject **result)

This function always succeeds (like PyType_IsSubtype).
Return true if the type or its superclasses have a valid token that is also the same as the given one.

If the result argument is not NULL, set *result to the new reference to the first found class, or set it to NULL if not found.

[encukou (Petr Viktorin)]

@neonene

it will not have a particular offset, which PyType_GetSlot() will not support, raising an error

If disabling PyType_GetSlot was necessary, something like -2 would be the way to go. But I don’t think we need to disable PyType_GetSlot here.
In fact, we could not add PyType_GetToken, and tell people to use PyType_GetSlot. (GetSlot does have issues, but I don’t think GetToken solves them…)

@neonene

This function always succeeds

Generally, new API functions should be allowed to raise exceptions, so that (among other things) they can emit deprecation warnings in the future.

[neonene]

@encukou

In fact, we could not add PyType_GetToken,

When we miss PyType_GetToken, what about extending PyType_GetBaseByToken()? Some static types and objects (e.g. Py_None) could be passed to the token argument with other meaning:

PyTypeObject *type_token;
PyType_GetBaseByToken(type, Py_None, &type_token);
if ((void*)type_token == &foo_spec) {
    ...
}

---

Applied the suggestion:

int PyType_GetBaseByToken(PyTypeObject *type, void *token,
                          PyTypeObject **result)

Find a class whose token is valid and equal to the given one, from the type and superclasses.

• If found, set *result to the new reference to the first type and return 1.
• If not found, set *result to NULL and return 0.
• On error, set *result to NULL and return -1 with an exception.
• The result argument accepts NULL if you need only the return value.

The token is a memory layout ID to identify the class. You can store the preferred one in a heap type through PyType_FromMetaclass(), if you know that:

• The pointer outlives the class, so it’s not reused for something else while the class exists.
• It “belongs” to the extension module where the class lives, so it won’t clash with other extensions.

For the entry, enable the Py_tp_token slot:

PyType_Slot foo_slots[] = {
    ...
    {Py_tp_token, &pointee_in_the_module},
}

The slot accepts NULL via the Py_TP_USE_SPEC identifier, with which a heap type holds the spec pointer passed to PyType_FromMetaclass().

// Be careful when the spec is dynamically created
{Py_tp_token, Py_TP_USE_SPEC},

To disable the feature, remove the slot.

---

void *PyType_GetToken(PyTypeObject *type)

Return a token if the heap type contains, NULL otherwise.

[neonene]

@neonene

• The result argument accepts NULL if you need only the return value.

We could also propose an exact check here:

• The result argument accepts NULL if you verify only the type.

[encukou (Petr Viktorin)]

@neonene

When we miss PyType_GetToken, what about extending PyType_GetBaseByToken()?

I think PyType_GetSlot(type, Py_tp_token) should give you the token.

If we do have a dedicated GetToken function, it should be:

int PyType_GetToken(PyTypeObject *type, void *result);
// on error: set *result to NULL, set an exception, return -1
// if there's no token: set *result to NULL, return 0
// otherwise: set *result to the non-NULL token, return 1

The PyType_GetBaseByToken API looks good to me!

[neonene]

I have posted a reference implementation on the bug tracker.

PyType_GetToken() may be more likely to be inlined on PGO builds. If not, PyType_GetSlot() would suffice.

[neonene]

Would it be OK if I dropped PyType_GetToken() from the proposal? It seems that the function is not so beneficial at least for the performance, as far as I can see.

[encukou (Petr Viktorin)]

@neonene

Would it be OK if I dropped PyType_GetToken() from the proposal?

Definitely; I think PyType_GetSlot is enough :‍)

[encukou (Petr Viktorin)]

The proposal looks pretty good now!
Do you want to turn it into a short PEP, or just open an issue in Issues · capi-workgroup/decisions · GitHub ?

[neonene]

I’ve posted an updated proposal: Add Py_tp_token slot and PyType_GetBaseByToken function · Issue #34 · capi-workgroup/decisions · GitHub

[neonene]

This feature has been landed on 3.14. Thank you.