What does the existence of a struct in a header file imply about the C-API

[markshannon]

BPO	43054
Nosy	@gvanrossum, @ronaldoussoren, @vstinner, @encukou, @markshannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-04.14:42:14.203>
created_at = <Date 2021-01-28.11:12:05.946>
labels = []
title = 'What does the existence of a struct in a header file imply about the C-API'
updated_at = <Date 2021-05-04.14:42:14.202>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2021-05-04.14:42:14.202>
actor = 'Mark.Shannon'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-04.14:42:14.203>
closer = 'Mark.Shannon'
components = []
creation = <Date 2021-01-28.11:12:05.946>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 43054
keywords = []
message_count = 6.0
messages = ['385851', '385854', '386853', '386856', '387806', '392904']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ronaldoussoren', 'vstinner', 'petr.viktorin', 'Mark.Shannon']
pr_nums = []
priority = 'normal'
resolution = 'postponed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43054'
versions = []

[markshannon]

Given the lack of explicit documentation on this subject, and differing opinions among core developers, I though it would be good to discuss how the existence of a struct in a header file constrains the C-API.

Original PR provoking this discussion: #24298

Suppose a header file, e.g. funcobject.h, contains a struct, e.g. PyFunctionObject, to what extent is that struct part of the API?

If a struct is present in a header file, there are three options for what means in terms of the API (that make sense to me).

1. That the struct is not part of the API and may be freely changed or deleted.
2. That the struct is produced, or initialized, by an API function, which implies that existing fields will continue to exist, but they can be reorder or added to.
3. That the struct is consumed by an API function, which implies that the struct must keep its exact shape, only adding fields if flags are present in the pre-existing fields to indicate the use of the extension.

PyTypeObject is an example of (3).

We should be able to infer which of the above cases applies, if not explicitly documented, for any struct.

Using PyFunctionObject in funcobject.h as an example:

There is no API function or macro that directly produces or consumes the struct, which would imply case 1. But, the struct is documented as the struct for Python functions, and PyFunction_Check() exists, which strongly implies that the following code is OK:

if (PyFunction_Check(obj)) {  
    PyFunctionObject *func = (PyFunctionObject *)obj;
    ...

which therefore implies that (2) applies.
(3) does not apply as there is no API that takes a PyFunctionObject struct as a parameter.

Similar logic can be applied to other parts of the API.

Rather than go through this tortuous analysis for all headers, it might be better to document which structs are part of the API.

[encukou]

When we're talking just about API, not the stable ABI (which contains only a few structs), reordering and additions should be fair game.

Rather than go through this tortuous analysis for all headers, it might be better to document which structs are part of the API.

Yup. I'm trying to do this for stable ABI/limited API, and it's unfortunately taking quite a long time.

[ronaldoussoren]

To channel Victor: Another thing to look into is to introduce accessors for struct fields in category 1 and 2 so that the struct can be made private in the future.

The difference between category 1 and 2 is sadly not very clear cut. Anything defined in public headers could be used 3th-party code.

In this particular instance there is no documentation for the fields of the struct, which may indicate that the struct is private. However, this struct is used by code outside of the stdlib and there are currently no accessors that can replace this usage.

[markshannon]

Why do you think the distinction between category 1 and 2 is not clear?

If the struct if not produced, or initialized, by an API function, then it cannot be accessed in the first place. If you can't access the struct in the first place, then you can't access its fields.

[ronaldoussoren]

Sorry about the slow response.

I misread your initial message, the distinction between (1) and (2) in your list is clear.

to (3): New fields can be added while maintaining API (but not ABI) compatibility as long as the default value is the default value for static initialisers (assuming the usual way we initialise structs in CPython). That is, adding a new field to PyTypeObject is ok, as long as it at the end and defaults to NULL or 0.

Especially with category 1 it is not entirely clear which structs are in that category. Is PyLongObject in this category? The struct is not documented, but has a name that seems to indicate that it is public. Likewise for PyTupleObject, where the shape of the struct is used by documented APIs but the shape of the strut itself is not documented.

BTW. For my own code I do directly access structs where necessary, even if they aren't documented. I totally expect that this will require adjustments for new Python releases (such as when the unicode representation changed).

[markshannon]

Thanks for the responses.
Probably nothing to do for now.