Document PyObject_CallFunction() special case more explicitly

[vstinner]

BPO	28977
Nosy	@vstinner, @serhiy-storchaka
Files	call_doc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-21.22:16:07.980>
created_at = <Date 2016-12-15.08:36:01.061>
labels = ['interpreter-core', '3.7', 'docs']
title = 'Document PyObject_CallFunction() special case more explicitly'
updated_at = <Date 2021-09-21.22:16:07.979>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-21.22:16:07.979>
actor = 'vstinner'
assignee = 'docs@python'
closed = True
closed_date = <Date 2021-09-21.22:16:07.980>
closer = 'vstinner'
components = ['Documentation', 'Interpreter Core']
creation = <Date 2016-12-15.08:36:01.061>
creator = 'vstinner'
dependencies = []
files = ['45908']
hgrepos = []
issue_num = 28977
keywords = ['patch']
message_count = 6.0
messages = ['283256', '283370', '283390', '283391', '283393', '402390']
nosy_count = 3.0
nosy_names = ['vstinner', 'docs@python', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue28977'
versions = ['Python 3.7']

[vstinner]

The PyObject_CallFunction() function has a special case if the format string is "O". See my email thread on python-dev:
https://mail.python.org/pipermail/python-dev/2016-December/146919.html

Serhiy wrote: "It is documented for Py_BuildValue(), and the documentation of PyObject_CallFunction() refers to Py_BuildValue()." which is true, but I would prefer to be more explicit to avoid bad surprises (see issues bpo-21209 and bpo-28920).

Attached patch modifies PyObject_CallFunction() doc to mention the "O" format special case in the .rst doc and the .h comment.

The documentation of PyObject_CallMethod() and _PyObject_CallMethodId() should also be modified, but I would prefer to wait for a first review on the added text before modifying all functions.

I proposed to only modify Python 3.7 because Include/abstract.h was a in a bad shape: I rewrote this file to make it easier to maintain, but only in Python 3.7, see issue bpo-28838.

[serhiy-storchaka]

"O" is not the only special case. "S" and "N" unpack a tuple argument too. "O&" converter should return a tuple, otherwise it is an error. All other format codes are illegal for single argument.

I would just deprecate this feature (in PyObject_CallFunction, not in Py_BuildValue). The behavior of PyObject_CallFunction with a single argument can be made more consistent and useful.

[vstinner]

Serhiy: "O" is not the only special case. "S" and "N" unpack a tuple argument too. "O&" converter should return a tuple, otherwise it is an error. All other format codes are illegal for single argument.

Oh, I didn't know. I should update my documentation.

Serhiy: I would just deprecate this feature (in PyObject_CallFunction, not in Py_BuildValue). The behavior of PyObject_CallFunction with a single argument can be made more consistent and useful.

PyObject_CallFunction(func, "O", arg) should call func(arg). I don't want the "O" format (and "S" and "N").

What do you mean exactly by deprecating the feature? Emit a warning if and only if te format string is "O" (or "S" or "N") and Py_BuildValue() returns a tuple?

I guess that PyObject_CallFunction(func, "(O)", arg) doesn't need to be modified, it already always call func(arg), even if arg is a tuple.

I am also tempted to fix PyObject_CallFunction() (and similar functions) rather than documenting the special case, but I dislike the idea of a deprecation.

Let's say that calling PyObject_CallFunction(func, "O", arg) where arg is a tuple would emit a DeprecationWarning and call func(*arg) in Python 3.7, but call func(arg) with no warning in Python 3.8. I dislike this path because developers would try to make the warning quiet in Python 3.7, for example use "(O)" format string, which is less obvious and looks like a hack to me.

More and more applications use Python bleeding edge (the development branch, default), and more and more developers quickly test their application on the new Python stable release. Maybe we can "simply" fix the behaviour of PyObject_CallFunction() with no transition period:

• Python 3.6: PyObject_CallFunction(func, "O", arg) calls func(arg), or func(*arg) if arg is a tuple
• Python 3.7: PyObject_CallFunction(func, "O", arg) always calls func(arg)

The special case was never documented. In my experience, almost no developer rely on this feature. Most developers don't expect the feature and so write code which doesn't work with tuple arguments.

[vstinner]

If the behaviour of PyObject_CallFunction(func, "O", arg) is changed, an application relying on the current behaviour would have to replace:

   res = PyObject_CallFunction(func, "O", arg);

with:

/* code working on any Python version */
if (PyTuple_Check(arg))
res = PyObject_Call(func, arg, NULL);
else
res = PyObject_CallFunction(func, "O", arg);

[serhiy-storchaka]

All other format codes are illegal for single argument.

I was wrong. PyObject_CallFunction() handles correctly a single non-tuple
argument. The problem is only with tuple result of Py_BuildValue().

What do you mean exactly by deprecating the feature? Emit a warning if and
only if te format string is "O" (or "S" or "N") and Py_BuildValue() returns
a tuple?

Emit a FutureWarning if the format string contains the single format code and
Py_BuildValue() returns a tuple. In 3.8 or 3.9 the behavior will be changed.

I dislike this path
because developers would try to make the warning quiet in Python 3.7, for
example use "(O)" format string, which is less obvious and looks like a
hack to me.

PyObject_CallFunctionObjArgs(func, arg, NULL) seems obvious and perhaps even
more efficient in all versions.

More and more applications use Python bleeding edge (the development branch,
default), and more and more developers quickly test their application on
the new Python stable release. Maybe we can "simply" fix the behaviour of
PyObject_CallFunction() with no transition period:

Ask on Python-Dev. I afraid this is too dangerous. We should have at least one
release with a FutureWarning.

Most developers don't expect the feature and so write
code which doesn't work with tuple arguments.

That is why we should add a FutureWarning. If the code doesn't fail by
accident, but just works incorrectly, this can be unnoticed. FutureWarning
could help to fix possible bugs in older Python versions.

[vstinner]

I don't know what to do with this old issue, so I prefer to simply close it.