Skip to content

bpo-37140: Fix StructUnionType_paramfunc() #15612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
vstinner merged 3 commits into from
Aug 30, 2019
Merged

bpo-37140: Fix StructUnionType_paramfunc() #15612

vstinner merged 3 commits into from
Aug 30, 2019

Conversation

@vstinner
Copy link
Member

@vstinner vstinner commented Aug 30, 2019 •
edited by bedevere-bot
Loading

Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(del) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.

https://bugs.python.org/issue37140

@vstinner
bpo-37140: Fix StructUnionType_paramfunc()
2579e56 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
@vstinner vstinner mentioned this pull request Aug 30, 2019
Closed
@vstinner
Copy link
Member Author

vstinner commented Aug 30, 2019

I wrote an unit test. I made sure that the unit test fails without the fix:

FAIL: test_pass_by_value_finalizer (ctypes.test.test_structures.StructureTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/master/Lib/ctypes/test/test_structures.py", line 449, in test_pass_by_value_finalizer
    self.assertEqual(finalizer_calls, [])
AssertionError: Lists differ: ['called'] != []

First list contains 1 additional elements.
First extra element 0:
'called'

- ['called']
+ []

Without the fix, Test.del() is called once internally during the call to the C function ("func()"), and then a second time when the object is destroyed (s = None).

serge-sans-paille
serge-sans-paille reviewed Aug 30, 2019
View reviewed changes
Modules/_ctypes/_ctypes.c
(PyObject *)Py_TYPE(self), new_ptr);
copied_self->b_needsfree = 1;

obj = (&StructParam_Type)->tp_alloc(&StructParam_Type, 0);
Copy link
Contributor

@serge-sans-paille serge-sans-paille Aug 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/obj/arg_obj/ to increase readability ?

Copy link
Member Author

@vstinner vstinner Aug 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose "obj" name to get "parg->obj = obj" which looks nice to me :-)

Lib/ctypes/test/test_structures.py
s = None
support.gc_collect()
self.assertEqual(finalizer_calls, ["called"])

Copy link
Contributor

@serge-sans-paille serge-sans-paille Aug 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<3 the test case

@vstinner vstinner merged commit 96b4087 into python:master Aug 30, 2019
@vstinner vstinner deleted the ctypes_struct_param branch August 30, 2019 12:30
@miss-islington
Copy link
Contributor

miss-islington commented Aug 30, 2019

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

@bedevere-bot
Copy link

bedevere-bot commented Aug 30, 2019

GH-15613 is a backport of this pull request to the 3.8 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 30, 2019
@vstinner @miss-islington
bpo-37140: Fix StructUnionType_paramfunc() (pythonGH-15612)
a1c6b17 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
(cherry picked from commit 96b4087)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington added a commit that referenced this pull request Aug 30, 2019
@miss-islington @vstinner
bpo-37140: Fix StructUnionType_paramfunc() (GH-15612)
17f61ed 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
(cherry picked from commit 96b4087)

Co-authored-by: Victor Stinner <[email protected]>
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@vstinner @lisroach
bpo-37140: Fix StructUnionType_paramfunc() (pythonGH-15612)
8c06825 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@vstinner @DinoV
bpo-37140: Fix StructUnionType_paramfunc() (pythonGH-15612)
082c315 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
@vstinner @websurfer5
bpo-37140: Fix StructUnionType_paramfunc() (pythonGH-15612)
18c22ef 
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
@ynkdir ynkdir mentioned this pull request Dec 12, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@serge-sans-paille serge-sans-paille serge-sans-paille left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@vstinner @miss-islington @bedevere-bot @serge-sans-paille @the-knights-who-say-ni