Replace uuid ctypes usage with an extension module.

[gustavo]

BPO	20519
Nosy	@amauryfa, @abalkin, @pitrou, @vstinner, @ned-deily, @bitdancer, @meadori, @serhiy-storchaka
PRs	bpo-11063, bpo-20519: avoid ctypes and improve import time for uuid #3796
Superseder	bpo-11063: Rework uuid module: lazy initialization and add a new C extension
Files	gc.py: test program uuid-no-ctypes.diff issue20519_10941v2.diff uuid.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-28.13:29:10.292>
created_at = <Date 2014-02-05.11:52:23.220>
labels = ['ctypes', 'type-feature', '3.7']
title = 'Replace uuid ctypes usage with an extension module.'
updated_at = <Date 2017-09-28.21:03:08.306>
user = 'https://bugs.python.org/gustavo'

bugs.python.org fields:

activity = <Date 2017-09-28.21:03:08.306>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-28.13:29:10.292>
closer = 'vstinner'
components = ['ctypes']
creation = <Date 2014-02-05.11:52:23.220>
creator = 'gustavo'
dependencies = []
files = ['33925', '33932', '40865', '40874']
hgrepos = []
issue_num = 20519
keywords = ['patch']
message_count = 17.0
messages = ['210306', '210316', '210320', '210322', '210328', '210331', '210335', '210346', '251509', '251510', '253493', '253494', '253511', '253598', '303219', '303233', '303280']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'gustavo', 'pitrou', 'vstinner', 'ned.deily', 'r.david.murray', 'meador.inge', 'serhiy.storchaka']
pr_nums = ['3796']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '11063'
type = 'enhancement'
url = 'https://bugs.python.org/issue20519'
versions = ['Python 3.7']

[gustavo]

If you try the attached program, you will find that for every iteration the uuid.uuid4() call generates objects that contain reference cycles and need the help of the garbage collector. This is not nice. If I make the ctypes module not able to import, then no garbage is generated.

This problem appears in 2.7, 3.3, and 3.4, at least.

[bitdancer]

I'm not sure that this is a real problem, but have you identified what the cycle is?

[gustavo]

Well, this isn't a big problem, but I have an application that needs to run with the GC disabled, since it causes pauses of a couple of seconds each time a full collection runs (we have a few million objects allocated). I will run the GC only once every 3 hours. So it would be nice if this uuid call didn't generate cycles.

No, I didn't identify the cycle. All I know is that the garbage below is produced. If the ctypes module is unavailable, uuid still works but doesn't generate the garbage.

>>> gc.garbage
[(<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>]

[gustavo]

I have narrowed it down to one line of code:
ctypes.create_string_buffer(16)

That is enough to create 7 objects that have reference cycles.

[<class 'ctypes.c_char_Array_16'>, {'__module__': 'ctypes', '__doc__': None, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, 'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, 'value': <attribute 'value' of 'c_char_Array_16' objects>, '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>}, (<class 'ctypes.c_char_Array_16'>, <class '_ctypes.Array'>, <class '_ctypes._CData'>, <class 'object'>), <attribute '__weakref__' of 'c_char_Array_16' objects>, <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, <attribute '__dict__' of 'c_char_Array_16' objects>]

So maybe the bug is in ctypes itself, not the uuid module.

[bitdancer]

Yes, I was pretty sure it was in cytpes, from looking at the UUID code.

[gustavo]

Regardless, if you don't mind, take this patch for Python 3.5 to avoid ctypes, at least in the Linux case (I don't have Windows to test). Creating a proper extension module is safer and really not that hard...

[bitdancer]

Thanks. This looks like a good idea, but I'll leave it to someone with more experience with this module to review it.

Let's change this issue to an enhancement request for uuid instead. If someone wants to work on the cycle-in-ctypes problem they can open a new issue.