Rework uuid module: lazy initialization and add a new C extension

[KeithDart]

BPO	11063
Nosy	@ncoghlan, @orsenthil, @pitrou, @vstinner, @tiran, @ned-deily, @merwok, @bitdancer, @methane, @skrah, @ambv, @berkerpeksag, @hynek, @vadmium, @serhiy-storchaka, @aixtools
PRs	bpo-11063: Create _uuid1 submodule #3795 bpo-11063, bpo-20519: avoid ctypes and improve import time for uuid #3796 bpo-5885: fix slow uuid initialization #3684 bpo-11063: Fix _uuid module on macOS #3855 bpo-11063: Add a configure check for uuid_generate_time_safe #4287 bpo-11063: Use more reliable way to detect if it exists #4343 bpo-11063: Handle uuid.h being in default include path #4565
Files	issue11063.patch: Fix issue Rework uuid module: lazy initialization and add a new C extension #55272 issue11063_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-11-08.20:10:08.173>
created_at = <Date 2011-01-29.17:10:59.772>
labels = ['easy', '3.7', 'library', 'performance']
title = 'Rework uuid module: lazy initialization and add a new C extension'
updated_at = <Date 2017-11-26.03:04:48.497>
user = 'https://bugs.python.org/KeithDart'

bugs.python.org fields:

activity = <Date 2017-11-26.03:04:48.497>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-11-08.20:10:08.173>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2011-01-29.17:10:59.772>
creator = 'Keith.Dart'
dependencies = []
files = ['20685', '29220']
hgrepos = []
issue_num = 11063
keywords = ['patch', 'easy']
message_count = 67.0
messages = ['127442', '127989', '127998', '128001', '128008', '128010', '128012', '128013', '128015', '128016', '128017', '128018', '128019', '128020', '128021', '178293', '178297', '182778', '182873', '182875', '264151', '264998', '265109', '265122', '265126', '265449', '303202', '303205', '303208', '303209', '303210', '303211', '303212', '303213', '303214', '303215', '303218', '303220', '303228', '303230', '303231', '303234', '303238', '303281', '303284', '303288', '303365', '303406', '303525', '303531', '303532', '303533', '303534', '303539', '303540', '303552', '303555', '305587', '305589', '305635', '305897', '305900', '305904', '305909', '305911', '306984', '306985']
nosy_count = 23.0
nosy_names = ['ncoghlan', 'orsenthil', 'kdart', 'pitrou', 'vstinner', 'christian.heimes', 'ned.deily', 'eric.araujo', 'Arfrever', 'r.david.murray', 'methane', 'skrah', 'nvetoshkin', 'lukasz.langa', 'knny-myer', 'nailor', 'Keith.Dart', 'berker.peksag', 'hynek', 'martin.panter', 'serhiy.storchaka', 'Michael.Felt', 'aixtools@gmail.com']
pr_nums = ['3795', '3796', '3684', '3855', '4287', '4343', '4565']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue11063'
versions = ['Python 3.7']

[KeithDart]

When the uuid.py module is simply imported it has the side effect of forking a subprocess (/sbin/ldconfig) and doing a lot of stuff find a uuid implementation by ctypes. This is undesirable in many contexts. It would be better to perform those tasks on demand, when the first UUID is actually requested. In general, imports should avoid unnecessary system call side effects. This also makes testing easier.

[nvetoshkin]

I think launching external tools like ifconfig and ipconfig can be avoided pretty easily. There are many recipes around the net how to use native API's.
About ctypes' horrible logic during find_library call - don't know yet.

[knny-myer]

With the attached patch the "heavy work" will be done on request, when calling uuid1() or uuid4() not on import.

I am working off from the py3k svn branch. Is it necessary to submit a separate patch for py2 branch?

[orsenthil]

Kenny, I don't see a problem with uuid is *imported*, it just creates a couple of STANDARD UUID class objects for use later. And this seems to just set the number and validates it. I don't see any subprocess calls performed. Perhaps you were referring to scenarios of using uuid1/uuid5 methods in mac and suggesting improvements to it by your patch?

[bitdancer]

If you do 'python -c "import uuid" under strace, _posixsubprocess is definitely loaded, and a pipe2 call is made.

Take a look at the code starting at (py3k trunk) line 418 (try:). That's where the weird stuff happens, which is what the patch is addressing.

Ken: thanks for working on this.

[pitrou]

Thanks for posting a patch! I have two comments:

• Have you run test_uuid? When I run it, it seems to go into an infinite loop somewhere and I need to kill the process.
• uuid should work even when ctypes is not available, so you can't just put an import statement at the top-level without a fallback

[nvetoshkin]

uuid should work even when ctypes is not available
A bit of offtopic: why can't we assume that ctypes is available?

[pitrou]

A bit of offtopic: why can't we assume that ctypes is available?

Because ctypes (or, actually, the libffi it relies on) needs specific low-level code for each platform it runs on, and not all platforms have such code.

Another reason is that ctypes is dangerous and some administrators might prefer to disable it (especially on shared hosting ala Google App Engine).

[knny-myer]

Thanks for pointing that out! I guess that is the reason you did the import
in a try block.

[nvetoshkin]

Maybe I understood and ctypes ImportError simply must be handled and fallbacked to something else. But there are only 3 ways of getting MAC address:

1. using popen
2. using ctypes and native calls
3. using C API and performing native calls in extension

And ctypes seems to be the best choice: it's portable across Python VMs (better that 3) and faster (better than 1).

[pitrou]

Maybe I understood and ctypes ImportError simply must be handled and
fallbacked to something else.

Indeed.

But there are only 3 ways of getting MAC address:

1. using popen
2. using ctypes and native calls
3. using C API and performing native calls in extension

And ctypes seems to be the best choice: it's portable across Python
VMs (better that 3) and faster (better than 1).

Perhaps, but doing without ctypes should still be possible, otherwise
it's a regression.

[kdart]

It's also possible using existing wrapped os system calls. One exaple is here: http://code.google.com/p/pycopia/source/browse/trunk/aid/pycopia/ifconfig.py

Although that one doesn't current support MAC addresses, but it could. The socket module also now support the netlink socket on Linux, so it shouldbe possible to use that for getting MAC address on Linux.