Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[doko42]

BPO	26839
Nosy	@malemburg, @rhettinger, @doko42, @ncoghlan, @vstinner, @larryhastings, @matejcik, @ned-deily, @alex, @skrah, @vadmium, @ztane, @dstufft, @Lukasa, @tpetazzoni
Files	nonblocking-getrandom.diff: Patch to py_getrandom to use nonblocking system call, and associated plumbing. getrandom-nonblocking-v2.patch: Patch random.c to use nonblocking getrandom() getrandom-nonblocking-v3.patch: Patch random.c to use nonblocking getrandom() (cleaned-up version). getrandom_nonblocking_v4.patch nonblocking_urandom_noraise.patch no-urandom-by-default.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-06-11.08:46:07.323>
created_at = <Date 2016-04-24.19:04:13.945>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()'
updated_at = <Date 2016-06-15.20:00:44.118>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2016-06-15.20:00:44.118>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-06-11.08:46:07.323>
closer = 'larry'
components = ['Interpreter Core']
creation = <Date 2016-04-24.19:04:13.945>
creator = 'doko'
dependencies = []
files = ['42837', '42842', '43265', '43267', '43278', '43282']
hgrepos = []
issue_num = 26839
keywords = ['patch']
message_count = 172.0
messages = ['264121', '264122', '264126', '264258', '264265', '264267', '264270', '264271', '264284', '264289', '264292', '264303', '265427', '265430', '265452', '265477', '265481', '265485', '265496', '265500', '265549', '265555', '266216', '267455', '267504', '267511', '267537', '267539', '267546', '267550', '267554', '267571', '267608', '267609', '267610', '267611', '267612', '267614', '267616', '267617', '267621', '267623', '267624', '267625', '267626', '267627', '267628', '267629', '267630', '267631', '267632', '267633', '267634', '267635', '267636', '267637', '267638', '267640', '267642', '267643', '267644', '267645', '267648', '267650', '267654', '267656', '267660', '267661', '267663', '267664', '267665', '267666', '267667', '267668', '267669', '267670', '267671', '267672', '267673', '267674', '267675', '267676', '267677', '267678', '267679', '267680', '267681', '267682', '267684', '267685', '267686', '267687', '267688', '267689', '267690', '267693', '267694', '267695', '267696', '267699', '267705', '267707', '267709', '267710', '267711', '267712', '267715', '267716', '267718', '267720', '267721', '267723', '267725', '267726', '267728', '267729', '267730', '267731', '267735', '267737', '267739', '267740', '267741', '267745', '267746', '267749', '267750', '267751', '267752', '267803', '267804', '267805', '267806', '267807', '267808', '267809', '267810', '267811', '267812', '267813', '267815', '267816', '267817', '267818', '267819', '267823', '267825', '267831', '267836', '267837', '267846', '267850', '267853', '267855', '267856', '267857', '267863', '267873', '267887', '267890', '267893', '267897', '267898', '267913', '267914', '267939', '268018', '268201', '268591', '268593', '268627', '268629']
nosy_count = 19.0
nosy_names = ['lemburg', 'rhettinger', 'doko', 'ncoghlan', 'vstinner', 'larry', 'matejcik', 'ned.deily', 'alex', 'skrah', 'python-dev', 'martin.panter', 'ztane', 'dstufft', 'pitti', 'Lukasa', 'thomas-petazzoni', 'Colm Buckley', 'Theodore Tso']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26839'
versions = ['Python 3.5', 'Python 3.6']

[doko42]

[forwarded from https://bugs.debian.org/822431]

This regression / change of behaviour was seen between 20160330 and 20160417 on the 3.5 branch. The only check-in which could affect this is the fix for issue bpo-26735.

3.5.1-11 = 20160330
3.5.1-12 = 20160417

Martin writes:
"""
I just debugged the adt-virt-qemu failure with python 3.5.1-11 and
tracked it down to python3.5 hanging for a long time when it gets
called before the kernel initializes its RNG (which can take a minute
in VMs which have low entropy sources).

With 3.5.1-10:

  $ strace -e getrandom python3 -c 'True'
  +++ exited with 0 +++

With -11:
$ strace -e getrandom python3 -c 'True'
getrandom("\300\0209\26&v\232\264\325\217\322\303:]\30\212Q\314\244\257t%\206\"", 24, 0) = 24
+++ exited with 0 +++

When you do this with -11 right after booting a VM, the getrandom()
can block for a long time, until the kernel initializes its random
pool:

11:21:36.118034 getrandom("/V#\200^O*HD+D_\32\345\223M\205a\336/\36x\335\246", 24, 0) = 24
11:21:57.939999 ioctl(0, TCGETS, 0x7ffde1d152a0) = -1 ENOTTY (Inappropriate ioctl for device)

[ 1.549882] [TTM] Initializing DMA pool allocator
[ 39.586483] random: nonblocking pool is initialized

(Note the time stamps in the strace in the first paragraph)

This is really unfriendly -- it essentially means that you stop being
able to use python3 early in the boot process or even early after
booting. It would be better to initialize that random stuff lazily,
until/if things actually need it.

In the diff between -10 and -11 I do seem some getrandom() fixes to
supply the correct buffer size (but that should be irrelevant as in
-10 getrandom() wasn't called in the first place), and a new call
which should apply to Solaris only (#ifdef sun), so it's not entirely
clear where that comes from or how to work around it.

It's very likely that this is the same cause as for bpo-821877, but the
description of that is both completely different and also very vague,
so I file this separately for now.
"""

[doko42]

other issues fixed between these dates:

- Issue bpo-26659: Make the builtin slice type support cycle collection.
- Issue bpo-26718: super.__init__ no longer leaks memory if called multiple
  times.  NOTE: A direct call of super.__init__ is not endorsed!
- Issue bpo-25339: PYTHONIOENCODING now has priority over locale in setting
  the error handler for stdin and stdout.
- Issue bpo-26717: Stop encoding Latin-1-ized WSGI paths with UTF-8.
- Issue bpo-26735: Fix :func:`os.urandom` on Solaris 11.3 and newer when
  reading more than 1,024 bytes: call ``getrandom()`` multiple times with
  a limit of 1024 bytes per call.
- Issue bpo-16329: Add .webm to mimetypes.types_map.
- Issue bpo-13952: Add .csv to mimetypes.types_map.
- Issue bpo-26709: Fixed Y2038 problem in loading binary PLists.
- Issue bpo-23735: Handle terminal resizing with Readline 6.3+ by installing
  our own SIGWINCH handler.
- Issue bpo-26586: In http.server, respond with "413 Request header fields too
  large" if there are too many header fields to parse, rather than killing
  the connection and raising an unhandled exception.
- Issue bpo-22854: Change BufferedReader.writable() and
  BufferedWriter.readable() to always return False.
- Issue bpo-6953: Rework the Readline module documentation to group related
  functions together, and add more details such as what underlying Readline
  functions and variables are accessed.

[vstinner]

Python 3 uses os.urandom() at startup to randomize the hash function. os.urandom() now uses the new Linux getrandom() function which blocks until the Linux kernel is feeded with enough entropy. It's a deliberate choice.

The workaround is simple: set the PYTHONHASHSEED environment variable to use a fixed seed. For example, PYTHONHASHSEED=0 disables hash randomization.

If you use virtualization and Linux is not feeded with enough entropy, you have security issues.

I just debugged the adt-virt-qemu failure (...)

If you use qemu, you can use virt-rng to provide good entropy to the VM from the host kernel.

[vstinner]

See also the issue bpo-25420 which is similar but specific to "import random".

[vstinner]

The issue bpo-25420 has been closed as a duplicate of this issue.

Copy of the latest message:

msg264262 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-04-26 12:05

I still believe the underlying system API use should be fixed rather than all the different instances where it gets used.

getrandom() should not block. If it does on a platform, that's a bug on that platform and Python should revert to the alternative of using /dev/urandom directly (or whatever other source of randomness is available).

Disabling hash randomization is not a good workaround for the issue, since it will definitely pop up in other areas as well.

[skrah]

Hmm. Why does os.urandom(), which should explicitly not block, use a blocking getrandom() function?

This is quite unexpected on Linux.

[skrah]

Wow, it's by design:

" os.urandom(n)

Return a string of n random bytes suitable for cryptographic use."

``man urandom'':

"A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is
not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a crypto-
graphic attack on the algorithms used by the driver."

[vstinner]

"Hmm. Why does os.urandom(), which should explicitly not block, use a blocking getrandom() function? This is quite unexpected on Linux."

I modified os.getrandom() in the issue bpo-22181 to use the new getrandom() syscall of Linux 3.17. The syscall blocks until the Linux kernel entropy pool is *initialized* with enough entropy. In a healthy system, it must never occur.

To be clear: you get read 10 MB (or 1 GB or more) of random data using os.urandom() even if the entropy pool is empty. You can test:

• In a terminal 1, run "dd if=/dev/random of=random" to ensure that the entropy pool is empty
• In a terminal 2, run "while true; do cat /proc/sys/kernel/random/entropy_avail; sleep 1; done" to see that entropy pool is empty (or very low, like less 100 bytes)
• In a terminal 3, get a lot of random data using os.urandom(): ./python -c 'import os; x=os.urandom(1024*1024*10)'

=> it works, you *can* get 10 MB of random data even if the kernel entropy pool is empty.

Reminder: getrandom() is used to avoid a file descriptor which caused various issues (see issue bpo-22181 for more information).

Ok, now this issue. The lack of entropy is a known problem in virtual machine. It's common that SSH, HTTPS, or other operations block because because of the lack of entropy. On bare metal, the Linux entropy pool is feeded by physical events like interruptions, keyboard strokes, mouse moves, etc. On a virtual machine, there is *no* source of entropy.

The problem is not only known but also solved, at least for qemu: you must attach a virtio-rng device to your virtual machine. See for example https://fedoraproject.org/wiki/Features/Virtio_RNG The VM can now reads fresh and good quality entropy from the host.

To come back to Python: getrandom() syscall only blocks until the entropy pool is *initialized* with enough entropy.

The getrandom() syscall has a GRND_NONBLOCK to fail with EAGAIN if reading from /dev/random (not /dev/urandom) would block because the entropy pool has not enough entropy:
http://man7.org/linux/man-pages/man2/getrandom.2.html

IMHO it's a deliberate choice to block in getrandom() when reading /dev/urandom while the entropy pool is not initialized with enough entropy yet.

Ok, now the question is: should python do nothing to support VM badly configured (with no real source of entropy)?

It looks like the obvious change is to not use getrandom() but revert code to use a file descriptor and read from /dev/urandom. We will get bad entropy, but Python will be able to start.

I am not excited by this idea. The os.urandom() private file descriptor caused other kinds of issues and bad quality entropy is also an issue.

[skrah]

It is clear how /dev/urandom works. I just think that securing enough
entropy on startup should be done by the init scripts (if systemd still
allows that :) and not by an application.

[Unless the application is gpg or similar.]

[vstinner]

Since many years, Linux systems store entropy on disk to quickly feed the
entropy pool at startup.

It doesn't create magically entropy on VM where you start with zero entropy
at the first boot.

[skrah]

I did not claim that it magically creates entropy. -- Many VMs are throwaway test beds. It would be annoying to setup some entropy
gathering mechanism just so that Python starts.