os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[jbeck]

BPO	25003
Nosy	@tim-one, @vstinner, @dstufft
Files	remove_get_entropy.patch urandom_solaris.patch getrandom_solaris.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2015-10-01.16:13:39.666>
created_at = <Date 2015-09-04.21:44:28.198>
labels = ['type-security', 'interpreter-core']
title = 'os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer'
updated_at = <Date 2015-10-01.16:13:39.665>
user = 'https://bugs.python.org/jbeck'

bugs.python.org fields:

activity = <Date 2015-10-01.16:13:39.665>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2015-10-01.16:13:39.666>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2015-09-04.21:44:28.198>
creator = 'jbeck'
dependencies = []
files = ['40397', '40412', '40437']
hgrepos = []
issue_num = 25003
keywords = ['patch']
message_count = 30.0
messages = ['249837', '250131', '250142', '250211', '250241', '250243', '250246', '250247', '250257', '250258', '250315', '250321', '250323', '250403', '250415', '250463', '250478', '250496', '250497', '250681', '250983', '250984', '250993', '250995', '250996', '252005', '252007', '252008', '252035', '252037']
nosy_count = 5.0
nosy_names = ['tim.peters', 'vstinner', 'python-dev', 'dstufft', 'jbeck']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue25003'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[jbeck]

A recent Solaris build upgrade resulted in a massive slowdown of a package operation (our package client is written in Python). Some DTrace revealed this was because os.urandom() calls had slowed down by a factor of over 300. This turned out to be caused by an upgrade of Python from 2.7.9 to 2.7.10 which included:

• Issue bpo-22585: On OpenBSD 5.6 and newer, os.urandom() now calls getentropy(),
  instead of reading /dev/urandom, to get pseudo-random bytes.

By adding ac_cv_func_getentropy=no to our configure options, we were able to get back the performance we had lost. But our security experts warned:

---

OpenBSD only has getentropy(2) and we are compatible with that.
Linux has both getentropy(2) and getrandom(2)
Solaris has getentropy(2) and getrandom(2).

The bug is in Python it should not use getentropy(2) for the implementation of os.urandom() unless it builds its own DRBG (Deterministic Random Bit Generator) around that - which will mean it is "caching" and thus only calling getentropy() for seeding very infrequently.

You can not substitute getentropy() for getrandom(), if you need randomness then entropy does not suffice.

They are using getentropy(2) correctly in the implementation of _PyRandom_Init().

I would personally recommend that the upstream adds os.getentropy() changes
os.urandom() to use getrandom(buf, sz, GRND_NONBLOCK) and os.random() to use
getrandom(buf, sz, GRND_RANDOM).

As it is the upstream implementation is arguably a security vulnerability since you can not use entropy where you need randomness.
---

where by "upstream" he meant "python.org". So I am filing this bug to request those changes. I can probably prototype this in the reasonable near future, but I wanted to get the bug filed sooner rather than later.

[vstinner]

Attached patch (for Python 3.4) removes code to use getentropy() in os.urandom().

[gvanrossum]

I got several long private emails from Theo De Raadt about this issue. I think the gist of it all is that most likely (a) the app most likely shouldn't be calling os.urandom() that often, and (b) Solaris getentropy() is apparently stunningly slow. Theo also seems to imply that "entropy" as a concept is debunked, and OpenBSD getentropy() and getrandom() are essentially the same thing except that the former's interface is faster (because it doesn't use a FD). So it is also possible that using getentropy() instead of getrandom() was unnecessary; possibly the speed difference would be noticeable via Python. Perhaps the getentropy() check can explicitly rule out Solaris (either at the autoconf level or in the random.c source code) if you prefer to keep the getentropy() call on OpenBSD.

[gvanrossum]

Also, Theo believes that our Mersenne Twister is outdated and os.urandom() is the only reasonable alternative. So we might as well keep it fast.

[tim-one]

Guido, you're clearly talking with someone who knows too much ;-) If we're using the Twister for _anything_ related to crypto-level randomness, then I'd appalled - it was utterly unsuitable for any such purpose from day 1. But as a general-purpose generator for non-crypto uses, it remains an excellent choice, and still a nearly de facto standard for "almost all" such purposes. There are general-purpose generators I like better now, but won't push in that direction until the Twister's "nearly de facto standard" status fades. Better to be a follower than a leader in this area.

[gvanrossum]

To Theo it's probably inconceivable that anyone would be using random numbers for anything besides crypto security. :-) His favorite is arc4random() (but he notes it's not based on RC4 anymore, but uses chacha -- I have no idea what any of that means :-). He did say userland PRNG a few times.

[dstufft]

(A)RC4 and ChaCha are just two stream ciphers that let you encrypt some data, they work by essentially producing a psuedo-random stream of data in a deterministic manner based off of a key, and than that is XOR'd with the data you want to encrypt. arc4random (ab)uses this and uses "real" entropy (e.g. randomness pulled from random noise on the network and such) as the "key" and then uses the psuedo-random stream of data as the values you get when you ask arc4random for some random data. The actual process is quite a bit more complex then that, but that's the basic gist.

Userspace PRNG's are not a very good idea for reasons better explained by an expert: http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

And yea, using MT for anything that needs a CSPRNG (that is, a Cryptographically Secure Psuedo Random Number Generator) is a real bad idea, because the numbers it outputs are not "really" random. I'm of a mind that the APIs should default to CSPRNGs (so random should default to SystemRandom) and using something like MT should be opt in via something like "UnsafeFastRandom) or something. That ship is almost certainly sailed at this point though.

[dstufft]

Oh yea, and (A)RC4 is broken and shouldn't be used for anything anymore, ChaCha is much better and is pretty great.

[vstinner]

Thread on python-dev: https://mail.python.org/pipermail/python-dev/2015-September/141439.html

[vstinner]

Perhaps the getentropy() check can explicitly rule out Solaris (either at the autoconf level or in the random.c source code) if you prefer to keep the getentropy() call on OpenBSD.

Ok, here is a patch implementing this option. It keeps getentropy() on OpenBSD for os.urandom(), but it disables getentropy() on Solaris for os.urandom().

I don't know if my py_getrandom() function calling syscall(SYS_getrandom, buffer, size, 0) works on Solaris. The flags are hardcoded, and I don't know if the <sys/syscall.h> include is enough to get the syscall() function. (Does Solaris uses the GNU C library?)

@jbeck: Can you please test this patch on the default branch of Python? Can you tell if the HAVE_GETRANDOM_SYSCALL check succeed on Solaris? (do you have "#define HAVE_GETRANDOM_SYSCALL 1" in pyconfig.h?)

The configure scripts tries to compile the following C program:

    #include <sys/syscall.h>

    int main() {
        const int flags = 0;
        char buffer[1];
        int n;
        /* ignore the result, Python checks for ENOSYS at runtime */
        (void)syscall(SYS_getrandom, buffer, sizeof(buffer), flags);
        return 0;
    }

[jbeck]

@Haypo: Yes, that patch works (applied to 3.5.0rc3; proxy problems are preventing me from updating my clone of the default branch at the moment) i.e., pyconfig.h shows:

#define HAVE_GETRANDOM_SYSCALL 1

To answer your other questions:

• Calling syscall(SYS_getrandom, buffer, size, 0) does work on Solaris.
• Our <sys/syscall.h> does declare syscall().
• Solaris does not use the GNU C library, but our own libc, which does
  support getrandom(3) as of Solaris 11.3 (which will be released in
  the near future) and Solaris 12 (which is in Beta but has a release
  date in the more distant future).

Prior to applying this patch, I had needed to tweak py_getrandom() to recognize EINVAL in addition to ENOSYS as sufficient reason to turn off getrandom_works. I still have that tweak in place, but have not yet tried backing it out to see if things still behave with your patch.

Let me know if there is any more information I can provide, and thanks for your attention to this issue.

[vstinner]

Prior to applying this patch, I had needed to tweak py_getrandom() to recognize EINVAL in addition to ENOSYS as sufficient reason to turn off getrandom_works.

In which case getrandom() fails with EINVAL?