Better platform.processor support

[jaraco]

BPO	35967
Nosy	@malemburg, @gpshead, @jaraco, @pitrou, @miss-islington
PRs	bpo-35967: Refreshed platform module with native processor resolvers #12230 bpo-35967 resolve platform.processor late #12239 bpo-35967: Baseline values for uname -p #12824 bpo-35967: Make test_platform.test_uname_processor more lenient to satisfy build bots. #19544 bpo-35967: Skip test with uname -p on Android #19577

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2022-03-20.21:10:53.429>
created_at = <Date 2019-02-11.14:29:23.506>
labels = ['3.9']
title = 'Better platform.processor support'
updated_at = <Date 2022-03-20.23:42:53.725>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2022-03-20.23:42:53.725>
actor = 'yan12125'
assignee = 'jaraco'
closed = True
closed_date = <Date 2022-03-20.21:10:53.429>
closer = 'jaraco'
components = []
creation = <Date 2019-02-11.14:29:23.506>
creator = 'jaraco'
dependencies = []
files = []
hgrepos = []
issue_num = 35967
keywords = ['patch']
message_count = 33.0
messages = ['335220', '335372', '337454', '337475', '337480', '337481', '337483', '337485', '337487', '337488', '337499', '337500', '337503', '337509', '337511', '337513', '337517', '337518', '337519', '338081', '340176', '340209', '366537', '366542', '366543', '366551', '366572', '366594', '366718', '368765', '368803', '369200', '415633']
nosy_count = 5.0
nosy_names = ['lemburg', 'gregory.p.smith', 'jaraco', 'pitrou', 'miss-islington']
pr_nums = ['12230', '12239', '12824', '19544', '19577']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35967'
versions = ['Python 3.9']

[jaraco]

or: Unable to implement 'uname' on Python due to recursive call
or: platform.uname() should avoid calling uname in a subprocess

In this issue, I stumbled across a strange and somewhat unintuitive behavior. This project attempts to supply a uname executable implemented in Python, so that such functionality could be exposed on any platform including Windows.

What I found, however, was that because the stdlib platform module actually invokes sh -c uname -p during most any call of the module (

cpython/Lib/platform.py

Line 836 in 9db56fb

processor = _syscmd_uname('-p', '')

), attempting to use that functionality on a system where uname is implemented by Python (and on the path), will probably fail after a long delay due to infinite recursion.

Moreover, the only call that's currently invoking uname in a subprocess is the processor resolution, which I suspect is rarely used, in part because the results from it are inconsistent and not particularly useful.

For example, on Windows, you get a detailed description from the hardware: 'Intel64 Family 6 Model 142 Stepping 9, GenuineIntel'

On macOS, you get just 'i386'.

And on Linux, I see 'x86_64' or sometimes just '' (in docker).

To make matters even worse, this uname -p call happens unconditionally on non-Windows systems for nearly any call in platform. As a result, it's impossible to suppress the invocation of uname, especially when functionality like pkg_resources and its environment markers is invoked early.

I suggest instead the platform module should (a) resolve processor information in a more uniform manner and (b) not ever call uname, maybe with something like this.

At the very least, the uname call should be late-bound so that it's not invoked unconditionally for rarely-used information.

After some period for comment, I'll draft an implementation.

[pitrou]

Your proposal sounds fine to me. You could fall back on platform.machine() instead of calling uname explicitly.

[malemburg]

As the documentation says, the API is intended as fairly portable implementation of the Unix uname helper across platforms. It's fine to redirect this directly to e.g. /proc output instead of using the executable, but in whatever you do here, the output of platform.uname() needs to stay compatible to what the function returned prior to such a change, which usually means: to the output of the uname helper on a system.

Could you please check that on most systems, the output remains the same ?

Thanks.

[jaraco]

It won't be possible in general to emit what the function returned before, as uname is a symbolic reference to an arbitrary executable, which can vary by platform and release and local environment.

What I might be able to do is find the implementation of "uname" and see if there's a way to get the value from the same source. I did find what I believe is the canonical source.

I'll explore if those calls can be translated to Python.

[jaraco]

The first call I see in that routine is to "sysinfo", but the signature of that function doesn't match what I find in the man pages for that function. So that function must be coming from elsewhere.

[malemburg]

Thanks. It would be good to do some before/after tests on popular
platforms, e.g. a few Linuxes, MacOS, Windows.

[jaraco]

Aha! It seems the 'sysinfo' call is for Solaris: https://docs.oracle.com/cd/E23823_01/html/816-5167/sysinfo-2.html

[jaraco]

Best I can tell, neither sysinfo nor sysctl are exposed in any way to Python, so it may not be possible to accurately load the processor information from those system calls without writing a wrapper in C. What I might try is to experiment with ctypes to see if I can prove the concept.

[jaraco]

Reading further, the 'sysctl' call seems to only be for BSD (https://www.freebsd.org/cgi/man.cgi?sysctl(3)). I could find the man page for sysctl for BSD but not Linux. There is a _sysctl in Linux (http://man7.org/linux/man-pages/man2/sysctl.2.html), but it's use is discouraged and it doesn't provide the necessary information.

Now I suspect that the aforementioned GNU coreutils 'uname' implementation is only for non-Linux systems, as none of the underlying system calls are relevant on Linux. I expect if one compiled that uname on Linux, 'uname -p' would emit 'unknown'.

Meaning I still don't know how to get a 'uname -p' result on Linux (without invoking uname -p).

[jaraco]

Hmm. But if I go to the Linux man page for uname (https://linux.die.net/man/1/uname) and follow the links to the source code, I end up at the same repository. So maybe the BSD man page is suitable for Linux. I'll work from that assumption for now.

[jaraco]

After fussing with sysctl for a while, I'm fairly confident that one can't use sysctl on Linux reliably (https://stackoverflow.com/a/55066774/70170). I'll keep digging to see if I can find another implementation of uname that's used on Linux.

[jaraco]

This answer is extremely helpful. uname -p isn't available on Linux except Fedora and late versions of Debian that apply the patch.

This lack of consistency means that platform.uname().processor and thus platform.processor() is an inherently unreliable interface.

[jaraco]

Correction on last comment: s/Debian/Ubuntu/