add sys.implementation

[ericsnowcurrently]

BPO	14673
Nosy	@loewis, @warsaw, @brettcannon, @amauryfa, @ncoghlan, @merwok, @briancurtin, @ericsnowcurrently
Files	sys_implementation.diff: implements sys.implementation and doc addition sys_implementation_2.diff: implements sys.implementation, a test, and a doc addition issue14673_as_module.diff issue14673_as_type.diff issue14673_as_structseq.diff issue14673_as_simple_namespace.diff issue14673_docs_and_tests.diff: documentation and tests for sys.implementation issue14673_as_simple_namespace_2.diff issue14673_docs_and_tests_2.diff issue14673_full.diff: updated patch including docs, tests, and types.SimpleNamespace issue14673_full_2.diff issue14673_full_3.diff: up-to-date complete patch with all tests passing issue14673_full_4.diff: updated docs for Amaury

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-06-04.20:23:14.644>
created_at = <Date 2012-04-26.05:16:11.202>
labels = ['interpreter-core', 'type-bug']
title = 'add sys.implementation'
updated_at = <Date 2012-06-04.21:49:57.609>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2012-06-04.21:49:57.609>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2012-06-04.20:23:14.644>
closer = 'loewis'
components = ['Interpreter Core']
creation = <Date 2012-04-26.05:16:11.202>
creator = 'eric.snow'
dependencies = []
files = ['25369', '25378', '25724', '25725', '25726', '25727', '25728', '25767', '25768', '25780', '25795', '25797', '25804']
hgrepos = []
issue_num = 14673
keywords = ['patch']
message_count = 41.0
messages = ['159357', '159365', '159377', '159390', '159393', '159394', '159415', '159425', '159435', '159441', '159444', '161223', '161632', '161679', '161960', '161963', '161965', '161969', '161981', '161982', '161983', '162047', '162049', '162081', '162098', '162099', '162100', '162118', '162119', '162125', '162127', '162186', '162215', '162217', '162227', '162261', '162262', '162279', '162289', '162297', '162299']
nosy_count = 11.0
nosy_names = ['loewis', 'barry', 'brett.cannon', 'amaury.forgeotdarc', 'ncoghlan', 'eric.araujo', 'Arfrever', 'brian.curtin', 'python-dev', 'sbt', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14673'
versions = ['Python 3.3']

[ericsnowcurrently]

(see thread at http://mail.python.org/pipermail/python-ideas/2012-April/014878.html)

This is a patch to add sys.implementation to Python (with doc addition). The main motivation is to have an explicit place to look for the name and version of the implementation for generating the import cache tag (a la PEP-3147).

[pitrou]

If it can contain a variable number of fields, I think it should be a dict rather than a tuple.

[merwok]

*version* is a version tuple, in the format of :data:`sys.version_info`,
which represents the version of the Python implementation, **not** the
version of the Python specification it implements.

If that version number is specific to each VM, then I’m not sure we should mandate a specific format.

[ericsnowcurrently]

@antoine - I wondered about that. In the end I got something up to start with.

The list of fields in sys.implementation may change over time, unlike sys.version_info, et al. However, during interpreter execution, I would expect that neither that list nor the contents would change. The variability would only be between implementations and between versions of those implementations.

A dict would imply to me that it might vary during interpreter execution. An immutable type makes it clear to me that it won't be changing. I'm fine with a dict, though, if you think that's better. Perhaps a dictproxy?

[ericsnowcurrently]

@Éric - that's a good point. I considered it for a little bit, but went with the quick and easy think to get it rolling.

There is a real benefit to mandating an API sys.implementation.version. importlib would use that version to calculate the tag to use for cached modules. Without a specified/uniform data structure, that job is trickier.

Having an explicit sys.implementation.cache_tag field would solve that, and the importlib code would check for that field first. However, I didn't want to start off with that as a "required" field, considering that only CPython would take advantage of module caches (as far as I know).

[ericsnowcurrently]

I'm going to write a PEP for this, after all. I want to make sure that it's easy to access, in one place, these points that are coming up and their resolutions.

[loewis]

If the main motivation for this is that importlib can use it, I fail to see the point to make it cross-implementation. Other implementations of Python may not use byte code files at all, or use different byte code syntaxes, or not use the marshal module to load byte code. So the part of importlib that deals with cached .pyc files will be specific to CPython, anyway - why make it a cross-implementation attribute?

[ericsnowcurrently]

Thanks for bringing this up, Martin.

sys.implementation is about having an implementation-specific location (hence sys) to consolidate explicit values concerning the implementation. It's partly about clarifying the run-time distinction between Python-the-language and Python-the-implementation.

'name' and 'version' are the initial fields in sys.implementation because they are the most obvious choices, and a good starting point. If sys.implementation were available now we'd use it in importlib. That's what has motivated me to pursue this.

You are correct that we can hard-code "cpython" in the bootstrap code to generate the cache tag. And perhaps that would be a better use of time. We can't just use platform.python_implementation(), however, since the frozen importlib can only use builtin modules.

I expect you're right that some of the importlib code (particularly w.r.t. cached modules) is CPython-specific. However, I'm sure Brett has tried to minimize that subset of the module. Maybe he knows if the other implementations have plans for using importlib or how it affects them. If not, I'll ask around.

Regardless, importlib is not the only motivator here. Right now platform.python_implementation() has to guess the implementation name. Having the different implementations specify the name explicitly in the sys module would be an improvement without a lot of work. Nick Coghlan has indicated that it would be useful for the test suite.

Ultimately, sys.implementation would become the source for information about a specific Python implementation. Others could expound the point better, but as time goes by this will be more important. This current effort would get the ball rolling. Consequent to the concerns so far, I'll try to get a PEP out in the next couple days.

[brettcannon]

So I'm w/ Antoine that a dict would be better so that the other VMs can add whatever they want into that object (e.g. Jython could specify what JVM they are running against) without causing confusion as to what some index means to each VM. The mutability of it is not important IMO.

As for the other implementations using importlib, they all plan to with (hopefully) no direct modification, but whether they support bytecode I don't know (I think Jython has talked about it, but who knows).

And pertaining to whether this is useful outside of importlib, I suspect it is. We already have 'jython' as an "OS" type to work around some differences. Adding this attribute would more clearly delineate when another VM is being used that might require working around some difference without overloading os.name (e.g. PyPy implementing something differently in their 1.7 release of Python 2.7 but is subsequently fixed in their 1.8 release).

[loewis]

When I added sys.subversion, people requested that it shall contain the CPython string. When sys._mercurial was introduced, it copied that. So there are plenty of ways already to figure out that it is CPython which you are using.

[ericsnowcurrently]

An updated patch using a dict. (FYI, I have the PEP up on python-ideas.)

[ericsnowcurrently]

FYI, the attached patches don't reflect the latest PEP. I'll get an updated patch up in the next day or two.

[ericsnowcurrently]

I've just attached 4 patches, one for each of the likeliest implementations. All 4 ran the test suite successfully.

In my mind, the "simple namespace" one is the best fit. However, it involves adding a new built-in type (though a private one). If that is too controversial or too much new stuff to iron out for 3.3, I'd be okay with any of the others. In reality I'd like to see that new namespace type added to Python, but that's a proposition I'll take up elsewhere. :)