Add sys.setasthook() to allow to use a custom AST optimizer

[vstinner]

BPO	17515
Nosy	@brettcannon, @vstinner
Files	setasthook.patch enable_astoptimizer.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-18.11:05:53.988>
created_at = <Date 2013-03-22.00:58:11.302>
labels = []
title = 'Add sys.setasthook() to allow to use a custom AST optimizer'
updated_at = <Date 2015-03-28.14:41:42.333>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-03-28.14:41:42.333>
actor = 'brett.cannon'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-18.11:05:53.988>
closer = 'vstinner'
components = []
creation = <Date 2013-03-22.00:58:11.302>
creator = 'vstinner'
dependencies = []
files = ['29539', '29540']
hgrepos = []
issue_num = 17515
keywords = ['patch']
message_count = 7.0
messages = ['184932', '184936', '184971', '184978', '184986', '238416', '239459']
nosy_count = 2.0
nosy_names = ['brett.cannon', 'vstinner']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue17515'
versions = ['Python 3.4']

[vstinner]

I wrote an optimizer rewriting the AST to implement various optimizatoions:
https://bitbucket.org/haypo/astoptimizer

To be able to use it, or use any other kind of AST transformation, I propose to add a new sys.setasthook() function to set a custom AST hook.

Attached patch is a proof-of-concept. The patch should be improved: document the function and add write unit tests.

Prototype of the hook:

def asthook(ast, filename):
   # ...
   return new_ast

I don't know if we need to add a compiler option (like _ast.PyCF_ONLY_AST) to skip the hook.

[vstinner]

enable_astoptimizer.patch: example of usage of the new function, you can use it to try astoptimizer.

[brettcannon]

This is unnecessary. I added the source_to_code() method (http://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader.source_to_code) to loaders to explicitly handle this case.

[vstinner]

This is unnecessary.

Your change only concerns imports. eval() and compile() cannot be
hooked using your method.

2013/3/22 Brett Cannon <report@bugs.python.org>:

Brett Cannon added the comment:

This is unnecessary. I added the source_to_code() method (http://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader.source_to_code) to loaders to explicitly handle this case.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue17515\>

---

[brettcannon]

And that's fine as it allows for more explicit control over what gets optimized. Basically if there is a global hook directly into the compiler there needs to be more control over detecting it's being optimized, what to do about .pyo files, etc. At least at the loader level it's much more modular and controlled as to what will and will not get the optimizations.

So if you want to pursue this I think there needs to be more thought than just a global sys hook for this sort of thing.

[vstinner]

I'm not more interested to work on my astoptimizer project, and nobody looks to need sys.setasthook(), so I prefer to close the issue.

[brettcannon]

The closing of this issue inspired me to think of a better way to do this, and I think providing something in importlib.util that accepted a NodeTransformer as an argument to a function that compiled source files to bytecode files would fill this issue thanks to PEP-488. It would mean that nothing is done dynamically and everything is done offline, but that also means startup costs are gone in regards to bytecode generation and it gets people to actually think about optimizations of Python instead of it just being an afterthought.

importlib.util.compile(path: Union[str, pathlib.Path], transformer: ast.NodeTransformer = None, *, optimization: Union[str, int] = None) -> pathlib.Path