__debug__ is not optimized out at compile time for anything but `if:` and `while:` blocks

[MojoVampire]

BPO	27169
Nosy	@arigo, @nascheme, @serhiy-storchaka, @eryksun, @MojoVampire
PRs	bpo-27169: The __debug__ constant is now optimized out at compile time. #4880 [3.6] bpo-27169: The __debug__ constant is now optimized out at compile time. (GH-4880) #4882 bpo-32365: Fix a reference leak when compile __debug__. #4916 [3.6] bpo-32365: Fix a reference leak when compile __debug__. (GH-4916) #4918

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2017-12-15.11:26:53.815>
created_at = <Date 2016-05-31.18:50:58.224>
labels = ['interpreter-core', 'type-bug', '3.7']
title = '__debug__ is not optimized out at compile time for anything but `if:` and `while:` blocks'
updated_at = <Date 2017-12-18.13:11:58.271>
user = 'https://github.com/MojoVampire'

bugs.python.org fields:

activity = <Date 2017-12-18.13:11:58.271>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2017-12-15.11:26:53.815>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2016-05-31.18:50:58.224>
creator = 'josh.r'
dependencies = []
files = []
hgrepos = []
issue_num = 27169
keywords = ['patch']
message_count = 7.0
messages = ['266769', '266771', '266780', '308378', '308383', '308550', '308552']
nosy_count = 6.0
nosy_names = ['arigo', 'nascheme', 'SilentGhost', 'serhiy.storchaka', 'eryksun', 'josh.r']
pr_nums = ['4880', '4882', '4916', '4918']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27169'
versions = ['Python 3.6', 'Python 3.7']

[MojoVampire]

Issue bpo-22091 points out a quirk in the compile function and use of the debug "constant" causing inconsistent behavior when the optimize level of the compile call differs from that of the main interpreter; debug in an if or while statement is compiled out, but all other uses load it dynamically (at runtime), so in a mixed environment (interpreter at optimization=0, compile at optimize=2), you get non-obvious behavior.

This behavior appears to be a consequence of debug being handled by a special case for the if and while statements in compile.c that statements of that form to be compiled out, but not for similar constructs, e.g. a if __debug__ else b and __debug__ or a are always evaluated at runtime, whether or not compile is involved. The expr_constant function here https://hg.python.org/cpython/file/fd0ac7ba091e/Python/compile.c#l3542 is responsible for this (and only called in the same file, for if and while statements).

I'm not sure I understand the peephole optimizer, but if it can operate recursively (that is, an initial replacement of A->B where B could be optimized from B->C is optimized in a subsequent pass, turning all uses of A to C eventually), it seems like the "correct" solution would be to piggyback on optimizations for True and False, by having the peephole optimizer replace LOAD_NAME/LOAD_GLOBAL for debug with an appropriate LOAD_CONST, True or False, based on the compile environment. This would fix this bug (making debug evaluate in the compile call, so the environment when the compiled code is executed doesn't matter), and it would optimize all the other cases that the current special cases for if and while don't cover by letting the recursive pass optimize them out the same way uses of literal True and False is optimized, so uses of ternary expressions, non-if/while boolean operations, and all other operations using debug are optimized using a constant (and possibly optimized out of existence) without needing to independently maintain separate optimizations all over the codebase for debug. Might also allow the removal of the explicit special case for debug in compile.c for if and while, simplifying that code a bit.

This would fix the problem from bpo-22091 and also make __debug__ reliably useful for "optimization", since all uses of it would "compile out" to optimized runtime code, where right now only two cases do so.

Does this seem reasonable?

Note: I added the nosy list from bpo-22091 here, since that bug is really just a special case of this one for compile only.

[serhiy-storchaka]

I think that the complete solution of this problem is making __debug__ a named constant like None, True, False.

This needs changing the grammar of Python.

[MojoVampire]

That would also work. The argument I'd give in favor of performing a pass that replaces it with a literal True or False is that you don't have update as many places, don't have to worry about missing a place, and you don't have to decide if __debug__ is a reference to True or False, or a new object entirely.

It's just too easy to miss a case where __debug__ should be special and not notice (because optimizations aren't heavily tested for specific byte code outputs or anything), where a missed optimization for the True or False constant is much less likely to go unnoticed.

[serhiy-storchaka]

New changeset 3325a67 by Serhiy Storchaka in branch 'master':
bpo-27169: The __debug__ constant is now optimized out at compile time. (bpo-4880)
3325a67

[serhiy-storchaka]

New changeset b82da9e by Serhiy Storchaka in branch '3.6':
[3.6] bpo-27169: The __debug__ constant is now optimized out at compile time. (GH-4880) (bpo-4882)
b82da9e

[serhiy-storchaka]

New changeset bd6ec4d by Serhiy Storchaka in branch 'master':
bpo-32365: Fix a reference leak when compile __debug__. (bpo-4916)
bd6ec4d

[serhiy-storchaka]

New changeset 5659743 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-32365: Fix a reference leak when compile __debug__. (GH-4916) (bpo-4918)
5659743