__debug__ in compile(optimize=1)

[arigo]

BPO	22091
Nosy	@arigo, @vstinner, @serhiy-storchaka, @eryksun, @MojoVampire
PRs	bpo-27169: The __debug__ constant is now optimized out at compile time. #4880 [3.6] bpo-27169: The __debug__ constant is now optimized out at compile time. (GH-4880) #4882

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-15.11:28:34.891>
created_at = <Date 2014-07-27.11:32:00.282>
labels = ['interpreter-core', 'type-bug', '3.7']
title = '__debug__ in compile(optimize=1)'
updated_at = <Date 2017-12-15.11:28:34.888>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2017-12-15.11:28:34.888>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-15.11:28:34.891>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2014-07-27.11:32:00.282>
creator = 'arigo'
dependencies = []
files = []
hgrepos = []
issue_num = 22091
keywords = ['patch']
message_count = 9.0
messages = ['224119', '224120', '224126', '266770', '308377', '308379', '308380', '308382', '308384']
nosy_count = 6.0
nosy_names = ['arigo', 'vstinner', 'SilentGhost', 'serhiy.storchaka', 'eryksun', 'josh.r']
pr_nums = ['4880', '4882']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22091'
versions = ['Python 3.6', 'Python 3.7']

[arigo]

The documentation of the built-in compile() function is not 100% clear but I think it says that giving the "optimize=1" argument turns "__debug__" to false in the compiled code ( https://docs.python.org/3.5/library/functions.html?highlight=compile#compile ). It doesn't work this way in practice, though:

    python3.5
    >>> exec(compile("print(__debug__)", "exec", "exec", optimize=1))
    True

I'd recommend to fix either the documentation or the source code.

[SilentGhost]

I can reproduce your example on 3.4, but for the comparison:

>> exec(compile("if __debug__: print(42)", "exec", "exec", optimize=1))

>>> exec(compile("if __debug__: print(42)", "exec", "exec", optimize=0))
42

So, it's not as straightforward as one might imagine.

[eryksun]

If __debug__ were referenced from the code object's co_consts instead of checking locals, globals and builtins (LOAD_NAME), then optimize=1 would work consistently for a given code object.

Currently in 3.4.1:

    >>> dis.dis(compile("if __debug__: x = 1", "", "exec", optimize=0))
      1           0 LOAD_CONST               0 (1)
                  3 STORE_NAME               0 (x)
                  6 LOAD_CONST               1 (None)
                  9 RETURN_VALUE

    >>> dis.dis(compile("if __debug__: x = 1", "", "exec", optimize=1))
      1           0 LOAD_CONST               0 (None)
                  3 RETURN_VALUE

    >>> dis.dis(compile("x = __debug__", "", "exec", optimize=0))
      1           0 LOAD_NAME                0 (__debug__)
                  3 STORE_NAME               1 (x)
                  6 LOAD_CONST               0 (None)
                  9 RETURN_VALUE

    >>> dis.dis(compile("x = __debug__", "", "exec", optimize=1))
      1           0 LOAD_NAME                0 (__debug__)
                  3 STORE_NAME               1 (x)
                  6 LOAD_CONST               0 (None)
                  9 RETURN_VALUE

With the current design, an exec can override builtins.__debug__ in globals or locals:

    >>> exec("print(__debug__)", globals(), {"__debug__": "spam"})
    spam

[MojoVampire]

I just opened bpo-27169: "__debug__ is not optimized out at compile time for anything but `if:` and `while:` blocks" as a more general case of this bug.

This bug covers inconsistent behavior, but ultimately, it's caused by incomplete optimization: `__debug__` doesn't act as a compile time constant outside of `if:` and `while:` blocks, so the behavior is inconsistent for all other uses in the `compile` case where `optimize` doesn't match the main interpreter session, and inefficient for all other uses whether or not `compile` is involved.

If bpo-27169 is resolved as I suggested (or in some other similar way), it would also resolve this bug.

[serhiy-storchaka]

New changeset 3325a67 by Serhiy Storchaka in branch 'master':
bpo-27169: The __debug__ constant is now optimized out at compile time. (bpo-4880)
3325a67

[serhiy-storchaka]

Is it worth to backport this change?

[vstinner]

Is it worth to backport this change?

It's an optimization, usually the answer is no.

The change is short and can be seen as a bugfix, so it's up to you, it's ok to backport to 3.6, and maybe even 2.7, if it's not too hard to write the backport.

[serhiy-storchaka]

New changeset b82da9e by Serhiy Storchaka in branch '3.6':
[3.6] bpo-27169: The __debug__ constant is now optimized out at compile time. (GH-4880) (bpo-4882)
b82da9e

[serhiy-storchaka]

The backport to 3.6 is straightforward, but backporting to 2.7 would require larger changes.