DeprecationWarnings should be visible by default in the interactive REPL

[njsmith]

BPO	24294
Nosy	@terryjreedy, @ncoghlan, @vstinner, @rbtcollins, @ezio-melotti, @bitdancer, @njsmith, @asmeurer, @takluyver, @vadmium, @JimJJewett, @serhiy-storchaka, @The-Compiler, @jayvdb, @Carreau
Files	enable_deprecation_warnings_in_repl.patch: enable DepWarning in REPL enable_deprecation_warnings_in_repl_no_check_tty.patch: Enable deprecationWarnign in REPL without checking for tty.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-10-18.12:46:53.493>
created_at = <Date 2015-05-27.08:06:30.608>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'DeprecationWarnings should be visible by default in the interactive REPL'
updated_at = <Date 2018-10-18.12:46:53.491>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2018-10-18.12:46:53.491>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-10-18.12:46:53.493>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-05-27.08:06:30.608>
creator = 'njs'
dependencies = []
files = ['39612', '42500']
hgrepos = []
issue_num = 24294
keywords = ['patch']
message_count = 39.0
messages = ['244147', '244148', '244162', '244166', '244240', '244242', '244245', '244276', '244279', '244286', '244450', '244794', '244852', '245805', '245994', '248789', '248893', '248895', '248899', '248901', '248902', '248903', '248907', '249075', '249076', '249108', '249113', '249117', '249125', '249190', '249196', '257304', '261709', '263621', '306137', '306162', '306181', '306182', '327470']
nosy_count = 15.0
nosy_names = ['terry.reedy', 'ncoghlan', 'vstinner', 'rbcollins', 'ezio.melotti', 'r.david.murray', 'njs', 'Aaron.Meurer', 'takluyver', 'martin.panter', 'Jim.Jewett', 'serhiy.storchaka', 'The Compiler', 'jayvdb', 'mbussonn']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24294'
versions = ['Python 3.7']

[njsmith]

DeprecationWarning and PendingDeprecationWarning are invisible by default. The rationale for this is that they are only useful to people who are writing code, so they should not be shown to end-users who are merely running code.

If someone is typing stuff into the interactive REPL, though, and the code they type uses some deprecated functionality, then we should actually show them this warning. We know that the author is sitting right there. And they're probably going to take whatever they tested interactively and move it into a more permanent form and ugh.

This problem is particularly acute for packages that are primarily used through the interactive prompt, like numpy. I am tired of getting bugs like this: numpy/numpy#5919

The solution is simple: when entering interactive mode, the REPL should do something like:

warnings.filterwarnings("default", category=DeprecationWarning, module="__main__")
warnings.filterwarnings("default", category=PendingDeprecationWarning, module="__main__")

[njsmith]

I also filed a similar bug with ipython: ipython/ipython#8478

[vadmium]

I have learnt to run the interactive interpeter (and also most of my own scripts) with the -b -Wall options. But having these switched on automatically may not be a bad thing.

[serhiy-storchaka]

See discussion on Python-Ideas [1].

[1] http://comments.gmane.org/gmane.comp.python.ideas/32191

[ncoghlan]

+1 for showing DeprecationWarning by default (as these features may be going away in the next X.Y release of Python)

-0 for showing PendingDeprecationWarning by default (as these won't be going away until X.Y+1 at the earliest)

[njsmith]

*cough* You know, there's more to life than Python-X.Y.tar.gz :-). Not that I know how PendingDeprecationWarning is used in the wild. I've been thinking maybe we (numpy) should start using it for stuff that we want to discourage people from using (we know it was a bad idea or intrinsically broken or whatever), but don't yet have a full replacement to offer. It would help if there were some official guidance on what these things mean -- I can't find anything written down anywhere that even documents what you just said about how CPython proper uses them, so I imagine people have come up with all kinds of interpretations.

Anyway, my logic would be: when I am trying something out at the REPL, usually it is because I want to check how it works, to figure out what I should be doing in the real module code that I'm writing in the next window. When doing this, I definitely appreciate knowing that it will be removed and is being replaced -- it won't necessarily stop me from putting it into my code, but at least it's information that I can take into account.

I don't care a huge amount either way, though; if we decided to hide PendingDeprecationWarnings by default while showing DeprecationWarnings then numpy would just define a NumPyPendingDeprecationWarning subclass of DeprecationWarning and use that instead of PendingDeprecationWarning and all would work out fine :-).

[ncoghlan]

The difference between the two used to be clearer: prior to Python 2.7, PendingDeprecationWarning was hidden by default (and thus mainly only visible to folks testing with -Wall), while DeprecationWarning was visible by default.

We blurred the line between the two thoroughly when DeprecationWarning also became hidden by default, giving the status quo:

Test frameworks: both visible by default
Interactive REPL: both hidden by default
Non-interactive execution: both hidden by default

Splitting them in the interactive REPL case would restore a meaningful behavioural difference that can help pragmatically guide decisions as to which is more appropriate to use:

Test frameworks: both visible by default
Interactive REPL: DW visible by default, PendingDW hidden by default
Non-interactive execution: both hidden by default

The current unanswerable "How do my users interpret the difference between DW and PendingDW?" question would be replaced by the much simpler "Do I want folks using the interactive REPL to see this warning or not?"

[njsmith]

Okay, that sounds reasonable to me.

[terryjreedy]

For Idle, the addition could be made in current versions. Idle compiles user code in the idle process and ships it to the user process for execution. In particular, idlelib.run.Executive.runcode, line 351, is
exec(code, self.locals)
Am I to presume that a) the warnings that you want to display will escape the exec call and that b) they are currently being caught at the module level and c) that the filterwarnings line or lines would do what you want if added to run.main, such as at line 104? Do warnings from python code and user code get treated the same?

Please post test cases to enter by hand, both at the prompt and in the editor, to get system warnings for 2.7 and 3.4 or 3.5. Also, please copy the positive and negative examples you posted to python-ideas.

[njsmith]

There isn't really any magic in how warnings work. Basically someone calls warnings.warn(...), which is a regular Python function, and it gathers up some information about the specific warning message and calling context, checks a global variable ("warnings.filters") to decide how this message in this context should be handled, and then either does nothing, prints something to stderr, or raises an exception.

There are lots of warnings that are printed by default, and I'm sure IDLE is handling them fine already. It's just (Pending)DeprecationWarnings in particular that have an entry stuck into warnings.filters saying "please ignore these" (unless it gets overridden by something else).

So you just need to make sure that a filter is added to warnings.filters that says to treat DeprecationWarnings generated by the __main__ module using the "default" action. ("default" is the name of a specific way of handling warnings; for most types of warnings, the default handler is the one named "default", but for DeprecationWarning, the default handler is the one named "ignore". Obvious, right?)

So you just need to make sure to run the following line of code somewhere in the user process:

warnings.filterwarnings("default", category=DeprecationWarning, module="__main__")

Adjust as necessary if (a) you want to apply similar handling to PendingDeprecationWarning, (b) your user namespace has some value of __name__ that is different from "__main__".

Then to test, you can just type

warnings.warn("foo", DeprecationWarning)

at the prompt, and it should be printed.

Note that because the warnings module tries to suppress duplicate warnings (which is good), and it has a bug where it can't tell the difference between different lines of code at the REPL (this is bad -- see ipython/ipython#6611 , and there should probably be a python bug too but I haven't gotten around to filing it), then the *second* time you run that line of code in the same REPL, nothing will be printed. So the moral is just, when testing this, make sure you use a different warning message each time, or you'll get very confused.

[njsmith]

Recording this here so it doesn't get lost: Marc-Andre Lemberg suggested on python-ideas that for the builtin REPL, this should be enabled iff sys.stdin.isatty(). (I guess he is worried about 'cat script.py | python'.) I'm not really sure whether this falls on the useful or confusing side of the line myself.

[Carreau]

I gave that a shot.
Doing it "cleanly" in C as the warning module is initialized much earlier.
Though I'm not super used to CPython internals.
Doing just before the repl by using PyRun_SimpleString make the patch relatively small.

[njsmith]

For whatever it's worth as a non-core-developer, the patch looks good to me.