Repository navigation
python -d creates lots of tokenizer messages #89725
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-10-27.21:33:29.126>
created_at = <Date 2021-10-21.21:54:15.480>
labels = ['interpreter-core', '3.10']
title = 'python -d creates lots of tokenizer messages'
updated_at = <Date 2021-10-29.17:21:23.715>
user = 'https://github.com/malemburg'bugs.python.org fields:
activity = <Date 2021-10-29.17:21:23.715>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-27.21:33:29.126>
closer = 'pablogsal'
components = ['Parser']
creation = <Date 2021-10-21.21:54:15.480>
creator = 'lemburg'
dependencies = []
files = []
hgrepos = []
issue_num = 45562
keywords = ['patch']
message_count = 15.0
messages = ['404673', '404675', '404689', '404730', '404738', '404743', '404744', '404758', '404773', '405127', '405131', '405176', '405199', '405202', '405325']
nosy_count = 4.0
nosy_names = ['lemburg', 'lys.nikolaou', 'pablogsal', 'miss-islington']
pr_nums = ['29140', '29149', '29250', '29252', '29270', '29275']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45562'
versions = ['Python 3.10']
Activity
malemburg commentedon Oct 21, 2021
python3.9 -d:
Python 3.9.7 (default, Oct 21 2021, 20:51:19)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Loaded pyinteractive.py.
python3.10 -d:
Python 3.10.0 (default, Oct 21 2021, 23:13:32) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
line[1] = "\"\"\" pyinteractive.py\n" tok->done = 10
line[2] = "\n" tok->done = 10
line[3] = " This file is executed on interactive startup by Python\n" tok->done = 10
...
(not that in both cases a PYTHONINTERACTIVE script is loaded)
Is that intended ?
malemburg commentedon Oct 21, 2021
What's even worse is that those debug lines get written to stdout, not stderr.
pablogsal commentedon Oct 21, 2021
Yeah, those are expected in 3.10, but this should not appear in release versions (only on debug ones).
Python -d is normally used to debug the parser:
-d : turn on parser debugging output (for experts only, only works on
debug builds); also PYTHONDEBUG=x
miss-islington commentedon Oct 22, 2021
New changeset 86dfb55 by Pablo Galindo Salgado in branch 'main':
bpo-45562: Only show debug output from the parser in debug builds (GH-29140)
86dfb55
malemburg commentedon Oct 22, 2021
Yes, I know that (at the moment) it's only documented to work in the parser, but since Py_DebugFlag is a general purpose flag, this use could easily be extended to other parts of the interpreter as well, e.g. for parsing the command line or instrumenting the interpreter to collect debug stats.
In any case, thanks for the quick fix, Pablo.
pablogsal commentedon Oct 22, 2021
New changeset ae78ffd by Miss Islington (bot) in branch '3.10':
bpo-45562: Only show debug output from the parser in debug builds (GH-29140) (bpo-29149)
ae78ffd
pablogsal commentedon Oct 22, 2021
Thanks you for the catch!
malemburg commentedon Oct 22, 2021
I had left a comment on Github about using stderr instead of stdout, to make the output more consistent (other parser error messages go to stderr).
Note sure whether that's something you still want to change before closing the issue.
pablogsal commentedon Oct 22, 2021
Yeah, let me fix that in another PR
miss-islington commentedon Oct 27, 2021
New changeset 10bbd41 by Pablo Galindo Salgado in branch 'main':
bpo-45562: Print tokenizer debug messages to stderr (GH-29250)
10bbd41
pablogsal commentedon Oct 27, 2021
New changeset 038f452 by Miss Islington (bot) in branch '3.10':
bpo-45562: Print tokenizer debug messages to stderr (GH-29250) (GH-29252)
038f452
malemburg commentedon Oct 28, 2021
Hi Pablo,
I think you missed one instance:
print_escape(stdout, tok->cur, tok->inp - tok->cur);
Cheers
pablogsal commentedon Oct 28, 2021
New changeset cdc7a58 by Pablo Galindo Salgado in branch 'main':
bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270)
cdc7a58
malemburg commentedon Oct 28, 2021
Thanks, Pablo :-)
miss-islington commentedon Oct 29, 2021
New changeset d8ca47c by Miss Islington (bot) in branch '3.10':
bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270)
d8ca47c