Custom error messages when print & exec are used as statements

[ncoghlan]

BPO	21669
Nosy	@gvanrossum, @ncoghlan, @glyph, @alex
Files	custom_builtin_syntax_errors.diff: Custom syntax errors for print and exec used as statements issue21669_custom_error_messages.diff: Simpler approach based on a heuristic in SyntaxError.init issue21669_custom_error_messages_v2.diff: Also handle the compound statement cases

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-16.09:52:14.130>
created_at = <Date 2014-06-05.15:07:19.025>
labels = ['type-feature']
title = 'Custom error messages when print & exec are used as statements'
updated_at = <Date 2014-08-22.11:00:35.507>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2014-08-22.11:00:35.507>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-16.09:52:14.130>
closer = 'ncoghlan'
components = []
creation = <Date 2014-06-05.15:07:19.025>
creator = 'ncoghlan'
dependencies = []
files = ['35491', '35497', '35510']
hgrepos = []
issue_num = 21669
keywords = ['patch']
message_count = 13.0
messages = ['219816', '219822', '219854', '219855', '219880', '219890', '219892', '219932', '220048', '220077', '220707', '220708', '225657']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ncoghlan', 'glyph', 'alex', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21669'
versions = ['Python 3.5']

[ncoghlan]

I realised my experiment with supporting implicit calls could potentially be used as the basis for a patch that reported more specific error details when "print" and "exec" were used as statements, so I went ahead and updated it to do so.

The initial patch has a failure in test_source_encoding - this appears to be a general issue with moving syntax errors from the parser (which reliably sets the "text" attribute on the raised SyntaxError) to the AST compiler (which sets that to None when running from a string compiled directly from memory rather than from a file on disk).

I've also only flagged this as a patch for 3.5 - I can't think of a way to do it without changing the language grammar to allow the error to be generated in a later stage of the compilation process, and that's not the kind of thing we want to be doing in a maintenance release.

[gvanrossum]

I'm sorry, but I find this way too intrusive, and a little risky too (I'm not sure how to verify even that the new parser accepts exactly the same set of programs as the old version).

I would much prefer a solution to this particular issue along the lines of a heuristic based on detecting whether the line where the syntax error occurs starts with the token 'print' or 'exec' followed by anything except a left parenthesis.

Such a heuristic could also be added to 3.4.

[ncoghlan]

As in, putting something either in the SyntaxError constructor or else in
the parser code that emits them? I like that - the fact my initial approach
broke a test was rather concerning, and a change purely on the error
handling side should be much safer.

[gvanrossum]

Yes, something like that. Don't change the grammar, just hack the heck out of the error message.

[ncoghlan]

Heuristic based approach that just does a fairly simple check for the syntax error text starting with "print " or "exec " when the text doesn't contain a left parenthesis.

This will still miss a few cases where the left parenthesis is inside a larger expression (like a string, list or dict), but that extra check avoids false triggering on cases like "print (a.)".

[gvanrossum]

Nice! I put it through a bit of a torture test and found a few odd corners. E.g. it doesn't catch this:

    if 1: print 42

nor this:

    if 1:
        print 42

nor this:

    def foo():
        print 42

I also notice that if the printed expression starts with a unary + or -, it is not a syntax error but a type error. But I don't think we should try to do anything about that.

[gvanrossum]

I also found some amusing false positives (syntax errors that weren't valid print statements in Python 2):

print [/
print /
print )    # but not "print)" !
print]

None of these matter though. Perhaps more concerning is how many things are valid syntax, despite making little sense:

print [42]
print

Still I like the idea -- even if it only catches 50% of all print statements that would still be a huge win. (And I think it's probably closer to 80%.)

[ncoghlan]

Updated patch with the heuristics factored out into a helper function, with a more detailed explanation and additional logic to handle compound statements.

>>> def foo():
...     print bar
  File "<stdin>", line 2
    print bar
            ^
SyntaxError: Missing parentheses in call to 'print'

It's still just basic string hackery, though. The one liner handling, for example, relies on the fact that ":<whitespace>print " and ":<whitespace>exec " are going to be uncommon outside Python 2 code being ported to Python 3, so it just looks for the first colon on the line and checks from there, without worrying about slice notation or dicts.

[glyph]

Just my 2¢ here: rather than debating cases in the abstract, it would be interesting to 'pip install' a couple of popular 2.x-only packages and see if the error message is an improvement.

My experience is that learners don't hit this so much by writing their own code wrong, but by loading a dependency with incorrect metadata on the wrong Python. (Which suggests to me that a URL in the error message telling you how to download a different version of Python would be very helpful as well.)

[ncoghlan]

My main aim here is to offer a custom error message that can be looked up
in an internet search (likely ending up at a Stack Overflow answer - which
we could create in advance of the 3.4.x release that includes this change).

That answer can then explain the various reasons this error can come up,
like:

• following a Python 2 tutorial in Python 3
• running a Python 2 script in Python 3
• attempting to install a Python 2 only dependency on Python 3

And possible solutions like:

• if it's your own code, changing the print statements to be compatible
  with both Python 2 & 3 by using string formatting and surrounding
  parentheses (& print("") for blank lines)
• looking for a Python 3 tutorial instead
• using Python 2 instead of Python 3
• looking for an alternative package that runs on Python 3

It's never going to be possible to fit all that into an error message,
hence why I consider "can be looked up in an internet search more easily
than the generic 'invalid syntax' error message" to be the most important
characteristic of the custom error message. I don't expect the exact
wording to really matter all that much.

[python-dev]

New changeset 2b8cd2bc2745 by Nick Coghlan in branch '3.4':
Issue bpo-21669: Special case print & exec syntax errors
http://hg.python.org/cpython/rev/2b8cd2bc2745

New changeset 36057f357537 by Nick Coghlan in branch 'default':
Merge issue bpo-21669 from 3.4
http://hg.python.org/cpython/rev/36057f357537

[ncoghlan]

I went ahead and committed the v2 patch, as I think it's already an improvement over the generic message. Any further significant tweaks to the heuristics or changes to the error messages can be handled in separate issues.

[ncoghlan]

For the record, I just posted the Q&A reference to Stack Overflow: https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python/