bpo-40334: Catch E_EOF error when the tokenizer returns ERRORTOKEN

[lysnikolaou]

An E_EOF error was only being caught after the parser exited before
this PR. There are some cases though, where the tokenizer returns
ERRORTOKEN and has set an E_EOF error (like when EOF directly
follows a line continuation character) which weren't correctly
handled before.

https://bugs.python.org/issue40334

[gvanrossum]

A nice improvement.

(I presume @ambv has already cut the alpha6 release though?)

[pablogsal]

(I presume @ambv has already cut the alpha6 release though?)

The tag is still not here:

https://github.com/python/cpython/releases

but who knows ¯_(ツ)_/¯

[gvanrossum]

The tag is still not here:

https://github.com/python/cpython/releases

Doesn't mean a thing -- I checked for that for something I wanted in the previous alpha, but IIUC the release managers first agree on a commit to build, then all build it locally, compare results, and finally push the tag upstream.

[ambv]

(I presume @ambv has already cut the alpha6 release though?)

Unfortunately so: https://github.com/ambv/cpython/tree/v3.9.0a6

[pablogsal]

Doesn't mean a thing -- I checked for that for something I wanted in the previous alpha, but IIUC the release managers first agree on a commit to build, then all build it locally, compare results, and finally push the tag upstream.

Ohhhh, that makes sense.

[ambv]

Yes, the point is: I put the tag on my tree, Steve and Ned build installers for their respective systems. It's not common but it did happen in the past that building those installers surfaced bugs and I had to move the tag. So I always wait with tagging until everything got cleanly built and tested (we also install the installers and run tests).

[pablogsal]

Yes, the point is: I put the tag on my tree, Steve and Ned build installers for their respective systems. It's not common but it did happen in the past that building those installers surfaced bugs and I had to move the tag. So I always wait with tagging until everything got cleanly built and tested (we also install the installers and run tests).

Cool! Thanks for the explanation :)