Add new C API functions PyPegen_*FromFileObject and use them in interactive mode

[lysnikolaou]

No description provided.

[gvanrossum]

There's still something wrong:

(v39) ~/cpygen$ ./python.exe -p new
Python 3.9.0a5+ (heads/pr/38:431f2e06d0, Apr  3 2020, 15:09:29) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Cannot read termcap database;
using dumb terminal settings.
>>> f() = 1
Segmentation fault: 11
(v39) ~/cpygen$ 

But with your last commit you're probably on the right track...

[lysnikolaou]

There is one problem here, whose solution is non-trivial.

In case one presses Ctrl-D in the interactive interpreter, the current parser returns NULL and signifies an error, which is then caught in pythonrun.c and the interpreter exits. No AST is generated in the process.

I haven't dug into the tokenizer code, but is there a token we can parse, which we could use to signify an EOF?

[gvanrossum]

The tokenizer (which we didn't change) is supposed to detect EOF, and it is supposed to return ENDMARKER (which is 0). There isn't any sequence of characters you can feed the tokenizer that also causes it to return this.

Is this related to the segfault I reported?

[lysnikolaou]

The tokenizer (which we didn't change) is supposed to detect EOF, and it is supposed to return ENDMARKER (which is 0). There isn't any sequence of characters you can feed the tokenizer that also causes it to return this.

Could we then change the grammar to accept an ENDMARKER as well when in Py_single_input mode? But even if we do, I'm having difficulties seeing how we are going to propagate that up to the caller, PyRun_InteractiveOneObjectEx in this case.

Is this related to the segfault I reported?

I'm currently investigating, but it does not appear so.

[gvanrossum]

I suppose you could accept ENDMARKER and special-case it somewhere else? It must be special-cased somewhere with the old parser right?

[lysnikolaou]

There's still something wrong:

(v39) ~/cpygen$ ./python.exe -p new
Python 3.9.0a5+ (heads/pr/38:431f2e06d0, Apr  3 2020, 15:09:29) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Cannot read termcap database;
using dumb terminal settings.
>>> f() = 1
Segmentation fault: 11
(v39) ~/cpygen$ 

But with your last commit you're probably on the right track...

This is related to the call to PyErr_ProgramTextObject in raise_syntax_error. PyErr_ProgramTextObject accepts a filename object and tries to create a FILE * from it, which returns NULL when we are in interactive mode.

[lysnikolaou]

I suppose you could accept ENDMARKER and special-case it somewhere else? It must be special-cased somewhere with the old parser right?

Yes, that's the way.

[gvanrossum]

Good enough.

There's one edge case where the old parser is still different: in interactive mode, type

>>> if 1:
... ^D

The old parser quits without an error (which is arguably wrong), the new parser produces a syntax error and then prompts again. I think this is actually in some sense an improvement, so let's see if anyone disagrees.

[lysnikolaou]

I think this is actually in some sense an improvement, so let's see if anyone disagrees.

Agreed.