Interactivity suffers?

[bitprophet]

When running run("python"):

• >>> print 'hm' prints the next line's prompt + "hm" on the same line.
  • Implies mixing of stdout/err poorly still; using print >>sys.stderr, 'hm' correctly deals w/ newlines
• Trying to hit up-arrow for shell history prints ^[[A (control chars) instead of interacting with readline.

Neither of these problems exist when pty=True. I feel like this is the same as in Fabric and any other solution where a pty is not the default behavior. At the very least we should make this explicit in the docs, probably by making an FAQ.

[geertj]

Should pty=True be the default behavior?

[bitprophet]

@geertj - Fabric waffled on this in the past (started out with pty=False, then since 1.0 has been pty=True) and it's been problematic either way. The breakdown, IIRC:

• Pty enabled: impossible to differentiate stdout and stderr (big!), init scripts often silently fail (less big, but still annoying).
• Pty 'disabled' (aka "actually talking to the subprocess" :)): programs written to assume a real interactive terminal (many! :() frequently misbehave.

It comes down to average use case: if you're using this kind of tool for its programmatic benefits, you're likely to want distinct stdout/stderr, which means pty=False is cleaner and more obvious.

But if you're using it as a Make/Rake/etc replacement, you'll expect it to behave more "shelly", which here means pty=True because we can't hook the process up to your terminal's pipes (or we lose ability to capture any output at all, perform output hiding, etc.)

I can't say offhand which is more likely in the intended userbase; my gut is honestly leaning towards the make/etc replacement, meaning pty=True, and practically it would also mean less of a break with Fabric 1.x (Fabric 2.x will build on Invoke). OTOH I value the ability to become "purer" with this kind of rewrite.

---

As things stand this ticket is here so I make sure the above duality is documented and in peoples' face first thing if they have problems; idea being that since neither choice is perfect, go with the cleaner one (pty=False) and require folks to identify truly interactive invocations & slap pty=True on them.

Will probably poll other smart people before I go 1.0 and see if they have angles I'm not seeing.

[bitprophet]

This may be outdated now, can't remember when exactly we got hiding+capturing working with pty=True but it definitely does now (and may do so even more cleanly post-#235). Also wondering if I was conflating the pipe hookup with the fact that pexpect is simply using the pty module (i.e., both 'versions' of our command execution are technically using shells, so the difference truly is in the use of a PTY or not.)

[bitprophet]

Some possible hints in the (stdin-focused) fcntl additions from this Paramiko ticket.

[bitprophet]

Some notes about some of this are in #289 since that ticket's implementation changed more of how streams are handled.

[bitprophet]

#361 notes that the use cases they're having issues with worked better under Fabric, but I have a followup question for @hjc1710:

Re: Fabric handling it, did you mean local or run? Fabric's local was "just subprocess" so it behaved better on this front, but was significantly worse an API (no ability to intercept the streams and capture at the same time, etc). If you meant run then that is more interesting because run, at a high level, is similar to Invoke's run re: all streams being directly handled.

[hjc]

In answer to your question, @bitprophet, my example of Fabric handling this better was using local and not run (I linked the repo I used in my Asciinema, which is the exact code run).

It makes since that the API's are significantly different, because I distinctly remember capture=True from Fabric, not having that print output, and then having to mirror my output myself.

As stated, #361 gives my primary use case, which is I want to wrap all my unit test running in invoke, and using interactive debuggers when Invoke is wrapping your unit tests is... pretty difficult right now.

Thanks for looking into this and let me know if there's any way I can help or test!

[bitprophet]

Adding a note here that my 'real complex and also a thing I want to be able to launch with Invoke pretty often' use case, vim, falls under this - specific notes from #289 are in this comment and subsequent.

[bitprophet]

Also, it's clear that (if possible with current architecture) there should be an option to completely circumvent the "smart passthrough/stream mirroring+capturing" behavior in some situations - e.g. to do what subprocess' default behavior is re: hooking up user's terminal pipes directly to the subprocess'.

This isn't ideal because it's only possible with truly local commands (remote ones cannot do this) but practicality needs to beat purity some of the time.

So while I very much want this generally solved, a partway workaround would be e.g. run('vim', capture=False), or direct=True, or similar.

[bitprophet]

Poking a little more at "why isn't our mirroring working quite right for vim"...

Don't think I recorded this before but I remember it happening before, and it's still a thing now: hitting up/down arrow acts like I'm hitting "A" and/or "B", and even stranger, it can get into this "delayed" state where switching directions is delayed by 1 keypress.

E.g. up-up-up-up ... will record stdin reads of 'A' over and over. Then 'down' will...still record a stdin read of 'A'. But then a subsequent 'down' will record 'B', and then all subsequent 'down' are 'B'. Vice versa for going back to 'up' - 1st 'up' still records 'B', only 2nd thru Nth record 'A'.

Suspect this has to do with what my terminal emulator (Terminal.app) does for arrow keys, and that might provide a clue re: our stdin read and/or write (as well as maybe stdout/err) doing wrong things, e.g. the "one byte at a time" approach. (Which is another knob to try conditionally turning, even though I still don't know of a graceful fix for the "must read SOME hardcoded chunk of stdin at a time because we can't afford to block forever if it's all done" problem.)

[bitprophet]

OK, part of the issue is ready_for_reading isn't always working right:

• Load vim session
• Hit down arrow
• Down arrow evaluates to ^[OB (generally)
  • Of note, this also explains the 'AB' crap - combining various sequences of <Esc>, O and [AB] in vim will, of course, write out individual lines of A or B (because O in normal mode inserts a newline above cursor & enters insert mode)
• What I see logged, is that the ^[ (rather, \x1b gets read from stdin (and written to proc stdin), but the OB don't show up - at all - until the next keypress.
• And that next keypress also includes stdin reads up to the subsequent Escape character (which handily explains the 'delay' I see - each read is basically a 'backwards' character, with the crap after Escape followed by the next/current keypress' Escape).

So the question is why ready_for_reading isn't seeing the post-Escape bytes; if we can solve that it might make a lot of this stuff work better.

---

The main loop in this case is:

• Runner.handle_stdin is the thread/controller
• It context-manages character_buffered around a while True loop that calls Runner.read_our_stdin.
• read_our_stdin calls the nonblocking ready_for_reading function on stdin, returning None if that is False, or a single read byte otherwise.
• ready_for_reading is using select.select in this case, and the select call seems to be what is problematic, as it's not returning stdin as often as I think it should be.

Debug output of pressing down-arrow twice in a row after a vim session has loaded is below.

Note that I am only debugging when select returns non-empty readers list; I did check to make sure it's being called, and returning an empty readers list, when keys aren't pressed. That is of course super noisy so I then turned it off.

<press down arrow>

DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: '\x1b'
DEBUG:invoke:...and decoded: u'\x1b'
DEBUG:invoke:stdin written: '\x1b'
DEBUG:invoke:stdout read: '\x07'

<everything stops here until I press down-arrow again...>

DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: 'O'
DEBUG:invoke:...and decoded: u'O'
DEBUG:invoke:stdin written: 'O'
DEBUG:invoke:stdout read: '\x1b[?25l\x1b[1;40r\x1b[m\x1b[1;1H\x1b[L\x1b[1;41r\x1b[41;1H\x1b[K\x1b[1;1H\x1b[?12l\x1b[?25h'
DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: 'B'
DEBUG:invoke:...and decoded: u'B'
DEBUG:invoke:stdin written: 'B'
DEBUG:invoke:stdout read: '\x1b[?25lB\x1b[?12l\x1b[?25h'
DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: '\x1b'
DEBUG:invoke:...and decoded: u'\x1b'
DEBUG:invoke:stdin written: '\x1b'
DEBUG:invoke:stdout read: '\x08'

Condensed into just the stdin-read & stdout-read related bits:

<press down arrow>

DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: '\x1b'
DEBUG:invoke:stdout read: '\x07'

<everything stops here until I press down-arrow again...>

DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: 'O'
DEBUG:invoke:stdout read: '\x1b[?25l\x1b[1;40r\x1b[m\x1b[1;1H\x1b[L\x1b[1;41r\x1b[41;1H\x1b[K\x1b[1;1H\x1b[?12l\x1b[?25h'
DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: 'B'
DEBUG:invoke:stdout read: '\x1b[?25lB\x1b[?12l\x1b[?25h'
DEBUG:invoke:select.select came back with reads [<open file '<stdin>', mode 'r' at 0x101b7b0c0>]
DEBUG:invoke:stdin read: '\x1b'
DEBUG:invoke:stdout read: '\x08'

So for some reason, select is only returning stdin in the readers list once initially, but does it three times on each subsequent keypress (i.e.: what I expected it to do on the first, and every, such keypress involving 3 bytes), leaving us in this shifted state.

What I don't recall clearly is the exact semantics of select and whether it acts as a solid "there's crap in your stdin buffer you could be reading" flag or if there's nuance.

[bitprophet]

I can recreate the same select behavior with a simple client program, like sleep 600, which rules out any sort of bizarro terminal capacity setting vim and friends might be doing. (Not that I expected such was the culprit, but, troubleshooting 101...).

Only real difference is that the input is appearing as ^[[B instead of ^[OB, but the actual behavior - select returning ready only on the initial ^[, and then on 2nd keypress, seeing [B^[ - is 100% identical.

Rummaging around docs, google, manpages etc hasn't helped me a ton so far re: pinning down exactly what select means when it thinks a given FD is "ready". As far as I recall this ought to all be pretty basic IPC/network like pipe bullshit where the keypress dumps bytes into my stdin pipe, that pipe is now holding (in this case) 3 bytes, and I assume select should be returning a ready-read status until all 3 bytes have been read.

Double-checked what I saw before re: the loop getting back a definitive "nope, nothing to read from stdin", and yes, it's definitely the case.

Wondering if this is ye olde buffering issues with Python processes & pipes...?