Prompt/etc autoresponder

[bitprophet]

This represents porting Fabric 1.x's limited ability to autorespond to sudo prompts, to Invoke, and making it Better™.

For now, I've gone with a basic "when you see X, always respond with Y" functionality that can be easily configured via a {pattern: response} dict.

This isn't the most powerful implementation; e.g. one might want "only respond to patterns once", "respond to X pattern the first 3 times only", "respond to pattern X but only if you've already responded to pattern Y", etc etc.

However it's arguably more flexible than using e.g. pexpect (which requires you to anticipate exactly what prompts will happen, in what order; Invoke/Fabric use cases tend to have prompts appear arbitrarily) and I expect it will be enhanced in the future, perhaps by ripping out the responder aspects into their own class or something.

The functionality is already pretty much complete after a ~week of hacking, fucking up, getting confused, becoming less confused, and more hacking. I'm making this ticket just so I have something to link in the changelog & a checklist for the random minor tasks left to perform.

---

Left to do:

• Deal with recently-discovered funkiness in interactive stdin use case
• Make sure the basic functionality works for Fabric 2 as well as with local/subprocess
• Reinstate the Windows support for handle_input, which mirrors real stdin to the subprocess' stdin.
• Doublecheck whether we actually need Windows getch as per Handle stdin on Windows using msvcrt functions #304 (comment)
• Figure out whether to retain the sleep call in handle_input; git-blame the Fab 1 line and doublecheck the rationale. Probably do want it back tho, probably chews more CPU than it really needs to as-is.
• Reinstate the "print read stdin data to stdout when pty=False" behavior from fab 1 - it is necessary
• Ditto the "halt stdin mirroring while the local codebase is explicitly trying to prompt for shit it needs" functionality from Fab 1.
  • ...does this mean Fab 1 literally can't deal with users manually executing things like raw_input and getpass? Cuz if it can then that implies said halting/locking might not have been necessary? Or is a corner case?
  • See notes in desc of Convenient sudo() wrapper #294 - probably not going to go down the same road as in Fab 1 which means we wouldn't have to worry about this wrinkle.
• Centralize & normalize the encoding related noise, as per comments by @pfmoore in comments below.
  • Quite possibly via work/discussion being done over on Fixed a number of encoding issues (in run context) #274 and its linked tickets
• Try to replicate Pressing enter while running can expose sudo password fabric/fabric#1339 (both in Fabric 1, and here, and in Fabric 2)
• Figure out if it's worth implementing a(n easily disabled and well documented!) convenience "add \n to response values if not present" feature.
  • Ye olde "helpfulness can be evil"...
  • but in MY OWN TESTING I've fucked up and assumed an implicit \n multiple times already, so there's no way tons of users wouldn't make the same assumption.
  • Spun off as "Conveniently" append newline when autoresponding? #368
• Determine what's up with swallowed exceptions in the out/err/in threads when the main thread gets an eg KeyboardException. Feels like a thing we need to at least partially port from fab v1.
• Python 3 test pass so Travis doesn't get mad at me
• Ascertain why the test suite got slower again and how much of it's due to this branch
• Double-check the doc situation; may want to make a "feature list!" section on index page?
  • Explicit/detailed feature list in the docs #394
• Changelog entry

[bitprophet]

Looks like I traded autoresponding for interactive stdin; now, stdin from the terminal user a) doesn't actually appear in Invoke's sys.stdin (& thus to the subprocess, after being replayed) until Ctrl-D (EOF), and b) that input is echoed automatically by the local terminal and by the subprocess' stdout, at least in some cases (Python subprocess running raw_input yes, regular call to eg rm -i no).

Reasonably sure I've just neglected to port additional terminal manipulation bits from fabric v1.

Poking more:

• Noninteractive stdin works correctly, not really surprisingly, e.g. echo 'blah' | python a-script-using-invoke-run.py - no double echo, etc (and multiline input shows up with 1st line eaten by e.g. raw_input).
• Looked around fab v1 and sure 'nuff, we've been wrapping stdin reads with a "please be byte instead of line buffered" wrapper forever (using a combo of termios and tty libs).
  • The initial add of this was in fabric/fabric@b412733
  • No real info there besides "started using select". Meh.
  • It was made into a context manager, yet we only ever use it in the one place; reasonably sure there's no actual need to expose it as its own public method. (But having it as a contextmanager would save us some indent levels, so...yea.)

[bitprophet]

After a bunch of dumb futzing around, realized that we also need to read from stdin 1 byte at a time, as fab v1 does, for that to work. I'd actually been wondering earlier if changing the stdin reader to the same 'chunked' size as the output streams, was wise -- apparently now it isn't!

[bitprophet]

Diddled around some more; I now have things looking pretty good under Python 2:

• Can run a vim session reasonably well...
  • I can do editing, mode changes, navigation, etc
  • Can use plugins which redraw lots of stuff on the fly, like CtrlP (though trying to mash Enter to select the file doesn't work - but ^X to split-window on the file selected works).
  • Signals still need love. E.g. ^C which is what I tend to use in vim instead of Esc, isn't passed through, neither is ^] which is useful for columnar editing.
    • Though whether we transmit signals down the pipe or hoard them for ourselves has always been a nontrivial question to answer. Think there's tickets open for that. Offhand thinking we can afford to always submit downpipe and then also honor if they have meaning for us too...but this can wait. Alpha.
• Simple raw_input based prompts in subprocesses now work fine both via human interaction and the autoresponder.
• getpass based prompts also work (when pty=True, as noted in the FAQ now).

Naturally, almost none of this shit works anymore under Python 3, and it's not immediately clear why. Hoping Ye Olde You Gotta Wrap A List Around An Iterator or similar easily fixed bullshit.

EDIT: ah yes, it wants bytes, not str. Duh. Of course. Does remind me that I have an exception-swallowing problem...

[bitprophet]

Exception issue is pretty funny (and yet again there's echoes from Fabric v1 past & some parts of its present...):

• Main execution body spins off threads, waits for completion of subprocess, then joins the threads & checks them for stored exceptions.
• The exception-storing is effected by swallowing BaseException and storing sys.exc_info(). I don't remember exactly why we do this.
  • IIRC it's so one has the ability to see all exceptions that happen, instead of the first one to pop off "winning" and potentially hiding the rest.
  • But that seems mildly specious to me right now; sure, it's nice? But as we've seen, it means they get swallowed forever if anything prevents the stored exception from being dug up later...
• Normally, a subprocess will end of its own volition, so wait() completes, the threads are joined and the exceptions raise -- all good.
• But when the subprocess is waiting for input, it sits there -- forever. Our local process sits in wait() until killed, and the exception-raising never happens since it's performed later on.
• Because the bug(s) causing these exceptions have been happening in the stdin-handler, it means the user's attempt to satisfy the subprocess' prompt cannot succeed. Thus it's literally impossible to exit without just killing the process.

So, it's kind of a perfect-storm, but still annoying / has made hacking on this stdin functionality a PITA. I'm guessing the exception handling is there for a reason. EDIT: yea, a code comment mentions fabric/fabric#208, tl;dr "thread exceptions don't bubble up to the main thread, so they print out but cause the rest of the system to hang/get gummed up/time out". Makes sense.

I also note that this specific comment there is an eerie echo of our current problem.

---

For now, I think the best solution is to do something that seems sensible anyways - log the exceptions to the debug log when they occur, as well as saving for later raising. This means somebody paying close attention will see that it's going on.

I don't see any other way out of this that isn't reverting to the apparently-problematic "just let exceptions happen" behavior.

[bitprophet]

With that bug fixed, vim works better under Python 3 but there's still some oddball stuff going on at startup, some terminal chimes & refusal to honor the first few keystrokes; but after that, works as well as Python 2. Might poke, but at some point I need to exit this rabbit hole...

Simpler use cases like input and getpass now work totally fine, too.

[bitprophet]

Shamelessly borrowing (and tweaking into a contextmanager) the approach from https://github.com/graphite-project/carbon/blob/eeba61142edcae4af13dca03765188bb13a26c5d/lib/carbon/instrumentation.py#L47 for measuring CPU in an integration test - this way we can actually A) prove that adding an IO sleep has a measurable difference and B) protect against regressions.

Downside, of course, is that the % of a CPU core used for literally anything, will vary system to system, but the difference between sleeping and not sleeping (and just for the stdin mirroring at present) is currently an order of magnitude (0.98% usage vs 98.84% usage) so I'm thinking just setting the test to assume <2%, <5% or similar will suffice.

[bitprophet]

Checking on just how slow I can get that test to run while yielding measurable results (because while it is the integration suite, I'd still rather not have a test chew cud for 5 seconds when 1 second will do)...

Starting with an arbitrary run of 100 cycles of "print to stdout and stderr and sleep 0.1s" (which means running for about 10 seconds of course):

• With our stdin sleep enabled, CPU usage is: 0.91%, 0.71%, 0.73%
• With it disabled: 97.9%, 96.3%, 99.1%

Shrunk to 10 cycles or ~1s:

• Enabled: 1.17%, 1.26%, 1.33%
• Disabled: 99.5%, 99.5%, 99.6%

Probably unsurprising is that the longer runtime makes both figures a tiny bit lower (probably as it amortizes startup time or something).

I figure for now, just saying "fail if it took more than 5% CPU" should more than do us, and do that 10-cycle/1-second run.

[bitprophet]

While poking briefly at how these changes affect #263 I notice that vim is actually semi borked in Python 2 as well. Seems almost intermittent (which is why I didn't catch it originally) - some sessions there's a single terminal bell and then it works "well", others everything seems lagged by one character.

[bitprophet]

Under Python 2, at least, the initial console alarm seems caused by one byte of read/written stdin, despite zero manual interaction performed after launching Invoke: \x1b, aka the character encoded by hex 1b, aka ascii 27, aka: Escape. I certainly was't hitting Escape but that would yield a terminal bell in vim when it starts out and is in normal mode.

So, first, something is inserting an Escape, somewhere. Fun.

Second, when I hit a single keystroke afterwards - say, i for insert mode - I get another terminal bell and vim does not enter insert mode. On a subsequent i, insert mode occurs and things work pretty normally.

Debugging, I find a bunch more mirrored stdin occurs when I press i: one read/write byte action (due to how our loop is constructed) for each of: [>84;0;0ci (with the i being the character I really typed; e.g. if I repeat the process with a, it's [>84;0;0ca).

This looks distressingly familiar but I can't put my finger on it.

---

Normal troubleshooting steps to try and narrow down the source/cause of this mystery stdin:

• Changing vim to less ~/.vimrc doesn't exhibit any of this oddity; only my real keystrokes appear in the debug log.
• Ditto nano - everything works great, no extra input, things like ^X work, etc.
• So this behavior is specific to vim, or rather, something vim is doing (that presumably other apps might do, or at least, if I can figure it out it'll give a hint to what we're doing "wrong" here). E.g. perhaps this is its terminal autodetection at work, or something like that.
• vim /a/file behaves identically to vanilla, no-arguments vim
• vim -T xterm-256color (aka, try overriding terminal detection) - no dice, same deal
• :help terminal, under *raw-terminal-mode*, mentions that Vim sets the terminal into raw mode, sending strings t_ti and t_ks to the terminal. So I'm wondering if the data spat back is my actual terminal device or application responding to this automatically.
• Certainly, on startup, my debugging notices a whole slew of escape code laden gobbledygook coming out of stdout, even when one sees and accounts for the actual stuff being drawn (e.g. the left-hand column of tildes)

While snooping around vim's settings trying to see if I can lessen the amount of data spat out on startup, I notice other mention of terminfo settings, which are all ^[ followed by various characters. Seems to back up my suspicion that this data is terminfo-related junk. Going to educate myself further in this area.

[bitprophet]

Also of note: Fabric v1 doesn't actually have this particular problem; if one runs fab -H localhost --set="output_prefix=" -- vim, vim works right away with no fussing. Also worth investigating why, since at this point I thought we were doing pretty much everything Fabric v1 does wrt stdin mirroring.

[bitprophet]

Definitely not happening with Fabric, the only stdin received/mirrored is what I type, both on startup and on keypress.

Tried a few other obvious differences found from scanning code & output:

• Disabled the self.encode on stdout-writing in Invoke - noticed my debugging was showing Fabric writing regular strings and Invoke writing unicode strings. Figured perhaps this was somehow garbling those questions Vim asks of my terminal or something. No diff.
• Ran my shell env without tmux. No diff.
• Ran things in a Linux VM in its vmWare terminal (i.e.: not over ssh from my local terminal app); behavior was differentish yet similar:
  • No bell on startup (tho probably as no visual OR audible bell enabled for the VM) - but with no keypresses, logging shows the initial \x1b as with my local system.
  • Hitting i doesn't get mad, and does seem to trigger INSERT mode - but then it also types out another i too! And when I type a few more is they appear "normally" but upon hitting Escape, INSERT mode clears but another bunch of is appear that I didn't type (guessing as if I'd hit a number beforehand in Vim parlance).
    • This is also intermittent, sometimes things work totally normally and there's no initial i written out nor any duplication.
  • Log output upon that initial i keypress shows, after the Esc, [2;4Ri. As with my localhost, this is a repeatable cycle, and the final character is whichever key was actually pressed.

Offhand, this seems to continue to implicate something regarding the terminfo dialogue and then vim's attempted behavior afterwards. If I can't find something obvious after actually reading up on terminfo/etc, I'll punt for now - this is not worth rabbit-holing when there's an alpha to be finished.