Convenient sudo() wrapper

[bitprophet]

tl;dr Fabric 1's sudo() except implemented on top of generic, user-exposed features (#289, #293) and available at Invoke-on-up, since it's not actually specific to remote execution.

Given those user-exposed features, there's no technical necessity to implement it in core, because it should just turn into something like this:

@task
def identity(c):
    assert c.run("whoami").stdout.strip() == "myuser"
    sudo_prompt = "Sudo password plz:"
    with c.prefix("sudo -S -p '{0}' ".format(sudo_prompt):
        result = c.run("whoami", responses={sudo_prompt: "mysudopassword\n"})
        assert result.stdout.strip() == "root"

But clearly, even if you remove the asserts, that's annoying boilerplate we can erase by implementing a commonly used pattern as a convenience method.

---

MUCH LATER EDIT: There is a wrinkle here - in Fabric 1, this functionality includes "prompt the user ourselves when the sudo-prompt response value isn't pre-set, then cache the result from then on". That prompt+cache happens "inline" within the sudo() call. Brainstorm:

• The most-obvious answer is "you should store it in your config up front so no prompting is necessary".
• But storing that data on-disk is insecure; a runtime prompt is better.
  • A runtime "pulled from a secure store over the network, flash key, etc" is best. No prompt, no on-disk.
• But there's nothing saying we can't explicitly prompt up-front in sudo.
• Prompting "lazily" / as late as possible, is "nice" insofar as it means users with passwordless sudo never have to deal with a prompt at all - no inner prompt seen, no outer prompt displayed.
• In most cases, that is still just "nice" and not required - users who know up front that they will need the password, can simply enter it up front.
  • But the Fabric-specific use case of remote sudo is more complex - users with large fleets may not know whether the prompt will ever appear.
  • Counterpoint: users with large fleets will usually want "raise/store an exception and keep trucking" instead...
• Users may also have heterogenous sudo passwords across their fleets, so a single up-front prompt is no longer enough - it'd need to be one prompt per host/group/whatever.
  • Counterpoint, if you have that setup, you could be using Fabric 2's per-host/group configs
  • But then you're back to storing the data on-disk
  • So you really do need one prompt per remote password, no matter when it pops up.
  • If you know ahead of time where the passwords differ, we could then say "ok, you just need to prompt for each group" (perhaps with a config setting like e.g. {'hostname': {'prompt_for_password': True}})
  • If you don't, that is when runtime inline prompting becomes truly "necessary". Whether that's enough of a use case to be worth implementing this ourselves, hard to say.
  • NOTE that all of these concerns pop up with Fabric connection/login passwords too - with the exception that the point of prompting is at connection time and not anywhere in Runner.
• One way to enable "late but not inline" prompting is to have the prompt machinery raise an exception when it encounters a prompt key whose value is None, then sudo() can treat this as "prompt, fill config, try that same command again"
  • This isn't exactly the same as Fabric 1's behavior, because it involves running the command literally twice in a row.
  • But it feels cleaner because it doesn't require the truly-inline prompting.
  • It also arguably lets us push the entirety of "dealing with missing password caches" on the user, which is better for 'training' them to do the right thing, but worse from a usability standpoint.
• If we did go with fully inline prompting again, at least we can make it more generic - again, anything whose value is None could trigger the pause-and-prompt, not just sudo specific passwords.
  • Though there's the added wrinkle of "when to use getpass.getpass vs (raw_)input"...
• Yet another reason not to do inline prompting is that it adds implementation complexity - e.g. in Fabric 1 we need the prompt machinery to talk to the stdin handler so the stdin handler pauses itself while prompting is happening (otherwise getpass or raw_input never see the stdin, as it's being intercepted).

[bitprophet]

In the interests of getting the fab 2 alpha out the door, I think my initial target for this feature, re: sudo password prompting, will be "it requires the autoresponse password value to be non-None". Users configuring it - as many do (despite it being technically insecure) in Fabric 1 via ~/.fabricrc + sudo_password = xxx - never have to see a prompt; or it will prompt up front, prior to actual command execution, if the value has been left unset.

We can revisit "prompt lazily & rerun", "prompt inline during autoresponding", and "options to control whether prompting is done at all" later.

[bitprophet]

Another minor fun wrinkle that just occurred to me: Fabric 2 rebinds Invoke's run as local...what should it do with Invoke's sudo? Grump.

Upon reflection, while it's technically possible to offer a local sudo, I just don't see it being that commonly used (the number of legitimate local privilege-escalation needs is vanishingly small compared to that of remote ones), and it's also not cross-platform like run is either. So that's a poor tradeoff (losing API space in the client lib of Fabric 2 for little gain in Invoke itself).

Should I somehow be wrong about this we can certainly change things later.

So for now, this is just for "make the Fabric 2 implementation possible" (which should be almost entirely #293 now) and probably closed soon.

[bitprophet]

Occurred to me recently: the old "run a remote-oriented script locally instead" use case for Fabric (fabric/fabric#98) is one major reason to make sudo implemented in Invoke proper. (Also had a user asking about it on IRC recently too...)

The problem of how to expose the API in both libs remains open, none of the solutions I can think of are awesome unfortunately:

• Don't rebind, just retarget: Context.sudo executes local sudo, Remote.sudo executes remote sudo, and there is no Remote.local_sudo or similar.
  • "Solves" the API collision problem, and leans on the part where I sincerely doubt a task targeting remote servers would need to do anything superuser-ly locally. And is of no consequence in the fab-98 situation - sudo is sudo is sudo.
  • But is inconsistent with run-vs-local, and perhaps unnecessarily annoys anyone who does inexplicably need to sudo on both ends.
• Just rebind it, you doofus: Context.sudo becomes Remote.local_sudo.
  • Inelegant, sure, but does what it says on the tin, and because it's barely going to be used by anybody, who really cares? It's like 3 lines in the API doc.
• Somehow turn "sudo-ing" into a kwarg to run/local, e.g. Context.run('xxx', sudo=True).
  • Not amazing as it pushes the sudo implementation (including all the phase-2 prompt related logic noted in ticket desc) inside run in some fashion, which at the very least means users tweaking the sudo implementation is now intertwined with any manipulation of run
  • OTOH, this stuff can all still happen at the Context level (i.e. would not actually impact Runner) so it could maybe be as simple as `return (self._sudo if sudo else self._run)(command, **kwargs).
  • But still feels like a lot of mental overhead for implementers and users just to avoid some renaming...
  • Also doesn't play nice with Update "use_sudo to be "via" or similar fabric/fabric#25, aka "how to select internal sudo for things like put/get and friends", since it removes sudo as a distinct top level execution target.

Given that brain dump I think "oh fuck it who cares if the names aren't 100% beautiful and if we don't expect sudo to be used much" is the obvious choice here, and will probably pull the stuff I've been hacking on Fabric 2 back into Invoke as Context.sudo.

What I probably will do is not mention sudo in the tutorial, since that was what prompted all this in the first place - it feels dumb to use up valuable new-user mental time for something we do not expect them to really use. It should still be listed in the Fabric 2 tutorial though.

[bitprophet]

Moved this back to Invoke and continued actually implementing it. Mostly there, sticking-up bits that need addressing now instead of later:

• Thought I'd be clever by selecting an obviously-not-a-real-sudo-prompt string as the default forced password prompt, so that debugging would be more obvious. Unfortunately, if not paired with a Fabric 1 style "implicitly strip out that prompt from the mirrored+captured stderr stream", this looks pretty ugly.
  • Even if it was a regular-looking prompt, I assume it could still be pretty confusing to some users, especially if execution somehow got lagged or hung while the prompt was the last thing on the screen. Which is another argument for implementing "strip out things being responded to automatically".
  • But a counterargument is that magic is bad, and also that this would require more poking around inside run just to support sudo (though - if done it probably should be done for all auto-responses anyways).
  • For now I'll just revert to a "regular-looking" prompt, maybe with some text in it still making it clear it's part of the auto-response system.
• Integration testing this feels hard to do cleanly; even if we mock out getpass, there's the glaring issue of requiring all developers running integration tests, mutate their local system config so the tests can pass w/o human interaction.
  • Not only is that ugly and fragile, it's flat-out insecure as either this test-only user's creds need persisting in the public repo, or the user must store them in e.g. ~/.invoke.yml, which is still putting a sudo-capable user's creds on disk.
  • I think the best middle ground is to make a selected-by-Travis-only integration test/suite, which of course can then rely on something always being present, and is ephemeral / not anybody's real system.

[bitprophet]

Made #365 for the hide-sudo-prompts-maybe angle.

[bitprophet]

And as noted in #365 too...without that in place, we get a silly thing where subsequent stdout starts on same line as the sudo prompt (which is stderr) as the autoresponded newline (if one's even present; it's not automatically added at this point) is not echoed to the local terminal.

I bet that'll come up in the alpha, but I'll wait until it does, for now. Not critical.

[bitprophet]

Reverted the sudo prompt to something more realistic looking (as implied above).

Integration test, grump, I forgot we'd switched to Travis' sudo-less (faster) architecture. Would be nice not to have to switch back, especially since they will presumably remove sudo-capable test runners entirely sometime. (OTOH, we hard require it for Fabric 1's suite so we can fix some frustrating and random-ass system level bugs, so same boat there if still applicable.)

Double checked their docs; maybe they will keep it around or we can work around it some other way, going by https://docs.travis-ci.com/user/ci-environment#Virtualization-environments and https://docs.travis-ci.com/user/trusty-ci-environment the 'Trusty' env is still being poked at / lives side by side with both legacy and containerized setups.

May well still be the case that tests are slower on Trusty than containerized, but if that's the price we have to pay, so be it?

For now, I'm going to punt on that, may make it another spinoff ticket.

[bitprophet]

Ran into another snag: I'd totally forgotten how Fabric 1 not only autoresponds with sudo password, but also detects the "Sorry, try again" output from sudo and uses that to determine whether it succeeded / whether to re-prompt / etc.

Without that, any incorrect password results in a mildly silly/annoying back & worth waiting for sudo to hit its max number of retries (typically 3) before things exit & fail.

In a generic, Invoke/Fab 2 world, that requires #289's functionality to get beefed up sooner instead of later, so it can do more than simple always-respond-to-x-with-y. Will spin off another ticket for that soon after I have a brief think.