'Inappropriate ioctl for device' when using run() in a headless process

[txomon]

Hello, I have no idea what happened in the last release, but invoke 0.14.0 is breaking a setup for me when I run invoke remove_reference from a git commit-msg hook. I don't know if this gives any clue.

I will post back whenever I come up with a way to properly reproduce.

@task                                                                                                                                                                                                                
def remove_reference(ctx):                                                                                                                                                                                                    
    """Remove references"""                                                                                                                                                                
    ctx.run('rm -rf docs/reference/')                                                                                                                                                                                

  File "/home/javier/projects/test/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/program.py", line 274, in run
    self.execute()
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/program.py", line 389, in execute
    executor.execute(*self.tasks)
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/executor.py", line 113, in execute
    result = call.task(*args, **call.kwargs)
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/tasks.py", line 111, in __call__
    result = self.body(*args, **kwargs)
  File "/home/javier/projects/ridee/jinn-api/tasks.py", line 16, in remove_reference
    ctx.run('rm -rf docs/reference/')
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/context.py", line 60, in run
    return runner_class(context=self).run(command, **kwargs)
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/runners.py", line 256, in run
    return self._run_body(command, **kwargs)
  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/runners.py", line 358, in _run_body
    raise ThreadException(thread_exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x7ff17286cc88>>}

Traceback (most recent call last):

  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/util.py", line 125, in run
    super(ExceptionHandlingThread, self).run()

  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)

  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/runners.py", line 621, in handle_stdin
    data = self.read_our_stdin(input_)

  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/runners.py", line 583, in read_our_stdin
    bytes_ = input_.read(bytes_to_read(input_))

  File "/home/javier/projects/test/lib/python3.6/site-packages/invoke/platform.py", line 162, in bytes_to_read
    return struct.unpack('h', fcntl.ioctl(input_, termios.FIONREAD, "  "))[0]

OSError: [Errno 25] Inappropriate ioctl for device

[txomon]

Ok, the minimal setup for this not to work, tested with python3.6.

In a folder for this test:

virtualenv3 ve
source ve/bin/activate
pip install invoke==0.14.0
mkdir test; cd test
git init
echo '#!/bin/bash
invoke git_pre_commit_hook
exit $?' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo 'from invoke import task

@task
def git_pre_commit_hook(ctx):
    """Hook to be run by git, not called directly"""
    ctx.run("echo asdfaasdf asdfasdf")' > tasks.py
git add tasks.py
git commit -am 'ASDF'

However, the same setup with invoke 0.13.0 works

virtualenv3 ve
source ve/bin/activate
pip install invoke==0.13.0
mkdir test; cd test
git init
echo '#!/bin/bash
invoke git_pre_commit_hook
exit $?' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo 'from invoke import task

@task
def git_pre_commit_hook(ctx):
    """Hook to be run by git, not called directly"""
    ctx.run("echo asdfaasdf asdfasdf")' > tasks.py
git add tasks.py
git commit -am 'ASDF'

[bitprophet]

Almost definitely covered under this FAQ: http://www.pyinvoke.org/faq.html#i-m-getting-ioerror-inappropriate-ioctl-for-device-when-i-run-commands

Given you started having the issue in 0.14 IIRC that means your issue is on stdin specifically?

Do comment again if that doesn't get you pointed in the right direction!

[txomon]

Did you check the code? Because the snippet I posted is the only thing you need to reproduce. Also it happens with 0.14 but not 0.13

…

On Fri, Feb 3, 2017, 17:23 Jeff Forcier ***@***.***> wrote: Almost definitely covered under this FAQ: http://www.pyinvoke.org/faq.html#i-m-getting-ioerror-inappropriate-ioctl-for-device-when-i-run-commands Do comment again if that doesn't get you pointed in the right direction! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#425 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN7mnL-zCVowiKTuzBpggTbHmn86Gzzks5rY2KegaJpZM4L1Arc> .

[paiv]

I can confirm @txomon sample has described issue running on macOS.

I myself faced this on AWS CodeBuild just after upgrading to 0.14. Not sure that Git hook is involved on CodeBuild, but it might be.

[bitprophet]

The core problem here is still going to be related to stdin not being a "real enough" pipe, but if it's coming up this often we should try to figure out if there's a graceful method of handling that situation. Invoke definitely gets run headless a lot. Reopening.

[tuukkamustonen]

I seeing this on 0.16.3 (after upgrade from 0.11.1), with something like:

@task
def foo(ctx):
    ctx.run('git describe')

In general, it works, even when I run it in background:

inv foo &

However, running in background only works when I run it in interactive login shell. When I run it through Jenkins, it fails to similar stacktrace as in original report.

My jenkins slave is a docker container, but slaves connect to them over SSH as usual. No idea if docker here makes any difference.

It used to work in 0.11.1.

[bitprophet]

@tuukkamustonen Are you able to get extra info about the value of sys.stdin in that situation? (I don't have a handy Jenkins lying around right now, though I do actually have plans to do some CI work in the near-ish future...)

As in my last comment, the core issue should be that it's evading our attempt to detect whether sys.stdin is attached to a TTY, meaning it's somehow returning True when asked if it has/passes isatty or has a fileno that os.isatty says is a TTY.

I'd assume in a truly headless situation that sys.stdin would be None or at least that it would have isatty() returning False, or that it would lack a fileno, all of which ought to result in us going "ok, is not real, will not try to manipulate".

[tuukkamustonen]

I investigated it a bit more. Unfortunately, my initial pointers were a bit off. Sorry for that.

Running task in background actually works just ok, both in login shell and in Jenkins. The problem comes when you try to run in the background in a (bash) sub shell:

(inv foo)& wait

Again, it works in login shell, but fails in Jenkins. Also, running in sub shell works, but running in sub shell combined with running in the background, doesn't.

(Why would I do something like this? Yeah I know it's a bit hacky, but I've been using that for convenience - I run some (deployment) tasks in parallel with something like:

{
    (inv deploy_component1)&
    (inv deploy_component2)&
    wait;
}

This allows not having to have separate Jenkins job for each component... I'm just lazy.)

I'm not sure what debug information to provide about sys.stdin, but in Jenkins repr is:

<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>

For sys.stdin.isatty(), sys.stdout.isatty(), sys.stderr.isatty(), all return False in Jenkins (and True in login shell).

I tried running the commands in parallel using GNU parallel like:

parallel ::: "inv deploy_component1" "inv deploy_component2"

That works also with latest invoke, no issues. I assume GNU parallel works in quite a bit different ways...

---

This may or may not be related to this ticket, please advice if I should open a separate ticket about this.

Consider a task of:

@task
def foo():
    import time
    print("sleeping...")
    time.sleep(2)
    print("running command...")
    run('whoami')
    print("sleeping again...")
    time.sleep(2)
    print("DONE")

When running this in background:

inv foo &

With 0.11.1:

12:35 $ inv foo &
[1] 12364
12:35 $ sleeping...
running command...
whoami
musttu
sleeping again...
DONE

[1]+  Done                    inv foo
12:35 $

With 0.12.0+:

12:36 $ inv foo &
[1] 12515
12:36 $ sleeping...
running command...
whoami

[1]+  Stopped                 inv foo
12:36 $ # task gets halted, like it's waiting for stdout or something
12:38 $ fg  # let's resume it
inv foo
musttu
sleeping again...
DONE
12:38 $

So, in other words, 0.12.0+ halts in the background does not proceed until you bring it to foreground. In 0.11.1, it completes without delay in the background.