Intermittent tty timeouts on AIX

[refack]

We have a different but intermittent problem on AIX:

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23617/console

08:32:35 ok 2550 pseudo-tty/test-async-wrap-getasyncid-tty
08:32:35   ---
08:32:35   duration_ms: 0.166
08:32:35   ...
08:42:35 Build timed out (after 10 minutes). Marking the build as failed.
08:42:35 gmake[1]: *** [test-ci] Terminated

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23618/console

09:35:15 ok 2550 pseudo-tty/test-async-wrap-getasyncid-tty
09:35:15   ---
09:35:15   duration_ms: 0.176
09:35:15   ...
09:45:15 Build timed out (after 10 minutes). Marking the build as failed.
09:45:15 gmake[1]: *** [test-ci] Terminated

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23622/console

11:38:22 ok 2539 message/vm_dont_display_syntax_error
11:38:22   ---
11:38:22   duration_ms: 0.110
11:38:22   ...
11:48:22 Build timed out (after 10 minutes). Marking the build as failed.
11:48:23 gmake[1]: *** [test-ci] Terminated

easy to reproduce:

> su iojs
> cd /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
> NODE_NAME=test-osuosl-aix61-ppc64_be-3 . build/jenkins/scripts/select-compiler.sh
> python tools/test.py pseudo-tty/console-dumb-tty

or

python -m trace -t tools/test.py --report pseudo-tty

to see a trace of all lines. Seems like there's an issue with sub-process spawning and ptty, since the process stalls on:

test.py(648):   if faketty:
test.py(649):     os.close(rest['stdout'])

where rest['stdout'] is fd_out created by (out_master, fd_out) = pty.openpty()

Originally posted by @refack in #1815 (comment)

[richardlau]

Possibly related to nodejs/node#9728? cc @gireeshpunathil

[mhdawson]

I think @sam-github has been looking at this one.

[sam-github]

Nope, first I've heard of it.

I seem to recall that years ago, during the first ports to AIX, that there was a problem with python involving ptys. @gireeshpunathil or @richardlau, you were around then, do you remember this?

[richardlau]

Nope, first I've heard of it.

I seem to recall that years ago, during the first ports to AIX, that there was a problem with python involving ptys. @gireeshpunathil or @richardlau, you were around then, do you remember this?

I think it was the nodejs/node#9728 I referenced earlier.

[sam-github]

@refack just to be clear, python tools/test.py --report pseudo-tty is how to repro the issue, but its not what we are actually doing in CI? I ask, because if we were doing that, a possible workaround would be to just stop using --report pseudo-tty.

[refack]

@sam-github, yes.
--report pseudo-tty are two independent args --report is just informative, pseudo-tty is a positional arg telling the harness which suite to run.

I updated the recipe to:

> su iojs
> cd /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
> NODE_NAME=test-osuosl-aix61-ppc64_be-3 . build/jenkins/scripts/select-compiler.sh
> python tools/test.py pseudo-tty/console-dumb-tty

[refack]

I think @sam-github has been looking at this one.

This might be related to the issue behind https://ci.nodejs.org/job/node-test-commit-aix/23596/ 🤷‍♂

[sam-github]

That would nice if its true. I reset the TERM on the problem host, so I'll keep an eye on this.

[richardlau]

https://ci.nodejs.org/computer/test-osuosl-aix61-ppc64_be-2/ (the problematic host) is currently offline (disconnected by @joyeecheung since this morning).

[mhdawson]

@sam-github I guess its needs another look.