MacOS: posix_spawnp: invalid argument (Bad file descriptor)

[psibi]

Using the process shipped with GHC 9.2.3 (and even the latest master), I have been able to reproduce a bug while running integration tests for Stack. Following points about it:

• Reproducible only on MacOS. They work fine on both Windows and Linux.
• It's trivial to reproduce it on Mac with the stack integration test suite. I tried to get a minimal example, but my attempts have been futile.

For reproducing it via Mac, use this branch:

• macos-posix-spawn-bug: https://github.com/commercialhaskell/stack/tree/macos-posix-spawn-bug
• Lots of tests are affected by it. The logs usually look like this:

/Users/ec2-user/.stack/programs/x86_64-osx/ghc-9.2.3/bin/ghc-pkg-9.2.3: startProcess: posix_spawnp: invalid argument (Bad file descriptor)
Main.hs: Exited with exit code: ExitFailure 1
CallStack (from HasCallStack):
  error, called at /Users/ec2-user/stack/test/integration/lib/StackTest.hs:63:34 in main:StackTest
  stack, called at /Users/ec2-user/stack/test/integration/tests/script-extra-dep/Main.hs:4:8 in main:Main

End of log for script-extra-dep
Failed tests:
- script-extra-dep - ExitFailure 1

A quick way to reproduce it doing something like this stack-integration-test -m 111-custom-snapshot.

Note that the above branch is sprinkled with lots of logs. It was done to help debugging. Let me know if you want me to remove it. I have also sprinkled some logs on my process fork and it indicates to me that the bug happens when creating the ghc-pkg process. Also the log indicates that I get the exit status code of 9.

I invoked dtruss using the following command:

sudo dtruss ~/.local/bin/stack-integration-test -m 111-custom-snapshot 

This is the dtruss logs for the above command: https://gist.github.com/psibi/0c3c89dd2b90012d7d9f3a64ceffb73a

[psibi]

There is some more info about the bug here: commercialhaskell/stack#5763 (comment)

[bgamari]

Thanks for opening this!

Unfortunately, the dtruss output isn't terribly helpful as it exhibits a much different failure mode from what you describe here:

...
Using runghc located at /Users/ec2-user/.stack/programs/x86_64-osx/ghc-9.2.3/bin/runghc
runghc 9.2.3
Initializing/updating the original Pantry store
Checking for project config at: /Users/ec2-user/stack/stack.yaml
Loading project config file stack.yaml
You are not the owner of '/Users/ec2-user/.stack/'. Aborting to protect file permissions.
Retry with '--allow-different-user' to disable this precaution.
stack-integration-test: Received ExitFailure 1 when running
Raw command: /Users/ec2-user/.local/bin/stack update

I suspect the sudo is at fault here. Could you try again with --allow-different-user? Also, do add -f to your dtruss invocation to ensure that forked processes are followed.

[psibi]

Could you try again with --allow-different-user?

Not sure what you mean by that since stack accepts --allow-different-user. Here I'm reproducing the issue with the executable named stack-integration-test which is different than stack and doesn't accept that flag:

ec2-user@ip-10-0-4-6 stack % ~/.local/bin/stack-integration-test --help
Initiating Stack integration test running
Stack integration tests

Usage: stack-integration-test [-s|--speed SPEED] [-m|--match STRING]

Available options:
  -h,--help                Show this help text

Dtruss logs for the following command:

sudo dtruss -f ~/.local/bin/stack-integration-test -m 111-  

Log link: https://gist.github.com/psibi/dc56d065590253f033e1de6a0a820c4d

[bgamari]

Not sure what you mean by that since stack accepts --allow-different-user

@psibi, you may need to either:

• modify stack-integration-test to pass --allow-different-user to stack
• temporarily chown root -R /Users/ec2-user/.stack/ to circumvent the check, or
• convince Darwin to allow you to execute dtruss as an unprivileged user

Dtruss logs for the following command:

It looks like this failed for yet another reason:

stack-integration-test: Executable named runghc not found on path: ["/usr/local/bin","/usr/bin","/bin","/usr/sbin","/sbin"]

Are you sure PATH is set properly?

[psibi]

@bgamari I did these things:

• Configured stack to use --allow-different-user
• temporarily changed ownership also to root.

I cannot allow Darwin to execute dtruss as an unprivileged user since it seems it's not possible to do in AWS EC2's Mac instance. This is the command I ran:

sudo dtruss -f ~/.local/bin/stack-integration-test -m 111-custom-snapshot

Log link: https://gist.github.com/psibi/8c65361bab44fda64523e20d45790dc0

[bgamari]

Unfortunately it appears that dtruss isn't capturing calls to posix_spawnp, likely because it is not a system call. I suppose I will need to try to reproduce this locally. Any hints for doing so, @psibi ?

[psibi]

@bgamari It's straightforward to reproduce it in MacOS with the stack codebase. You can follow these steps:

• Use macos-posix-spawn-bug branch of the stack git tree: https://github.com/commercialhaskell/stack/tree/macos-posix-spawn-bug
• Build the branch and create the executable stack-integration-test out of it.

Once you have the executable ready, executing something like this will reproduce the bug:

$ stack-integration-test -m 111-custom-snapshot

The above command will run the integration test of 111-custom-snapshot and that will lead straight into this bug.

[bgamari]

Build the branch and create the executable stack-integration-test out of it.

Specifically how should I go about building it such that it links against my process tree? Apologies for the silly question; I have relatively little experience with stack.

[psibi]

@bgamari Sorry, I should been more clear. These are the steps:

• Clone that repo and checkout to macos-posix-spawn-bug branch
• Download last released stack executable to your macOS: https://github.com/commercialhaskell/stack/releases/tag/v2.7.5
• You can see that I'm linking my fork of process in the stack.yaml file: https://github.com/commercialhaskell/stack/blob/c1114ee6bc77014f294ea2101ebd64874268063a/stack.yaml#L23 (You can change to your fork etc so that you can link to any arbitrary process package)
• This should install both the new stack and stack-integration-test binary into ~/.local/bin:

stack install --flag=stack:integration-tests stack

• And then you can reproduce the issue with the stack-integration-test binary:

stack-integration-test -m 111-custom-snapshot

Let me know if you are stuck or need any more help. :-) (I don't have a personal Mac and I did my entire testing using Amazon's EC2 Mac instance. I can create a new instance and help you further, if any steps are unclear)

[bgamari]

Thanks @psibi! Very helpful.

[mpickering]

@psibi I can't build with your instructions. Can you please help?

process                      > [1 of 2] Compiling Main             ( /private/var/folders/zm/mbjlbmnj1bg9gnpsf60mwq7c0000gq/T/stack-31b596dfe164f4b4/process-1.6.14.0/Setup.hs, /private/var/folders/zm/mbjlbmnj1bg9gnpsf60mwq7c0000gq/T/stack-31b596dfe164f4b4/process-1.6.14.0/.stack-work/dist/x86_64-osx/Cabal-3.6.3.0/setup/Main.o )
process                      > [2 of 2] Compiling StackSetupShim   ( /Users/matt/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /private/var/folders/zm/mbjlbmnj1bg9gnpsf60mwq7c0000gq/T/stack-31b596dfe164f4b4/process-1.6.14.0/.stack-work/dist/x86_64-osx/Cabal-3.6.3.0/setup/StackSetupShim.o )
process                      > Linking /private/var/folders/zm/mbjlbmnj1bg9gnpsf60mwq7c0000gq/T/stack-31b596dfe164f4b4/process-1.6.14.0/.stack-work/dist/x86_64-osx/Cabal-3.6.3.0/setup/setup ...
process                      > Configuring process-1.6.14.0...
process                      > Warning: The 'build-type' is 'Configure' but there is no 'configure' script.
process                      > You probably need to run 'autoreconf -i' to generate it.
process                      > setup: configure script not found.

[mpickering]

Solved by adding autoreconf to my path.

[mpickering]

Seems that the error is due to using (setStdin closed pc) in sinkProcessStdout if you change it to use setStdin nullStream pc then it works.