[Atom-1.28.0-beta1][OSX] Mocha tests fail in headless mode, but not in headed mode

[lierdakil]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

I'm using https://github.com/BinaryMuse/atom-mocha-test-runner to run package specs, and I started noticing something weird recently.

The latest Atom beta (1.28.0-beta1) fails to run specs that require package activation on OSX.

The package is activated via await atom.packages.activatePackage(path.join(__dirname, '..')) (since atom-mocha-test-runner doesn't automatically load/activate the package in question). This works on Atom 1.27, and on Windows/Linux on 1.28.0-beta1, but for some reason not on OSX 1.28.0-beta1.

I've went out of my way to get access to an OSX machine, and apparently, specs run fine, more or less, from Atom via window:run-package-specs, but fail completely via atom-beta --test

Overall it appears as if the package fails to activate completely when specs are run with atom --test

Additional Information

Some build examples: https://travis-ci.org/atom-community/markdown-preview-plus/builds/384340951

[lierdakil]

I was able to track this down to the dependency on fsevents, which silently fails to load in Package.getIncompatibleNativeModules, and further investigation yields the following trace:

Method bytesRead called on incompatible receiver #<TTY>
TypeError: Method bytesRead called on incompatible receiver #<TTY>
    at Function.getOwnPropertyDescriptor (<anonymous>)
    at names.map (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/rpc-server.js:35:29)
    at Array.map (<anonymous>)
    at getObjectMembers (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/rpc-server.js:34:16)
    at getObjectPrototype (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/rpc-server.js:52:14)
    at valueToMeta (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/rpc-server.js:95:18)
    at EventEmitter.<anonymous> (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/rpc-server.js:387:25)
    at emitThree (events.js:136:13)
    at EventEmitter.emit (events.js:217:7)
    at WebContents.<anonymous> (/private/var/folders/jv/t7bl0bt111n7m1nct94bxdw80000gn/T/AppTranslocation/7212D351-985E-4707-BF34-DB60627600B2/d/Atom Beta.app/Contents/Resources/electron.asar/browser/api/web-contents.js:295:13)
    at emitTwo (events.js:126:13)
    at WebContents.emit (events.js:214:7)

[lierdakil]

Here's a test package: https://github.com/lierdakil/atom-mocha-osx-test

Here's a typical build for this package: https://travis-ci.org/lierdakil/atom-mocha-osx-test/builds/384376894

[lee-dohm]

Hey @jasonrudolph ... this may be related to Electron v2.0? Can you take a quick look to give a 👍/:-1: on that?

[lierdakil]

Possibly related: nodejs/node#17636

[jasonrudolph]

@lierdakil: Thanks for the detailed issue with nice, clear reproduction steps. ✨

---

@maxbrunsfeld @daviwil: This issue leads me to believe that something differs in the way modules are loaded when running in headless mode versus headed mode, but I'm not yet sure what that difference is. Can you think of anything that might be causing the different behavior noted below, or can you think of a good place for me to look to try to track down the difference?

Headless

When running the tests from the command line, the activation and deactivation tests fail and log the following output:

$ atom --test spec

Package.isCompatible() =  false
Package.getIncompatibleNativeModules() =  [ { path: '/private/tmp/atom-mocha-osx-test/node_modules/fsevents',
    name: 'fsevents',
    version: '1.2.4',
    error: 'Method bytesRead called on incompatible receiver #<TTY>\nTypeError: Method bytesRead called on incompatible receiver #<TTY>\n    at Function.getOwnPropertyDescriptor (<anonymous>)\n    at names.map (/Applications/Atom.app/Contents/Resources/electron.asar/browser/rpc-server.js:35:29) ... <snip>' } ]

1 passing (159ms)
2 failing

Headed

When running the tests within Atom, they all succeed and log the following output:

Package.isCompatible() =  true
activation.test.js? [sm]:14 Package.getIncompatibleNativeModules() =  []

[jasonrudolph]

Hey @jasonrudolph ... this may be related to Electron v2.0? Can you take a quick look to give a 👍/👎 on that?

@lee-dohm: Probably 👍.

As far as I can tell, this issue does appear to be new in Atom 1.28, which means there's a good chance it's due to the Electron 2.0 upgrade.

With Atom 1.27, if an incompatible native module is present in the package's node_modules directory, the tests fail in both headless and headed modes (whereas the headed tests pass in Atom 1.28). For example, I copied an incompatible version of keytar into the node_modules directory of the test package, and observed the results shown below.

Headless

When running the tests from the command line, the activation and deactivation tests fail and log the following output:

$ atom --test spec

Package.isCompatible() =  false
Package.getIncompatibleNativeModules() =  [ { path: '/private/tmp/atom-mocha-osx-test/node_modules/keytar',
    name: 'keytar',
    version: '4.2.1',
    error: 'The module \'/private/tmp/atom-mocha-osx-test/node_modules/keytar/build/Release/keytar.node\'\nwas compiled against a different Node.js version using\nNODE_MODULE_VERSION 57. This version of Node.js requires\nNODE_MODULE_VERSION 54. Please try re-compiling or re-installing\nthe module (for instance, using `npm rebuild` or`npm install`).' } ]
!!

  1 passing (211ms)
  2 failing

$ atom --version
Atom    : 1.27.2
Electron: 1.7.15
Chrome  : 58.0.3029.110
Node    : 7.9.0

Headed

When running the tests within Atom, they fail in the same way that they failed in headless mode:

[jasonrudolph]

The latest Atom beta (1.28.0-beta1) fails to run specs that require package activation on OSX.

@lierdakil: Based on your findings in #17409 (comment) and my follow-up findings in #17409 (comment) and #17409 (comment), I think the root problem is different than what is described in the original issue body. I've described my current understanding below. Does that description seem accurate to you?

Steps to Reproduce

1. Identify the version of Electron used by your version of Atom (e.g., Atom 1.28.0-beta2 uses Electron 2.0.1).
2. Create a package that has a native dependency that is incompatible with that version of Electron (e.g., fsevents 1.2.4 appears to be incompatible with Electron 2.0.1)
3. Add a test that activates the package and asserts that activation succeeded. [example]
4. Run apm install to install the package's dependencies.
5. Launch Atom and open a project containing the package.
6. Run the Window: Run Package Specs command.

Expected behavior: The specs should fail, because the package specifies a dependency on an incompatible native module.

Actual behavior: The specs pass.

[lierdakil]

fsevents 1.2.4 appears to be incompatible with Electron 2.0.1

This is not accurate. fsevents happily loads in headed mode when required directly:

But the same thing fails right away in headless mode:

https://travis-ci.org/lierdakil/atom-mocha-osx-test/jobs/386513049

Find the code at lierdakil/atom-mocha-osx-test#3

[lierdakil]

The more I look into it, the more it seems that the root issue is not as much upgrade to Electron 2.0, as is upgrade to Node 8.9.3. nodejs/node#17636 seems very much relevant.

What I don't understand is where the apparent difference between headed and headless comes from. Perhaps this line from Electron is just never executed in headed mode?

[jasonrudolph]

@lierdakil: Thanks for the additional info. 🙇

The more I look into it, the more it seems that the root issue is not as much upgrade to Electron 2.0, as is upgrade to Node 8.9.3. nodejs/node#17636 seems very much relevant.

That sounds right to me. I see that nodejs/node#17636 is fixed in nodejs/node#17665, and that fix is currently in Node 10.0.0 and subsequent 10.x.y releases (nodejs/node@efffcc2).

The fix has been backported to Node 8.11.2 in nodejs/node#20478.

I'll check in with the Electron folks to see if there's a way to get an Electron 2.0.x release that uses Node 8.11.2, or if there's a different approach that could resolve this issue. I'll follow up here with my findings.

[jasonrudolph]

Cross-referencing electron/electron#13061, as it appears to be related.

[jasonrudolph]

I'll check in with the Electron folks to see if there's a way to get an Electron 2.0.x release that uses Node 8.11.2, or if there's a different approach that could resolve this issue. I'll follow up here with my findings.

I chatted with @ckerr, and he opened electron/node#38 to backport the fix from nodejs/node@efffcc2 into the version of Node used in Electron 2. 😅

Once there's a build of Electron that includes that fix, I'll upgrade Atom to use that build of Electron, and verify that it resolves this issue. 🤞

[jasonrudolph]

This is fixed by #17549, and that change will be included in the Atom 1.28 release.