Fix missing native modules for mac and windows build

[tegefaulkes]

Specification

Right now the mac and windows build outputs are missing the proper native modules. So far as I can tell, all of the required modules are configured to be included by pkg. However they are unavailable to be included during the build process.

This needs to be fixed before the windows and mac builds can be used.

The main problem is that the build process only has the Linux specific optional dependencies available when we are packaging. I'm still exploring the exact reason for this.

Tasks

1. Fix the build process to include the platform specific native modules in the output.
2. Test the windows build on windows to make sure it's working.
3. Test the mac build on macos to make sure it's working.

[tegefaulkes]

Right now I'm not sure why the proper platform specific optional dependencies are not being included. I'm still digging into that aspect.

[tegefaulkes]

I checked out a commit from before the flakes change and it has the same issue. So It's not a flake related issue.

[tegefaulkes]

Ok, so here is the core of the problem and what I've found out.

Some facts:

1. We have native modules packages such as @matrixai/quic and @matrixai/exec. This packages don't directly contain the native module but includes optional dependencies to sub packages that do.
2. These sub packages named like @matrixai/quic-darwin-x64 or @matrixai/quic-win-x64 depending on OS and CPU support. These have platform support define inside their package.json for the specific architecture they support.
3. When a npm package is included in the package.json as optional then it will only be installed if the platform is supported. If it was not optional then installing will fail.

So given these facts, from what I can tell. No matter what target platform we're building for. Only the optional dependencies that work on the linux platform are included. This means that for the mac and windows platform they're missing their platform specific native modules. And we only know this during runtime.

The solution is obvious, we need to make sure that the target platform native modules are available in the build environment to be included in the packaged executable. This is where I'm running into some stumbling blocks. Here is a general summary.

1. I checked and the prefetchNpmDeps does list out all of the native module packages we'd expect. Including all of the windows and macos ones we need.
2. I don't think buildNpmPackage has a way of overriding the os or cpu or forcing certain optional dependencies to be included.
3. buildNpmPackage documentaion mentions npmDeps as a way to manually specify packages but never explains it in detail so I'm unsure of how it actually works.
4. npm install and possibly npm ci has an --os and --cpu flag you can set which supposedly overrides the os and cpu when deciding what packages to include. But I don't think it's supported by our current version of npm and I haven't looked into it to test it fully yet.

Right now I think there just isn't good support for having optional dependencies that only support a specific platform. I can't find a clean work around to force include them in the build process. The only solution I can find that comes close is to include all of them and then prune the ones we don't need after installing. But that means removing the os and cpu limitation on the packages.

Right now what works is to add all of the optional native modules to the devDependencies and install with npm install --force. But this means we have to always include the --force flag.

[tegefaulkes]

So In my mind right now, the preliminary solution is to

1. Remove the os and cpu limitations to all of the native modules so then can be installed on all platforms, or at the minimum just linux.
2. Include all of the platform specific modules in the depencies.
3. Add a post-install script to prune the native modules we don't need for the target platform, default target to current platform.

[CMCDragonkai]

Yes I was aware of this in the beginning when the native packaging was changed from js-db to the new style. The new style was intended to be more optimal and the current behaviour is the correct behaviour. It reduces the bloat.

However the question is about building targetting different platforms and thus requiring the installation of additional optional dependencies.

The solution is simple. In the ci, just directly run npm install for those specific dependencies.

The problem is because while nix understands the difference between build platform and host platform and target platform, npm does not understand this. So we just need to do a little trick in the CI job to force install those dependencies at that point of building the Mac or Windows binary.

On the other hand - a long term solution might be to use the Mac and windows vms builder to build the PKG. In fact this is how the actual native objects/libraries are done, and so one can argue the same should be done in Mac and windows ci jobs.

This is further supported by the need of doing code signing which may require keeping the entire build pipeline platform specific.

[CMCDragonkai]

For now just npm install the optional package directly as an unsaved npm dependency (probably don't need saving, but if you do, you will need to do it before the npmDepsHash calculation).

[tegefaulkes]

I'm trying to go down the npm install the missing deps without saving them route but I'm running into other problems. I thought I could just install it with npm install @matrixai/quic-darwin-x64 --force. This will run but it will not install that package into the node_modules`. It seems to refuse to do this unless the package is included in the non optional dependencies.

And we can't do anything a modifies the npmDepsHash during the build since the flake sand-boxing explicitly dislikes that. I'm starting to think the only option is to include all of the native modules as normal dependencies regardless of platform and just prune the ones we don't need. The only downside with this is that it's a little more complicated and we have to download native modules for all platforms. Or we can just not prune any of it. It's only a few extra MB anyway.

This means we'll need to remove the platform and cpu restrictions for the native sub-packages.

[tegefaulkes]

So we know what the problem is in detail now, and we know how to fix it. But doing so is going to involve a general revamp of the CI build process to make use of platform specific runners to do the pkg phase of the build for each platform.

The following changes need to be made to the CI jobs

• 1. integration:builds job needs to be split up. Rather than building everything it will do 3 things,
  • a. Build the bundled code in dist and keep that as an artifact.
  • b. Build the docker image
  • c. the linux, win and mac outputs will no longer be built this way
• 2. the integration testing jobs will now handle the packaging with pkg just before the tests are run on the output. It will provide the packaged executables as artefacts
• 3. And since the win and mac outputs don't work I'll remove them from the flake.nix

[tegefaulkes]

For the final stage packaging we won't be using nix anymore. So it won't technically be a deterministic output. The main difference is the name of the final output will change.

[tegefaulkes]

I got the macos build and test working but there's an oddity. When building we have the arm64 native modules available. But this build is labelled as the x64 build. Come to think of it, do we have a x64 version mac runner we can use? If we need platform specific runner how to we make the x64 build? Should we just combine the native x64 and arm64 native modules into the one build? Something to look into.

We may have to explore setting up our own runners on the office mac and windows machines.

The windows job is still failing for now. There seems to be a problem with running the build script in windows. I need to explore the problem more.

[CMCDragonkai]

Long term I expect native integration into nix on the macos platform too with only Windows as the outlier.

[tegefaulkes]

Given we only have an arm64 runner for darwin I think we're pretty much forced to package a combined executable for darwin, we may just have to package a single darwin executable that includes the x64 and arm64 compiled native modules for darwin. This could be done in two ways.

1. Remove the cpu restriction from the darwin native builds.
2. Update the native builds to include a x64+arm64 build.

[tegefaulkes]

I've gotten both the windows and mac build to work. The mac build still needs some polish. It's only building for arm64 right now, I need to include some way to enable x64, possibly by combining the arm64 and x64 into one executable.

On reflection, It may still be possible to do cross compiling if we build and bundle with node20 and pkg with node21 targeting node20. I'll look into that after clean up.