CI instability - Windows issues

[joaocgreis]

Jenkins has been running slowly in the last few days, frequently resulting in 504 errors.

@refack found this is related to node-test-binary-windows CPU usage (#1493).

Yesterday, all node-compile-windows jobs were failing because the job parameters were not being set as environment variables, so @refack disabled the node-test-commit-windows-fanned job. This issue of not expanding environment variables has happened before in Jenkins, but we were never able to find out why. It usually just stops happening after a while, just like it did this time.

Today I noticed the Jenkins slowness issue seems to be related to Windows 2016 test jobs being left behind: the test job looks complete, node-test-binary-windows finishes, but the tests for win2016 are left as running in the workers and blocking the queue (for about 1 hour). During this time, Jenkins appears extremely slow to me.

I re-enabled the node-test-commit-windows-fanned job, but removed Windows 2016 from node-test-binary-windows. I will keep investigating this.

@nodejs/build FYI

[joaocgreis]

Side note: I used the time the job was down to add a "Flexible publish" step, that should run even if the job is aborted or failed, to delete .git if index.lock exists. This is not extremely reliable, but might be better than nothing.

[joaocgreis]

FWIW, I changed the node-compile-windows and node-test-binary-windows jobs to stop using the Jenkins Git plugin and use git commands directly, similar to what git-nodesource-update-reference and node-test-binary-arm already have. This will hopefully reduce the size of the build files, addressing #729 for those jobs, and reducing the load on Jenkins. This should also completely solve the index.lock issue for these jobs.

[joaocgreis]

Today I've seen the jobs left behind issue with win10 (Azure) and win2008 (Rackspace). So, the issue was not only on win2016.

I believe Jenkins wasn't updated since the last security release. Do we have a log of updated plugins? @nodejs/jenkins-admins did anyone update any plugin that could be causing this?

[joaocgreis]

I stopped Jenkins for a while and removed the git build data (#729) from all the runs of a few jobs, including the ones I changed above to no longer produce it. This seems to have improved the situation considerably. Added win2016 back to the Windows matrix.

[joaocgreis]

Removing the git build data seems to have fixed Jenkins for now.