Repository navigation
node-test-commit is broken #3959
Description
It looks like all jobs are failing is less than a second, starting with https://ci.nodejs.org/job/node-test-commit/75761/
00:00:00.379 Honoring refspec on initial clone
00:00:00.423 Cloning repository git@github.com:nodejs/node.git
00:00:00.423 > git init /home/iojs/build/workspace/node-test-commit # timeout=10
00:00:00.427 ERROR: Error cloning remote repo 'origin'
00:00:00.427 hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit
Activity
targos commentedon Nov 14, 2024
/cc @nodejs/build-infra
I'm in a meeting the whole day and cannot investigate this.
mhdawson commentedon Nov 14, 2024
@ryanaslett is this something you could take a look at?
ryanaslett commentedon Nov 14, 2024
I did some preliminary investigation into this and it appears as though all three of our jenkins workspace machines can no longer connect to github.
All the clone urls are git@github.com, and result in
So either the url's changed in the build and they used to be https urls, or something changed on the workspace machines to remove the private keys.
mhdawson commentedon Nov 14, 2024
Earlier issue where we had trouble with connectivity from jenkins to GitHub, but this looks different- #3254
mhdawson commentedon Nov 14, 2024
Being looking at the keys/setup with Ryan to try to understand what might have changed, have followed some threads but no answer yet.
mhdawson commentedon Nov 14, 2024
Found this issue which looks like it might be the same issue after looking at the error more closely and some googling - #3614
Looks like packages were updated yesterday (by looking at /var/log/dpkg.log) on
test-ibm-ubuntu2204-x64-3were some of the failures were being seen.Possible that it ran/failed so fast that all jobs were running failing there.
Have restarted the jenkins agent and job seems to be running now. Looks ike it was just that that
gitjust failing to run from java running the agent after the system updates like was the case reported earlier in #3614.Restarted the agent on the 3 workspace machines that I see being associated with the node-test-commit job.
mhdawson commentedon Nov 14, 2024
This job seems to be running ok - https://ci.nodejs.org/job/node-test-commit/75781/
Will leave this open until we confirm that we see jobs running ok across the different workspace machines.
ljharb commentedon Nov 15, 2024
still seeing the same issue on https://ci.nodejs.org/job/node-test-pull-request/63546/
aduh95 commentedon Nov 15, 2024
The issue for
node-test-commitis resolved, but there are still other jobs impacted (node-test-commit-linux and node-test-commit-custom-suites-freestyle (test-worker))targos commentedon Nov 15, 2024
I updated and rebooted all 4 ubuntu2204 hosts
aduh95 commentedon Nov 15, 2024
That seems to have fixed it, https://ci.nodejs.org/job/node-test-pull-request/63552/ doesn’t have the issue. Closing now, thanks everyone.
ljharb commentedon Nov 15, 2024
fwiw, https://ci.nodejs.org/job/node-test-pull-request/63557/ still has the same issue.
aduh95 commentedon Nov 15, 2024
No it does not
ljharb commentedon Nov 15, 2024
hmm - maybe i linked to the wrong thing. https://ci.nodejs.org/job/node-test-commit/75799/console ?
aduh95 commentedon Nov 15, 2024
I don't see any anywhere in those logs anything related to a connection problem with github.com, AFAICT it's working fine
ljharb commentedon Nov 15, 2024
alrighty, i'll dig into it, maybe it's an actual failure. sorry for the noise :-)
richardlau commentedon Nov 18, 2024
The two Hetzner hosted machines also needed to have their agents restarted. I've just done so. (node-test-commit-v8-linux and node-daily-coverity were still broken this morning prior to restarting the agent.)
FWIW AFAICT there was no connectivity issue with GitHub and this was another case of automatic updates on Ubuntu updating the Java Runtime Environment which breaks being able to run
git/bash(others?) until the Jenkins agent is restarted. The last time this happened it was suggested disabling automatic updates -- I've pulled that discussion into its own issue: #3962@ryanaslett @mhdawson FYI we don't store the keys for GitHub access on the machines -- instead we have Jenkins configured to inject a secret into the builds to allow fetches.