Repository navigation
java.net.ConnectException: Connection timed out on jenkins-workspace machines #3037
Description
Currently we're seeing a spate of java.net.ConnectException: Connection timed out on the jenkins-workspace machines in the CI. At first I thought it was the new jenkins-workspace machines but taking those offline and putting the old ones back online shows those are also exhibiting the issue.
e.g. https://ci.nodejs.org/job/node-test-commit-arm-debug/4135/console
test-packetnet-ubuntu1804-x64-1
22:13:11 > git clean -fdx # timeout=10
22:13:11 Exporting environment variable NODEJS_VERSION with Node.js version '19.0.0'
22:13:11 Exporting environment variable NODEJS_MAJOR_VERSION with Node.js major version '19'
22:13:11 Exporting parameter NODEJS_VERSION with Node.js version '19.0.0'
22:13:11 Exporting parameter NODEJS_MAJOR_VERSION with Node.js major version '19'
22:15:21 FATAL: Connection timed out
22:15:21 java.net.ConnectException: Connection timed out
22:15:21 at java.base/sun.nio.ch.Net.connect0(Native Method)
22:15:21 at java.base/sun.nio.ch.Net.connect(Net.java:579)
22:15:21 at java.base/sun.nio.ch.Net.connect(Net.java:568)
https://ci.nodejs.org/job/node-test-commit-linux/47617/console
test-ibm-ubuntu1804-x64-1
22:10:54 > git clean -fdx # timeout=10
22:10:54 Exporting environment variable NODEJS_VERSION with Node.js version '19.0.0'
22:10:54 Exporting environment variable NODEJS_MAJOR_VERSION with Node.js major version '19'
22:10:54 Exporting parameter NODEJS_VERSION with Node.js version '19.0.0'
22:10:54 Exporting parameter NODEJS_MAJOR_VERSION with Node.js major version '19'
22:13:04 FATAL: Connection timed out
22:13:04 java.net.ConnectException: Connection timed out
22:13:04 at java.base/sun.nio.ch.Net.connect0(Native Method)
22:13:04 at java.base/sun.nio.ch.Net.connect(Net.java:579)
22:13:04 at java.base/sun.nio.ch.Net.connect(Net.java:568)
It's not consistent (several jobs running on the same jenkins-workspace machines pass) but often that I've not had a clean node-test-commit run in the last couple of hours.
Note that this git server being contacted prior to the time outs is github.com and not the temp binary repository (which is only used by the windows-fanned and arm-fanned jobs).
Activity
richardlau commentedon Sep 23, 2022
The expected output, without the ConnectException, is
It's unclear to me whether the exception is occurring during the git step or the subsequent fetch of the VersionSelectorScript.groovy script.
richardlau commentedon Sep 24, 2022
I put an extra print statement in the Groovy Script Matrix Executor Strategy script for https://ci.nodejs.org/job/node-test-commit-linux/ so that it starts:
(the "Fetching..." line is the addition).
https://ci.nodejs.org/job/node-test-commit-linux/47624/console failed:
so it looks like it's the part of the Groovy script that fetches
VersionSelectorScript.groovythat is timing out.richardlau commentedon Sep 25, 2022
Possibly related:
https://ci.nodejs.org/job/node-test-commit-linux/47653/nodes=ubuntu1804-64/console looks like the first
curlfrom https://raw.githubusercontent.com/ took longer than expected and the build failed at the second one. In these cases Groovy isn't involved but both are accessing https://raw.githubusercontent.com/ URLs which the Groovy scripts also are doing.richardlau commentedon Sep 25, 2022
This one as well:
https://ci.nodejs.org/job/node-test-commit-linux/nodes=ubuntu1804-64/47621/console
In this case there was another issue failing that build (with the local git checkout) but the post build step also attempts to
curla https://raw.githubusercontent.com andcurltimed out after five minutes.targos commentedon Sep 26, 2022
Maybe this should be reported to GitHub?
richardlau commentedon Sep 26, 2022
How does one do that?
targos commentedon Sep 26, 2022
I'd try with https://support.github.com/contact/bug-report
targos commentedon Sep 26, 2022
Assuming it's https://raw.githubusercontent.com/ that sometimes never responds.
richardlau commentedon Sep 26, 2022
I've opened a ticket (https://support.github.com/ticket/personal/0/1810165) but it doesn't look like it is visible to anyone else despite selecting "Node.js" as the "Account or organization" 😞
richardlau commentedon Sep 26, 2022
I have a sporadic command line recreate on one of our Digital Ocean droplets with curl -v. (Used -4 on this host as it doesn't have a working IPv6 route):
Eventually this succeeds but only after the first few attempts to 185.199.110.133 and 185.199.111.133 timed out. 185.199.109.133 succeeded (note that the third octet is different in each case).
richardlau commentedon Sep 27, 2022
It appears that the recent builds today/last night are not encountering this issue, hopefully it's resolved itself 🤞. (FTR we haven't done anything to fix this and GH haven't responded to the ticket I raised.)