You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
No space left on device (Test machines now need >25G disk) #4516
The machine (test-ibm-ubuntu2404-x64-1) has about a 24G disk and the biggest users of space are as follows:
15GB in /home/iojs of which nearly 8G is the iojs jenkins workspace and just over 5G is ccache
over 2G in /var/log/journal. I've pruned that with journalctl --vacuum-size=100M but my expectation is that it will probably fill up again fairly quickly
We should look at the amount of workspace space and see if we need to do anything to increase the capacity on this machine given that the node codebase is only ever increasing. This is the current size of the contents of the workspace directory for each job:
I will aim to add an additional disk of ~25GiB which can be mounted on the workspace directory. All are in the Dallas 13 region except for rhel9-2 which is in San Jose 3.
OK this wasn't as easy as I expected. Tried on test-ibm-rhel9-x64-2. Created a 25G "block device" within classic infrastructure in the same location as the host (San Jose 3) and authorized the machine to access it.
Once it was complete the device seemed to show up three times (Once with the long name in /dev/mapper and also /dev/sda and /dev/sdb.
At the time of writing I did reboot it once but after the second time (once /etc/fstab was updated) it has not yet come back. On a positive we do have three rhel9 at the moment so if this doesn't came back we still have two.
If it becomes too complicated to add an additional disk we might fall back to reprovisioning the machines, selecting the larger boot disk on the replacement server.
Having just got another machine (a new one, not any existing one in the CI) into the same broken state I think reprovisioning with a 100G disk is the most appropriate way forward at this point.
OK test-ibm-rhel9-x64-2 is back and running with its new disk.
Step 1: Add a new block device within classic infrastructure in the same location and give the machine authorization to access it
Step 2: Follow the appropriate section of https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-started&interface=ui#mountingstorage to configure the OS to talk to the device over the network
Step 3: Add it to /etc/fstsab but DON'T SKIP THE _netdev option otherwise the machine will not come back after a reboot and will need to be rescued.
I've created one new RHEL8 and one RHEL9 system with a 100GB SSD which can replace one of each of the existing ones once set up next week.
Activity
sxa commentedon Oct 13, 2026
The machine (test-ibm-ubuntu2404-x64-1) has about a 24G disk and the biggest users of space are as follows:
/var/log/journal. I've pruned that withjournalctl --vacuum-size=100Mbut my expectation is that it will probably fill up again fairly quicklyWe should look at the amount of workspace space and see if we need to do anything to increase the capacity on this machine given that the node codebase is only ever increasing. This is the current size of the contents of the workspace directory for each job:
The figures for some of these may not be representative and lower than normal if they've hit a space limit while executing.
Link to machine: https://ci.nodejs.org/computer/test-ibm-ubuntu2404-x64-1/
richardlau commentedon Oct 13, 2026
I'm of the opinion that 25G disks (the older IBM Cloud machines) are now insufficient.
[-]No space left on device[/-][+]No space left on device (Test machines now need >25G disk)[/+]sxa commentedon Oct 28, 2026
Looks like the test systems that have the 25GB disks are:
I will aim to add an additional disk of ~25GiB which can be mounted on the workspace directory. All are in the Dallas 13 region except for rhel9-2 which is in San Jose 3.
sxa commentedon Oct 29, 2026
OK this wasn't as easy as I expected. Tried on test-ibm-rhel9-x64-2. Created a 25G "block device" within classic infrastructure in the same location as the host (San Jose 3) and authorized the machine to access it.
Followed the instructions in https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-mountingRHEL&interface=ui to allow the device to be accessed on the machine (Noting that I had to replace the existing entry in
/etc/iscsi/initiatorname.iscsiwith the new one during this process)Once it was complete the device seemed to show up three times (Once with the long name in
/dev/mapperand also/dev/sdaand/dev/sdb.At the time of writing I did reboot it once but after the second time (once
/etc/fstabwas updated) it has not yet come back. On a positive we do have three rhel9 at the moment so if this doesn't came back we still have two.richardlau commentedon Oct 29, 2026
If it becomes too complicated to add an additional disk we might fall back to reprovisioning the machines, selecting the larger boot disk on the replacement server.
sxa commentedon Oct 30, 2026
Having just got another machine (a new one, not any existing one in the CI) into the same broken state I think reprovisioning with a 100G disk is the most appropriate way forward at this point.
sxa commentedon Oct 30, 2026
Noting for reference that the existing machines are configured as:
allow_outboundandallow_ssh)sxa commentedon Oct 30, 2026
OK
test-ibm-rhel9-x64-2is back and running with its new disk.Step 1: Add a new block device within classic infrastructure in the same location and give the machine authorization to access it
Step 2: Follow the appropriate section of https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-started&interface=ui#mountingstorage to configure the OS to talk to the device over the network
Step 3: Add it to
/etc/fstsabbut DON'T SKIP THE_netdevoption otherwise the machine will not come back after a reboot and will need to be rescued.I've created one new RHEL8 and one RHEL9 system with a 100GB SSD which can replace one of each of the existing ones once set up next week.
sxa commentedon Nov 2, 2026
Looking at https://ci.nodejs.org/job/node-test-commit-linux/nodes=rhel9-x64/buildTimeTrend it does look like the rhel9-2 system with the network attached disk is quite a bit slower, so the new previsioned ones with 100GB SSD look like the preferred option here as opposed to modifying the current systems.