Torcx running out of diskspace

[moritz31]

on the flatcare edge version torcx is running out of diskspacen on the google cloud

error msg:

Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="common configuration parsed" base_dir=/var/lib/torcx/ conf_dir=/etc/torcx/ run_dir=/run/torcx/ store_paths="[/usr/share/torcx/store /usr/share/oem/torcx/store/2411.99.0 /usr/share/oem/torcx/store >
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-no path=/usr/share/torcx/profiles/docker-1.12-no.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-yes path=/usr/share/torcx/profiles/docker-1.12-yes.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=vendor path=/usr/share/torcx/profiles/vendor.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="no vendor profile selected by /etc/coreos/docker-1.12"
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="skipped missing lower profile" missing profile=oem
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=warning msg="no next profile: unable to read profile file: open /etc/torcx/next-profile: no such file or directory"
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="apply configuration parsed" lower profiles (vendor/oem)="[vendor]" upper profile (user)=
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="mounted tmpfs" target=/run/torcx/unpack
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-no path=/usr/share/torcx/profiles/docker-1.12-no.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-yes path=/usr/share/torcx/profiles/docker-1.12-yes.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=vendor path=/usr/share/torcx/profiles/vendor.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="new archive/reference added to cache" format=tgz name=docker path="/usr/share/torcx/store/docker:19.03.torcx.tgz" reference=19.03
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="new archive/reference added to cache" format=tgz name=docker path="/usr/share/torcx/store/docker:com.coreos.cl.torcx.tgz" reference=com.coreos.cl
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /usr/share/oem/torcx/store/2411.99.0: no such file or directory" path=/usr/share/oem/torcx/store/2411.99.0
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /usr/share/oem/torcx/store: no such file or directory" path=/usr/share/oem/torcx/store
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /var/lib/torcx/store/2411.99.0: no such file or directory" path=/var/lib/torcx/store/2411.99.0
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /var/lib/torcx/store: no such file or directory" path=/var/lib/torcx/store
Mar 01 21:29:08 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:08Z" level=error msg="failed to unpack: unpacking "/usr/share/torcx/store/docker:com.coreos.cl.torcx.tgz": error extracting tar: write /bin/containerd: no space left on device" image=docker reference=com.coreos.cl
Mar 01 21:29:08 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:08Z" level=error msg="apply failed: failed to install 1 images"

but on the root device is still enough space, only sum partitions seem to be limited

Filesystem Size Used Avail Use% Mounted on
devtmpfs 264M 0 264M 0% /dev
tmpfs 289M 0 289M 0% /dev/shm
tmpfs 289M 276K 289M 1% /run
tmpfs 289M 0 289M 0% /sys/fs/cgroup
/dev/sda9 27G 1.7G 24G 7% /
/dev/mapper/usr 985M 949M 0 100% /usr
none 289M 289M 0 100% /run/torcx/unpack
tmpfs 289M 0 289M 0% /media
tmpfs 289M 4.0K 289M 1% /tmp
/dev/sda6 108M 76M 24M 77% /usr/share/oem
/dev/sda1 127M 61M 66M 49% /boot
tmpfs 58M 0 58M 0% /run/user/0

[dongsupark]

Thanks for the bug report.
Interesting. It would be great to know how to reproduce it.

[moritz31]

So i just uploaded the image with the docker container to google cloud and started an f1.micro instance with 30gb of disk space. Just to make sure it is not an diskspace issue i tried it even with a 100gb instance but didn't make an difference

[anakaiti]

I'm having the same issue on AWS nano instances on version 2605.5.0. It runs normally on larger instances.

[alkar]

I'm having the same issue on AWS nano instances on version 2605.5.0. It runs normally on larger instances.

We see the same issue in both AWS (t3a.nano) and GCP (f1-micro) running 2605.5.0
Neither reboot nor a re-deploy fixes this.

[george-angel]

The problem is default tmpfs size is 50% of your physical RAM:

The size parameter also accepts a suffix %
to limit this tmpfs instance to that percentage of your physical RAM:
the default, when neither size nor nr_blocks is specified, is size=50%

ref: https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

And thats what torcx is using, default: https://github.com/flatcar-linux/torcx/blob/flatcar-master/internal/torcx/perform.go#L287

2512.3.0 - had unpacked size of ~200M - so it would run on any machine with 400M+ physical RAM.
2605.5.0 - has unpacked size of ~290M - and so now it will only run on machines with 580M+ RAM.

Perhaps flatcar/scripts#16 is relevant here to attribute the size increase, this part I'm not sure about.

[pothos]

Thanks for the detailed report. Yes, it looks like we need to specify the size of the tmpfs directly and not rely on the percentage.

[pothos]

I created a PR here flatcar/torcx#4

[pothos]

This will be addressed in the next Stable bugfix release (and any other releases).