Report rollbacks as Omaha error codes

[pothos]

When the system failed to boot into an update, a rollback to the
previous partition took place. However, an update success was reported
via Omaha.
Add a new error code that is reported for rollbacks. The detection is
done by comparing the currently running version number with the
previous version number that is already stored for reportting also
serves as a way to detect if a reboot is needed.

How to use

Build an image with this patch. Then run the following after booting it as VM:

sudo -s
echo "GROUP=stable" > /etc/flatcar/update.conf
# optional: use your own Nebraska instance:
# echo "SERVER=http://192.168.1.5:8000/v1/update" >> /etc/flatcar/update.conf
curl -L -o /tmp/key https://raw.githubusercontent.com/flatcar-linux/coreos-overlay/flatcar-master/coreos-base/coreos-au-key/files/official-v2.pub.pem
sudo mount --bind /tmp/key /usr/share/update_engine/update-payload-key.pub.pem
cp /usr/share/flatcar/release /tmp/release
sed -i "/FLATCAR_RELEASE_VERSION=.*/d" /tmp/release
echo FLATCAR_RELEASE_VERSION=0.0.1 >> /tmp/release
mount --bind /tmp/release /usr/share/flatcar/release
systemctl restart update-engine

Now force an update:

update_engine_client -update

When done, run poweroff and start the VM, but be quick to select the A partition in GRUB and not the default.
Now the rollback should be detected and reported instead of an update success (wait 45 seconds).
Review the output of journalctl --all -u update-engine -e.

Testing done

The above and some variations like restarting the service during an update download or after the update is prepared to make sure that no false alarm is triggered.

[t-lo]

Would it make sense to add a test for this to https://github.com/flatcar-linux/mantle/blob/flatcar-master/kola/tests/update/update.go ?

[pothos]

Yes, the update tests should be expanded, too.

[krnowak]

Some questions about the code. I also checked if nebraska will actually register this error, because it only accepts certain pairs of error type and error result (see nebraska db entries and event registration query). Looks like it will, we send type 3, result 0, which is inserted into the database.

[krnowak]

Is prev_version ever going to be empty? When we fail to get it from preferences, we set it to "initial".

[krnowak]

Also, will this branch be executed every time I restart the machine (for whatever reason, not only because of the update) and boot the partition with the previous version? Should we make a check that the update is in progress?

[pothos]

"initial" should be the value when no file exists while "" should be the value for normal operation.
Maybe I should introduce another boolean to take care of this? Now I relied on the assumption that "initial" won't be the current version.

[pothos]

The previous version file is only created when an update was applied. This should not alert a rollback in the case of normal reboots with selecting a different partition manually. (And even if it would fire I think it would be up for discussion if we want to distinguish between automatic and manual rollbacks. Currently I wanted to focus on the first reboot after an update.)

[krnowak]

"initial" should be the value when no file exists while "" should be the value for normal operation.
Maybe I should introduce another boolean to take care of this? Now I relied on the assumption that "initial" won't be the current version.

Nah, I'm just not familiar with the code. I was just assuming that this code would also run every time you reboot. And apparently that was a wrong assumption, so let's leave it as is.

The previous version file is only created when an update was applied. This should not alert a rollback in the case of normal reboots with selecting a different partition manually.

Thanks, I did not know it.

(And even if it would fire I think it would be up for discussion if we want to distinguish between automatic and manual rollbacks. Currently I wanted to focus on the first reboot after an update.)

Let's have it some other time.