CLOUDSTACK-7143: Refactoring of the systemvm build process

[lsimons]

E-mail thread:
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E

This started out as wanting the systemvm build to take
systemvm/patches/debian/{debian,vpn} from the local machine/branch,
rather than downloading from the apache git master [1]. In working out
how on earth to get veewee to do that cleanly (hint: you can’t, hence
resorting to shar usage) I got quite frustrated with the image rebuild
times.

It so happens that veewee has a --skip-to-postinstall instruction which
is quite useful while debugging these scripts. To get that working
requires the post install steps to be retryable/convergent. Of course,
our existing scripts weren’t set up for that. So I had to add a bunch
of tests whether changes had applied already. Which implied a pretty
significant refactor.

Summarizing this kind of thing is always hard...it’s many little
things...the interesting stuff is at the end/bottom, in particular
the two main improvements

schubergphilis/cloudstack@142d087
When working on the systemvm in isolation, or using vagrant or
similar tools, it can be useful to inject a custom SSH key before
merging a management server systemvm.iso into it. This option
allows that. It should not have effect on management-server-
managed vms which always get their SSH keys injected.

schubergphilis/cloudstack@e2240ea
The current build downloads its script from master by fetching a
cloudstack tarball. Besides being an unneeded load on the apache
git server, this is a problem when working on a branch and
wanting to inject a different set of scripts. It also makes it
pretty likely that the injected copy of the script will not match
what a production release wants, so there is very little chance of
not needing to overwrite the scripts.

Ideally we would just rsync over some files. However, veewee does
not provide an option to do that. In order to keep a 'cleanly
veewee-only' build possible, and work with any recent veewee
version, in this change we restor to using shar
(http://en.wikipedia.org/wiki/Shar) to produce an archive which
can execute as a script, which we feed to veewee to execute.

In order to avoid having to re-do this cleanup twice, I also ended up
merging the systemvm and systemvm64 template definitions, factoring out
their small differences by inspecting the os architecture.

schubergphilis/cloudstack@f570b39
schubergphilis/cloudstack@50e9121

Everything else…well it pretty much falls into two categories:

• general code cleanup without functional changes
• general code defensiveness to survive various jenkins build
  scenarios

All in all it should help with ongoing maintenance, I think.

Most of these commits are now a while old but I wanted to wait with
sending this upstream until we had sufficiently tested the systemvms
built with this changed approach locally.

[rohityadavcloud]

Hi Leo, thanks for the PR.

I see you've refactored the 32 and 64 bit building scripts to one which is great!
I'll be able to test it next week and help you get it fixed if needed and merged. Meanwhile you may ask Hugo to set a test build job for testing the appliances. And some basic tests using those built systemvm templates both 32 and 64 bits.

[lsimons]

Hey Rohit, thanks for reviewing!

We actually have a new vagrant-based component test setup for the systemvm to contribute, too; I'm working on extracting that now from the schubergphilis feature/systemvm-persistent-config branch. Our internal jenkins setup is already running those tests, as well as wider-scale marvin-driven integration tests that use these new-style systemvms. That's why I think this is ready for upstream: I'm pretty confident this stuff all still works properly.....tough of course that really really needs careful verification outside of our assumptions/environment.

I'll try and work with Hugo to port those jenkins builds to buildacloud.org...hope he has time for it :)

[rohityadavcloud]

Hi, I'll try to setup a build job for this PR and see if we can merge it on master soon after testing new builds.

[rohityadavcloud]

I've added a new build job for this PR: http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/

Can you check its build commands, and/or update the README.md file on how to build appliances using build.sh script

[lsimons]

Hey Rohit, like I thought, a ruby issue.....I'm surprised RVM isn't enabled already since the build was already using it (I think...). The fix should be along the lines of http://rvm.io/integration/jenkins .

The other part of the fix is making sure that the inline shell scripts act as login scripts so that rvm is loaded, i.e. they need to start with

#!/bin/bash -l

[rohityadavcloud]

Leo, From the jenkins job log [1] I see that the newly refactored build script is trying to setup ruby [2] and failing because of this. The jenkins job [1] already has rvm setup as I simply cloned it from previous systemvm jobs. Can you look at the job and fix it? If you don't have an account on [1], can may request access to Hugo, Edison, Chip and other PMC members.

[1] http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/3/console
[2] https://github.com/schubergphilis/cloudstack/blob/feature/systemvm-refactor-for-upstream/tools/appliance/build.sh#L272

[rohityadavcloud]

Looks like the build has passed now: http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/9/

I'll test it against KVM and if it works I'll go ahead and merge it. Thanks.

[rohityadavcloud]

Hi Leo, I spent last few hours in debugging systemvms. I tested the built systemvms from http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/9/
on KVM using both master and 4.4 branch as my systemvms were unable to connect to mgmt server.

At first, I was getting KVM clock error exceptions which I solved by disabling it from agent.properties (http://cjkz.wordpress.com/2014/08/08/apache-cloudstack-kvmclock-error/).

So after spending last few hours debugging them, I see the following from VNC/KVM:

It looks like the disk is corrupt for some reason, so to test the image I used the new KVM template to deploy a VM on another CloudStack deployment which perfectly worked for me. So, I'm guessing something in the KVM iso/patching process is messing with the systemvm. Now every time CloudStack tried to do HA and re-deploys systemvms I get the above on my test environment.

Can you please try this at your end using KVM or Xen and let me know if it works for you?

[rohityadavcloud]

I manually ran fsck on failing partitions of the systemvm and rebooted, then it worked for me.

@lsimons I think the issue is probably how we're partitioning and if we're cleanly shutting down the VM before exporting it in various formats? I think once you may help fix it, it should be good to merge.

[lsimons]

thanks for testing rohit. We've never seen this (with xen).

If you look at the error,

/dev/vda5: Superblock last mount time (Fri Sep 19 14:54...)
   now = Mon Sep 15 20:34... is in the future

the fsck is unhappy with the last recorded mount being sep 19 (last friday, which is correct, i.e. it matches with when we ran the build on jenkins) because the date of the system as its booting is in the past, sep 15. So somehow the system clock of the VM is wrong...?

Either during the build process the time was moved backward some time after mounting the filesystem, or during the boot process time moved backward, or ... ?

I've checked and the date of the build slave where the image was baked is correct and managed with ntp so very likely was always correct...so I really don't see how the build could result in that timestamp existing anywhere.

Can you check the date of your KVM host? Or perhaps if you have an NTP server with a wrong date?

[lsimons]

sorry, nevermind the last comment.

From the jenkins build log:

[2014-09-19 14:57:59] INFO: stoppping all virtualbox vms for jenkins
+ bundle exec ./vbox_vm_clean.rb
VBoxManage controlvm systemvmtemplate poweroff
VBoxManage: error: Invalid machine state: PoweredOff (must be Running, Paused or Stuck)
VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports
VBoxManage: error: Context: "PowerDown(progress.asOutParam())" at line 222 of file VBoxManageControlVM.cpp
VBoxManage controlvm systemvm64template poweroff
VBoxManage: error: Invalid machine state: PoweredOff (must be Running, Paused or Stuck)
VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports
VBoxManage: error: Context: "PowerDown(progress.asOutParam())" at line 222 of file VBoxManageControlVM.cpp
kill -SIGKILL 19324
kill -SIGTERM 19324
./vbox_vm_clean.rb:49:in `kill'
./vbox_vm_clean.rb:49:in `block (2 levels) in <main>'
/usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:237:in `block in ps'
/usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:110:in `foreach'
/usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:110:in `ps'
./vbox_vm_clean.rb:31:in `block in <main>'
./vbox_vm_clean.rb:14:in `each'
./vbox_vm_clean.rb:14:in `<main>'
VBoxManage controlvm systemvm64template-systemvm-refactor-for-upstream-9 poweroff
VBoxManage: error: Invalid machine state: Aborted (must be Running, Paused or Stuck)
VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports
VBoxManage: error: Context: "PowerDown(progress.asOutParam())" at line 222 of file VBoxManageControlVM.cpp

The lines

kill -SIGKILL 19324
kill -SIGTERM 19324

mean that we forcibly shut down a virtualbox process, probably the one that was running the systemvm that was built. This means that the line further up

bundle exec veewee vbox halt systemvm64template-systemvm-refactor-for-upstream-9

did not succeed in fully shutting down the new machine, and then the next attempt to cleanly power off

VBoxManage controlvm systemvm64template-systemvm-refactor-for-upstream-9 poweroff

also failed, and so we don't umount cleanly, and so stuff is broken.

FWIW, a successful build has something more like

VBoxManage controlvm systemvmtemplate-systemvm-persistent-config-4.5.0.78 poweroff
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%```

I'll investigate what went wrong.

We should probably also just fail the build if the poweroff fails rather than attempting a 'kill'.

[rohityadavcloud]

Thanks for checking the logs, the time on KVM host was today (22 sep). When I ran fsck it found and fixed couple of corrupt inodes. I think the force shutdown is the only issue we need to fix. After manually fixing the disk volumes I was able to deploy vms. Let me know when you've fixed the vbox shutdown/export issue, I'll retest it and we should be able to merge your PR to master. Cheers.

[lsimons]

Hey Rohit, looks like that did it. Can you try the outputs from http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/10/console when it finishes archiving? Cheers!

[rohityadavcloud]

Good news! It works for me now. I've tested it against 4.4 branch with KVM basic zone + SG and it LGTM. I'll go ahead with merging it to master now. Thanks for you work Leo!

[rohityadavcloud]

I've rebased your pull request against latest master, fixed some conflicts and merged to latest master. Please check the merge and send any other changes/fixes accordingly. Thanks.

[lsimons]

Thanks for all the help Rohit! I'm sure to most people some bash script rearrangement looks boring, but nevermind that, to me it just feels good to sumbit more than just simple one-line patches to an apache project after such a long time :-)