Shell Scripting | Open Source Hacker

This blog post is about how to duplicate your Linux services and configuration from one server to another. We use simple and hackable SSH, rsync and shell scripting to copy the necessary files to make a cold spare from your existing server installation.

Table Of Content

1. Preface

2. More powerful duplication tools

3. Building and running the duplication script

4. Testing the spare server

5. mirror-ubuntu.bash

1. Preface

The method describes here is quite crude and most suitable for making a spare installation of your existing server. In the case you lose your production server, you can boot your cold spare, point your (fail-over) IP address to the spare server and keep business going – something you might want to do if you run mom’n’pop web hosting business. Because of the simplicity of the method it works on default Linux installations, bare metal servers and old-fashioned file systems like Ext4.

The instructions here have been tested with Ubuntu Linux 12.04, but should work with other versions with minor modifications. I have used this method successfully with Heztner hosting (highly recommended for all the cheapskates out there) by having one production machine and one spare machine. The spare is mirrored weekly. Daily Duplicity backups can be restored on the spare if week is too long data loss period. Though in my case the servers are bare metal, the method works for VPSes as well.

The duplication script might be also useful for setting up a staging server from your production server for software developer and testing.

2. More powerful duplication tools

More fail safe, more engineer-oriented duplication approaches exist, but usually require additional preparation and tuning on the top of the default Linux installation. Applying these to existing running Linux installation might be tricky.

Use virtual machines like KVM and snapshot them, then migrate the snapshotted copy on another server.
Use a file system supporting snapshots like Brfs or ZFS.
Replicate your databases using native replication (MySQL, PostgreSQL) – if you cannot afford losing few database rows (hot spare).
Replicate your file system (hot spare)
(If you have any more ideas how to make hot or cold spares please post in the blog comments)

3. Building and running the duplication script

This script is run on the target server (cold spare) and it will clone the content of the source server (actual production machine) on itself. It uses SSH keys and SSH agent to create the initial connection, so make sure you are familiar with them.

Assumptions

The target server must be clean Linux installation, the same exact version as on your source server.
Your server has standard /etc/passwd user account management. This is copied first so that we correctly preserve file ownership (related ServerFault discussion).
Services you run (PHP, Django, Plone, Node.js, you-name-it) are under /srv as recommended by Linux filesystem hierarchy standard
Source and target MySQL servers must have initialy same password for the root. Set this when the script runs apt-get install mysql-server for the first time.

Limitations

The first run is interactive, as apt-get install asks bunch of stuff for packages like MySQL and Postfix.
MySQL dumping and loading the dump does not guarantee all of your data survives intact, especially when skip-lock option is used for the performance reason. For ordinary CMS operations this isn’t a big issue.
If you other services beside MySQL needing special flush or lock handling follow the related instructions.

For MySQL duplication make sure you have the following /root/.my.cnf file on the source server, as it allows you to interact with MySQL:

[mysqldump]
user=root
password=YOUR PASSWORD HERE

[client]
user=root
password=YOUR PASSWORD HERE

Run the script inside a screen, because the duplication process may take a while and you do not want to risk losing the process because your local internet connectivity issue.

scp mirror-ubuntu.bash targetserver:/tmp
ssh targetserver
screen 
cd /tmp
bash mirror-ubuntu.bash

4. Testing the spare server

After the duplication script has successfully finished mirroring the server you want to check if the services can be successful started and interacted on the cold spare.

Change internet-facing IP addresses in the related services to be the public IP address of the spare server. E.g. the following files might need changes:

/etc/default/varnish
/etc/apache2/ports.conf
/etc/apache2/sites-available/*

Spoof your local DNS to point the spare server on the tested sites. Edit your local /etc/hosts and add spoofed IP addresses like:

1.2.3.4 www.service1.example.com www.service2.example.com opensourcehacker.com

Access the sites from your web browser, see that database interaction works (login) and file interaction works (upload and download files).

5. mirror-ubuntu.bash

#!/bin/bash
#
# Linux server ghetto duplication script
# Copyright 2014 Mikko Ohtamaa http://opensourcehacker.com
# Licensed under MIT
#

# Everything is copied from this server
SOURCE=root@myserv.example.com

# Our network-traffic and speed optimized rsync command
RSYNC="rsync -a --inplace --delete --compress-level=9"

# Which marker string we use to detect custom init.d scripts
INIT_SCRIPT_MARKER="### BEGIN INIT INFO"

# As we will run in screen we need to detach
# from the forwarded SSH agent session and we use a local
# SSH key to perform the operations.
# Also overriding /root destroys our key.
# Create a key we use for the remaining operations.
TEMP_SSH_KEY=/tmp/mirror_rsa

# The software stack might have touched the following places.
# This list is compliled through trial-and-error,
# sweat and cursing.
# We cannot take /etc as a whole, because it contains
# some computer specific stuff (hard disk ids, etc.)
# and copying it as a whole leads to unbootable system.
CHERRYPICKED_ETC_TARGETS=(\
    "/etc/default" \
    "/etc/varnish" \
    "/etc/apache2" \
    "/etc/ssl" \
    "/etc/nginx" \
    "/etc/postfix" \
    "/etc/php5" \
    "/etc/cron.d" \
    "/etc/cron.daily" \
    "/etc/cron.monthly" \
    "/etc/cron.weekly" \
    "/etc/init.d")

# Create a key without a passphrase
# and put the public key on the source server
rm $TEMP_SSH_KEY 2>&1 > /dev/null
ssh-keygen -N '' -f $TEMP_SSH_KEY
ssh-copy-id -i $TEMP_SSH_KEY $SOURCE
# Detach from the currently used SSH agent
# by starting a session specific to this shell
eval `ssh-agent`
ssh-add $TEMP_SSH_KEY

# Assume the system have same Ubuntu base installation and no extra repositories configured.
# Bring target system up to date.
apt-get update -y
apt-get upgrade -y

# TODO: check that the kernel uname is same
# on the source and the target systems

# This is somewhat crude method to try to install all the packages on the source server.
# If the package is missing or replaced this command will happily churn over it
# (apt-get may fail). This MAY cause user interaction with packages
# like Postfix and MySQL which prompt for initial password. Not sure
# what would be the best way to handle this?
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1|while read pkg
do
    apt-get install -y $pkg
done

# As some packages might have changed due to version upgrades and
# deb renames the following step needs interaction
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1

# Copy user credentials first to make sure we
# get the user permissions and ownership correctly.
# http://serverfault.com/a/583336/74975
echo "Copying users"
$RSYNC $SOURCE:/etc/passwd /etc
$RSYNC $SOURCE:/etc/shadow /etc
$RSYNC $SOURCE:/etc/group /etc
$RSYNC $SOURCE:/etc/gshadow /etc

# Copy home so we have user home folders available
# Skip duplicity backup signatures
echo "Copying root"
$RSYNC $SOURCE:/root / --exclude=/root/.cache
# lost+found content is generated by fsck, uninteresting
echo "Copying home"
$RSYNC $SOURCE:/home / --exclude=/home/lost+found

echo "Copying /etc targets"
for i in "${CHERRYPICKED_ETC_TARGETS[@]}"
do
   $RSYNC $SOURCE:$i /etc
done

# Most of your service specific stuff should come here
echo "Copying /srv"
$RSYNC $SOURCE:/srv /

# Make sure stuff which went to /etc/init.d gets correctly reflecte across runlevels,
# as our /srv stuff has placed its own init scripts
for i in  /etc/init.d/*
do
    service=`basename $i`
    # Separate from upstart etc. jobs
    if grep --quiet "$INIT_SCRIPT_MARKER" $i ; then
        update-rc.d $service defaults
    fi
done

# Copy MySQL databases.
# Assume source and target root can connect to MySQL without a password.
# You need to set up /root/.my.cnf file on the source server first
# for the passwordless action.
# http://stackoverflow.com/a/9293090/315168
echo "Copying MySQL databases"
ssh -C $SOURCE mysqldump \
    --skip-lock-tables \
    --add-drop-table \
    --add-drop-database \
    --compact \
    --all-databases \
    > /root/all-mysql.sql

# MySQL dump restore woes
# http://stackoverflow.com/a/21087946/315168
mysql -u root -e "SET FOREIGN_KEY_CHECKS = 0; source /root/all-mysql.sql ; SET FOREIGN_KEY_CHECKS = 1;"

# Remove the key we used for the duplication
rm $TEMP_SSH_KEY

$\"\"$ Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Often you want to automatize something using shell scripting. In a perfect world your script robot works for you without getting tired, without hick-ups, and you can just sit at the front of your desk and sip coffee.

Then we enter the real world: Your network is disconnected. DNS goes downs. Your HTTP hooks and downloads stall. Interprocess communication hangs. Effectively this means that even if your script is running correctly from the point of operating system it won’t finish its work before you finish your cup of coffee.

Below is an example how to create timeouts and notifications in a shell script.

1. Never gonna give you up

Automation must work 100% and you must always know if it doesn’t do that. Otherwise if you cannot trust the automated systems you could have this glorious moment of “oh it has been broken for three months now”. No amount of coffee makes your day after that. Then you spend your nights pondering whether your scripting is running instead of playing Borderlands 2.

Some safety guards around your coffee cup include

Sane timeout thresholds for commands to avoid hang situations. E.g. your automated “make smallapp” should probably not run for 50 hours straight.
Get a notification if a fully automatized process does not end up as expected. So that you find out the problem right away, not three months later. Do not use email as the communication channel, it is unreliable and has awful signal-to-noise ratio. Better solutions include instant messengers (Skype), SMS (check twilio.com)
Get a notification also when a service, which should be running all the time, is restarted.

2. Timeout.bash

The shell scripting (sh, zsh, bash) does not offer built-in tools for timeouting commands by default (please correct me if I am wrong, but stackoverflow.com et. al folks did not know any better). In Bash cookbook, there exist a timeout wrapper script example (The orignal SO.com answer).

The trick is that when the timeout wrapper terminates the command, you’ll get the exit code of SIGTERM (143) or SIGKILL (137) to the parent script. This might not be entirely clear from reading the script.

3. Using timeout wrapper and sending failure notifications to Skype

Below is an example how you can hook timeout to your own script. Here we have a simple continous integration script (ghetto-ci) which polls version control repositories and on any change executes the test suite. A test failure is reported back to Skype by ghetto-ci using Sevabot Skype bot. The loop script uses a server specific (Ubuntu) way to set up a headless X server, so that the tests can run Firefox on the server (can one call a headless Firefox server “mittens”?) The continous integration loop is deployed simply as leaving it running on the screen‘ed terminal on a server.

Some protection we do: The script has extra checks to protect against hung processes by killing them using pkill before starting each test run. Selenium’s WebDriver seem to often cause situations where the Python tests don’t quit cleanly, leaving around all kind of zombie processes. In this case, the test command timeouts using the timeout wrapper and we get a notification to Skype. Based on this we can refine the test running logic, timeout delay and such to make the script more robust allowing us to focus more on drinking the coffee and less watching a looping process running in a UNIX screen for potential failures.

Pardon me for possibly ugly shell scripting code. SH is not my primary language. The example code is also on Github. Please see the orignal blog post for syntax colored example.

#!/bin/bash
#
# Run CI check for every 5 minutes and
# execute tests if svn has been updated.
#
# The script sets up xvfb (X window framebuffer)
# which is used to run the headless Firefox.
#
# We will post a Skype message if the Selenium WebDriver
# has some issues (it often hangs)
#
# We also signal the tests to use a special static Firefox build
# and do not rely (auto-updated) system Firefox
#
# NOTE: This script is NOT sh compliant (echo -n),
# bash needed

# Timeouting commands shell script helper
# http://stackoverflow.com/a/687994/315168
TIMEOUT=timeout.sh

# Which FF binary we use to run the tests
FIXED_FIREFOX=$HOME/ff16/firefox/firefox

# It's me, Mariooo!
SELF=$(readlink -f "$0")

# Skype endpoint information
SKYPE_CHAT_ID="1234567890"

SKYPE_SHARED_SECRET="toholampi"

SEVABOT_SERVER="http://yourserver.com:5000/msg/"

#
#  Helper function to send Skype messages from help scripts.
#  The messages are signed with a shared seceret.
#
#  Parameter 1 is the message
#
function send_skype_message() {
    msg="$1"
    md5=`echo -n "$SKYPE_CHAT_ID$msg$SKYPE_SHARED_SECRET" | md5sum`

    #md5sum pads a '-' to the end of the string. We need to get rid of that.
    for m in $md5; do
        break
    done

    result=`curl --silent --data-urlencode chat="$SKYPE_CHAT_ID" --data-urlencode msg="$msg" --data-urlencode md5="$m" $SEVABOT_SERVER`
    if [ "$result" != "OK" ] ; then
        echo "Error in HTTP communicating to Sevabot: $result"
    fi
}

# Tell the tests to use downgraded FF!6
# which actually works with Selenium
if [ -e $FIXED_FIREFOX ] ; then
    FIREFOX_PATH=$FIREFOX_PATH
    export FIREFOX_PATH
    echo "Using static Firefox 16 build to run the tests"
fi

send_skype_message "♫ ci-loop.sh restarted at $SELF"

while true
do
    # Kill hung testing processes (it might happen)
    pkill -f "bin/test"

    # Purge existing xvfb just in case
    pkill Xvfb

    sleep 5

    echo "Opening virtual X11"

    # Start headless X
    Xvfb :15 -ac -screen 0 1024x768x24 &

    # Tell FF to use this X server
    export DISPLAY=localhost:15.0

    # Run one cycle of continous integration,
    # give it 15 minutes to finish
    echo "Starting test run"
    $TIMEOUT -t 900 continous-integration.sh
    result=$?

    if [ "$result" == "143" ] ; then
        echo "------- CI TIMEOUT OOPS --------"
        send_skype_message "⚡ Continuous integration tests timed out - check the ci-loop.sh screen for problems"
    fi

    sleep 300
done

Voila. Back to the coffee.

$\"\"$ Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Open Source Hacker

Pushing the boundaries of free technology

Tag Archives: shell scripting

Linux server ghetto duplication

Table Of Content

1. Preface

2. More powerful duplication tools

3. Building and running the duplication script

4. Testing the spare server

5. mirror-ubuntu.bash

Timeouting commands in shell scripts

1. Never gonna give you up

2. Timeout.bash

3. Using timeout wrapper and sending failure notifications to Skype