This blog post is about how to duplicate your Linux services and configuration from one server to another. We use simple and hackable SSH, rsync and shell scripting to copy the necessary files to make a cold spare from your existing server installation.

Preface

The method describes here is quite crude and most suitable for making a spare installation of your existing server. In the case you lose your production server, you can boot your cold spare, point your (fail-over) IP address to the spare server and keep business going – something you might want to do if you run mom’n’pop web hosting business. Because of the simplicity of the method it works on default Linux installations, bare metal servers and old-fashioned file systems like Ext4.

The instructions here have been tested with Ubuntu Linux 12.04, but should work with other versions with minor modifications. I have used this method successfully  with Heztner hosting (highly recommended for all the cheapskates out there) by having one production machine and one spare machine. The spare is mirrored weekly. Daily Duplicity backups can be restored on the spare if week is too long data loss period. Though in my case the servers are bare metal, the method works for VPSes as well.

The duplication script might be also useful for setting up a staging server from your production server for software developer and testing.

More powerful duplication tools

More fail safe, more engineer-oriented duplication approaches exist, but usually require additional preparation and tuning on the top of the default Linux installation. Applying these to existing running Linux installation might be tricky.

• Use virtual machines like KVM and snapshot them, then migrate the snapshotted copy on another server.
• Use a file system supporting snapshots like Brfs or ZFS.
• Replicate your databases using native replication (MySQL, PostgreSQL) – if you cannot afford losing few database rows (hot spare).
• Replicate your file system (hot spare)
• (If you have any more ideas how to make hot or cold spares please post in the blog comments)

Building and running the duplication script

This script is run on the target server (cold spare) and it will clone the content of the source server (actual production machine) on itself. It uses SSH keys and SSH agent to create the initial connection, so make sure you are familiar with them.

Assumptions

• The target server must be clean Linux installation, the same exact version as on your source server.
• Your server has standard /etc/passwd user account management. This is copied first so that we correctly preserve file ownership (related ServerFault discussion).
• Services you run (PHP, Django, Plone, Node.js, you-name-it) are under /srv as recommended by Linux filesystem hierarchy standard
• Source and target MySQL servers must have initialy same password for the root. Set this when the script runs apt-get install mysql-server for the first time.

Limitations

• The first run is interactive, as apt-get install asks bunch of stuff for packages like MySQL and Postfix.
• MySQL dumping and loading the dump does not guarantee all of your data survives intact, especially when skip-lock option is used for the performance reason. For ordinary CMS operations this isn’t a big issue.
• If you other services beside MySQL needing special flush or lock handling follow the related instructions.

For MySQL duplication make sure you have the following /root/.my.cnf file on the source server, as it allows you to interact with MySQL:

[mysqldump]
user=root
password=YOUR PASSWORD HERE

[client]
user=root
password=YOUR PASSWORD HERE

Run the script inside a screen, because the duplication process may take a while and you do not want to risk losing the process because your local internet connectivity issue.

scp mirror-ubuntu.bash targetserver:/tmp
ssh targetserver
screen 
cd /tmp
bash mirror-ubuntu.bash

Testing the spare server

After the duplication script has successfully finished mirroring the server you want to check if the services can be successful started and interacted on the cold spare.

Change internet-facing IP addresses in the related services to be the public IP address of the spare server. E.g. the following files might need changes:

• /etc/default/varnish
• /etc/apache2/ports.conf
• /etc/apache2/sites-available/*

Spoof your local DNS to point the spare server on the tested sites. Edit your local /etc/hosts and add spoofed IP addresses like:

1.2.3.4 www.service1.example.com www.service2.example.com opensourcehacker.com

Access the sites from your web browser, see that database interaction works (login) and file interaction works (upload and download files).

mirror-ubuntu.bash

#!/bin/bash
#
# Linux server ghetto duplication script
# Copyright 2014 Mikko Ohtamaa http://opensourcehacker.com
# Licensed under MIT
#

# Everything is copied from this server
SOURCE=root@myserv.example.com

# Our network-traffic and speed optimized rsync command
RSYNC="rsync -a --inplace --delete --compress-level=9"

# Which marker string we use to detect custom init.d scripts
INIT_SCRIPT_MARKER="### BEGIN INIT INFO"

# As we will run in screen we need to detach
# from the forwarded SSH agent session and we use a local
# SSH key to perform the operations.
# Also overriding /root destroys our key.
# Create a key we use for the remaining operations.
TEMP_SSH_KEY=/tmp/mirror_rsa

# The software stack might have touched the following places.
# This list is compliled through trial-and-error,
# sweat and cursing.
# We cannot take /etc as a whole, because it contains
# some computer specific stuff (hard disk ids, etc.)
# and copying it as a whole leads to unbootable system.
CHERRYPICKED_ETC_TARGETS=(\
    "/etc/default" \
    "/etc/varnish" \
    "/etc/apache2" \
    "/etc/ssl" \
    "/etc/nginx" \
    "/etc/postfix" \
    "/etc/php5" \
    "/etc/cron.d" \
    "/etc/cron.daily" \
    "/etc/cron.monthly" \
    "/etc/cron.weekly" \
    "/etc/init.d")

# Create a key without a passphrase
# and put the public key on the source server
rm $TEMP_SSH_KEY 2>&1 > /dev/null
ssh-keygen -N '' -f $TEMP_SSH_KEY
ssh-copy-id -i $TEMP_SSH_KEY $SOURCE
# Detach from the currently used SSH agent
# by starting a session specific to this shell
eval `ssh-agent`
ssh-add $TEMP_SSH_KEY

# Assume the system have same Ubuntu base installation and no extra repositories configured.
# Bring target system up to date.
apt-get update -y
apt-get upgrade -y

# TODO: check that the kernel uname is same
# on the source and the target systems

# This is somewhat crude method to try to install all the packages on the source server.
# If the package is missing or replaced this command will happily churn over it
# (apt-get may fail). This MAY cause user interaction with packages
# like Postfix and MySQL which prompt for initial password. Not sure
# what would be the best way to handle this?
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1|while read pkg
do
    apt-get install -y $pkg
done

# As some packages might have changed due to version upgrades and
# deb renames the following step needs interaction
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1

# Copy user credentials first to make sure we
# get the user permissions and ownership correctly.
# http://serverfault.com/a/583336/74975
echo "Copying users"
$RSYNC $SOURCE:/etc/passwd /etc
$RSYNC $SOURCE:/etc/shadow /etc
$RSYNC $SOURCE:/etc/group /etc
$RSYNC $SOURCE:/etc/gshadow /etc

# Copy home so we have user home folders available
# Skip duplicity backup signatures
echo "Copying root"
$RSYNC $SOURCE:/root / --exclude=/root/.cache
# lost+found content is generated by fsck, uninteresting
echo "Copying home"
$RSYNC $SOURCE:/home / --exclude=/home/lost+found

echo "Copying /etc targets"
for i in "${CHERRYPICKED_ETC_TARGETS[@]}"
do
   $RSYNC $SOURCE:$i /etc
done

# Most of your service specific stuff should come here
echo "Copying /srv"
$RSYNC $SOURCE:/srv /

# Make sure stuff which went to /etc/init.d gets correctly reflecte across runlevels,
# as our /srv stuff has placed its own init scripts
for i in  /etc/init.d/*
do
    service=`basename $i`
    # Separate from upstart etc. jobs
    if grep --quiet "$INIT_SCRIPT_MARKER" $i ; then
        update-rc.d $service defaults
    fi
done

# Copy MySQL databases.
# Assume source and target root can connect to MySQL without a password.
# You need to set up /root/.my.cnf file on the source server first
# for the passwordless action.
# http://stackoverflow.com/a/9293090/315168
echo "Copying MySQL databases"
ssh -C $SOURCE mysqldump \
    --skip-lock-tables \
    --add-drop-table \
    --add-drop-database \
    --compact \
    --all-databases \
    > /root/all-mysql.sql

# MySQL dump restore woes
# http://stackoverflow.com/a/21087946/315168
mysql -u root -e "SET FOREIGN_KEY_CHECKS = 0; source /root/all-mysql.sql ; SET FOREIGN_KEY_CHECKS = 1;"

# Remove the key we used for the duplication
rm $TEMP_SSH_KEY

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+