Linux server ghetto duplication

This blog post is about how to duplicate your Linux services and configuration from one server to another. We use simple and hackable SSH, rsync and shell scripting to copy the necessary files to make a cold spare from your existing server installation.

Screen Shot 2014-03-26 at 00.17.09

1. Preface

The method describes here is quite crude and most suitable for making a spare installation of your existing server. In the case you lose your production server, you can boot your cold spare, point your (fail-over) IP address to the spare server and keep business going – something you might want to do if you run mom’n’pop web hosting business. Because of the simplicity of the method it works on default Linux installations, bare metal servers and old-fashioned file systems like Ext4.

The instructions here have been tested with Ubuntu Linux 12.04, but should work with other versions with minor modifications. I have used this method successfully  with Heztner hosting (highly recommended for all the cheapskates out there) by having one production machine and one spare machine. The spare is mirrored weekly. Daily Duplicity backups can be restored on the spare if week is too long data loss period. Though in my case the servers are bare metal, the method works for VPSes as well.

The duplication script might be also useful for setting up a staging server from your production server for software developer and testing.

2. More powerful duplication tools

More fail safe, more engineer-oriented duplication approaches exist, but usually require additional preparation and tuning on the top of the default Linux installation. Applying these to existing running Linux installation might be tricky.

3. Building and running the duplication script

This script is run on the target server (cold spare) and it will clone the content of the source server (actual production machine) on itself. It uses SSH keys and SSH agent to create the initial connection, so make sure you are familiar with them.

Assumptions

  • The target server must be clean Linux installation, the same exact version as on your source server.
  • Your server has standard /etc/passwd user account management. This is copied first so that we correctly preserve file ownership (related ServerFault discussion).
  • Services you run (PHP, Django, Plone, Node.js, you-name-it) are under /srv as recommended by Linux filesystem hierarchy standard
  • Source and target MySQL servers must have initialy same password for the root. Set this when the script runs apt-get install mysql-server for the first time.

Limitations

  • The first run is interactive, as apt-get install asks bunch of stuff for packages like MySQL and Postfix.
  • MySQL dumping and loading the dump does not guarantee all of your data survives intact, especially when skip-lock option is used for the performance reason. For ordinary CMS operations this isn’t a big issue.
  • If you other services beside MySQL needing special flush or lock handling follow the related instructions.

For MySQL duplication make sure you have the following /root/.my.cnf file on the source server, as it allows you to interact with MySQL:

[mysqldump]
user=root
password=YOUR PASSWORD HERE

[client]
user=root
password=YOUR PASSWORD HERE

Run the script inside a screen, because the duplication process may take a while and you do not want to risk losing the process because your local internet connectivity issue.

scp mirror-ubuntu.bash targetserver:/tmp
ssh targetserver
screen 
cd /tmp
bash mirror-ubuntu.bash

4. Testing the spare server

After the duplication script has successfully finished mirroring the server you want to check if the services can be successful started and interacted on the cold spare.

Change internet-facing IP addresses in the related services to be the public IP address of the spare server. E.g. the following files might need changes:

  • /etc/default/varnish
  • /etc/apache2/ports.conf
  • /etc/apache2/sites-available/*

Spoof your local DNS to point the spare server on the tested sites. Edit your local /etc/hosts and add spoofed IP addresses like:

1.2.3.4 www.service1.example.com www.service2.example.com opensourcehacker.com

Access the sites from your web browser, see that database interaction works (login) and file interaction works (upload and download files).

5. mirror-ubuntu.bash

#!/bin/bash
#
# Linux server ghetto duplication script
# Copyright 2014 Mikko Ohtamaa http://opensourcehacker.com
# Licensed under MIT
#

# Everything is copied from this server
[email protected]

# Our network-traffic and speed optimized rsync command
RSYNC="rsync -a --inplace --delete --compress-level=9"

# Which marker string we use to detect custom init.d scripts
INIT_SCRIPT_MARKER="### BEGIN INIT INFO"

# As we will run in screen we need to detach
# from the forwarded SSH agent session and we use a local
# SSH key to perform the operations.
# Also overriding /root destroys our key.
# Create a key we use for the remaining operations.
TEMP_SSH_KEY=/tmp/mirror_rsa

# The software stack might have touched the following places.
# This list is compliled through trial-and-error,
# sweat and cursing.
# We cannot take /etc as a whole, because it contains
# some computer specific stuff (hard disk ids, etc.)
# and copying it as a whole leads to unbootable system.
CHERRYPICKED_ETC_TARGETS=(\
    "/etc/default" \
    "/etc/varnish" \
    "/etc/apache2" \
    "/etc/ssl" \
    "/etc/nginx" \
    "/etc/postfix" \
    "/etc/php5" \
    "/etc/cron.d" \
    "/etc/cron.daily" \
    "/etc/cron.monthly" \
    "/etc/cron.weekly" \
    "/etc/init.d")

# Create a key without a passphrase
# and put the public key on the source server
rm $TEMP_SSH_KEY 2>&1 > /dev/null
ssh-keygen -N '' -f $TEMP_SSH_KEY
ssh-copy-id -i $TEMP_SSH_KEY $SOURCE
# Detach from the currently used SSH agent
# by starting a session specific to this shell
eval `ssh-agent`
ssh-add $TEMP_SSH_KEY

# Assume the system have same Ubuntu base installation and no extra repositories configured.
# Bring target system up to date.
apt-get update -y
apt-get upgrade -y

# TODO: check that the kernel uname is same
# on the source and the target systems

# This is somewhat crude method to try to install all the packages on the source server.
# If the package is missing or replaced this command will happily churn over it
# (apt-get may fail). This MAY cause user interaction with packages
# like Postfix and MySQL which prompt for initial password. Not sure
# what would be the best way to handle this?
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1|while read pkg
do
    apt-get install -y $pkg
done

# As some packages might have changed due to version upgrades and
# deb renames the following step needs interaction
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1

# Copy user credentials first to make sure we
# get the user permissions and ownership correctly.
# http://serverfault.com/a/583336/74975
echo "Copying users"
$RSYNC $SOURCE:/etc/passwd /etc
$RSYNC $SOURCE:/etc/shadow /etc
$RSYNC $SOURCE:/etc/group /etc
$RSYNC $SOURCE:/etc/gshadow /etc

# Copy home so we have user home folders available
# Skip duplicity backup signatures
echo "Copying root"
$RSYNC $SOURCE:/root / --exclude=/root/.cache
# lost+found content is generated by fsck, uninteresting
echo "Copying home"
$RSYNC $SOURCE:/home / --exclude=/home/lost+found

echo "Copying /etc targets"
for i in "${CHERRYPICKED_ETC_TARGETS[@]}"
do
   $RSYNC $SOURCE:$i /etc
done

# Most of your service specific stuff should come here
echo "Copying /srv"
$RSYNC $SOURCE:/srv /

# Make sure stuff which went to /etc/init.d gets correctly reflecte across runlevels,
# as our /srv stuff has placed its own init scripts
for i in  /etc/init.d/*
do
    service=`basename $i`
    # Separate from upstart etc. jobs
    if grep --quiet "$INIT_SCRIPT_MARKER" $i ; then
        update-rc.d $service defaults
    fi
done

# Copy MySQL databases.
# Assume source and target root can connect to MySQL without a password.
# You need to set up /root/.my.cnf file on the source server first
# for the passwordless action.
# http://stackoverflow.com/a/9293090/315168
echo "Copying MySQL databases"
ssh -C $SOURCE mysqldump \
    --skip-lock-tables \
    --add-drop-table \
    --add-drop-database \
    --compact \
    --all-databases \
    > /root/all-mysql.sql

# MySQL dump restore woes
# http://stackoverflow.com/a/21087946/315168
mysql -u root -e "SET FOREIGN_KEY_CHECKS = 0; source /root/all-mysql.sql ; SET FOREIGN_KEY_CHECKS = 1;"

# Remove the key we used for the duplication
rm $TEMP_SSH_KEY

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Moving/copying a WordPress site and domain name related file and database changes

If you move a WordPress installation to a new server, for example make a local develoment, copy, you need to fiddle with the database in order to make the site function correctly.

WordPress has a (flawed) design choice made where it stores full URLs, including domain name, in the database data and thus simply dumping and copying over the database does not help. It would be possible to do this without storing full URLs in the database, as other CMSs (Plone) can do it using a method called virtual host monster (special URL rewrite/proxy patterns in Apache configuration files). This prevents you copying the WordPress installation from the production server to your local computer and doing the development against localhost.

Here are some notes what tables and files you need update and how for WordPress Network (multisite) installations where the multisite method is folder based.

You need to be familiar with UNIX shell.

First dump database from the remote server (ssh + mysqldump instructions) and then rsync WordPress files (rsync instructions).

1. wp-config.php

define('DOMAIN_CURRENT_SITE', 'site.com' );
define('WP_HOME','http://site.com');
define('WP_SITEURL','http://site.com');
define('RELOCATE',true);

Note down username and password for MySQL database from wp-config.php. You are going to need them when running the next commands from MySQL shell.

2. wp_options table

Options storage for the primary site.

update wp_options set option_value="http://site.com" where option_name="home";
update wp_options set option_value="http://site.com" where option_name="siteurl";

3. wp_blogs and wp_sites tables

Here are stored multisite installations of WordPress.

update wp_site set domain="site.com";
update wp_blogs set domain="site.com";

More info in the previous blog post.

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Trouble-filled installation tutorial of WordPress Network (a.k.a multi-site)

WordPress Network is a feature allowing you to manage multiple WordPress installations within the same code base

  • all sites share PHP codebase and plug-ins
  • all sites reside within the same database
  • sites have different settings
  • sites can have centralized admin users through the primary site
  • sites can have different plug-ins enabled and different plug-in settings

This tutorial was written for Ubuntu/Debian server (not shared hosting). It was based on migrating very old WordPress installation (5+ years) to the current version and then turning on the multi-site feature.

Note: All plug-ins might not be multisite compatible. You will find it out hard way.

Note: WordPress is recommended to be installed at a virtual host root for Network feature to work well. Also it makes sense to have the WordPress running in its final domain name, as domain name will be written in all over the settings (in the case you prepare it on the test server first). Use /etc/hosts trick to spoof the domain name of the test server if needed.

These instructions are for WordPress 3.2. They are baed on the reference manul, with my own insight and troubles mixed in. This tutorial is targeted for professionals with advanced UNIX experience and lacks hand-holding.

1. Disable plug-ins

Disable all plug-ins (can be re-enabled later). Otherwise Network feature cannot be turned on.

2.  Enabling Network feature installation

Add line:

define('WP_ALLOW_MULTISITE', true);

to your wp-config.php.

3. Back-up the existing site

Copy PHP files to a back-up folder

cp -r your-site-folder your-site-folder-backup

Dump a copy of database

mysqldump -uDBUSER -p DBNAME > wordpress.sql

See that dump was succesful

ls -lh wordpress.sql
-rw-r--r-- 1 root root 7.0M Aug 22 05:34 wordpress.sql

4. Prepare file-system

You need to create blogs.dir folder which will contain content for each WordPressinstance.

# Ubuntu / Debian uses www-data user for Apache 2
cd wp-content/
mkdir blogs.dir
chown -R www-data:www-data blogs.dir/

5. Turn on Network feature

You should now see Network Tools in Tools menu in Dashboard.

It will display you a bunch of changes you need to do .htaccess and wp-config.php files which are domain name specific (ugly: I am not very keen to have something domain name specific in config files, as it makes moving the site much more difficult, but as WordPress/PHP does not do proper virtual hosting using Virtual Host Base domain name rewrite pattern)

Make sure you put wp-config.php changes in the middle of the file before  wp-settings.php line or you will be seriously screwed up.

Sanitized wp-login.php changes look somewhat like:

define('DB_COLLATE', ...)
define( 'AUTH_KEY', 'x' );
define( 'SECURE_AUTH_KEY', 'x' );
define( 'LOGGED_IN_KEY', 'x' );
define( 'NONCE_KEY', 'x' );
define( 'AUTH_SALT', 'x' );
define( 'SECURE_AUTH_SALT', 'x' );
define( 'LOGGED_IN_SALT', 'x' );
define( 'NONCE_SALT', 'x' );
...
// snip
...

define('WP_ALLOW_MULTISITE', true);

// These lines MUST come after wp-settings.php include
define( 'SUBDOMAIN_INSTALL', false );
$base = '/';
define( 'DOMAIN_CURRENT_SITE', 'site.com' );
define( 'PATH_CURRENT_SITE', '/' );
define( 'SITE_ID_CURRENT_SITE', 1 );
define( 'BLOG_ID_CURRENT_SITE', 1 );

WordPress failed to detect DOMAIN_CURRENT_SITE for me currently, so I had to fix it manually.

6. Relogin

Log-out and log in again

7. Fail #1: You do not have sufficient permissions to access this page

Well… it didn’t go well. The first login for me produced an error page:

You do not have sufficient permissions to access this page.

To honor PHP programming practices there is no log information or any hint what actually went wrong. So you can only guess.

It turns out wp-config.php edit can fail if you add change after wp-settings.php include (PHP include mechanism and define() is quite hacky and WordPress cannot warn you if you fail).

If you happened to do this then I suggest you roll back the database and files in this point and start from zero.

8. Fail #2: Error establishing database connection

After fixing wp-config.php settings order I ge the white screen of death

Error establishing database connection

This comes up when I set

define( 'MULTISITE', true );

Looks like the internet is full of posts and less and less uselful answers about what is going on. Also, WordPress Codex etc. material does not provide very helpful material what should happen when MULTISITE is turned on, when tables are migrated and so on. The only way to solve this is seems to dive head first to WordPress PHP. Argh. Well… if I cannot fix it, then who can`?

So I had to modify wp_functions.php to give me more meaningful trace back for the white screen of death:

function dead_db() {   
        global $wpdb;

        throw new Exception("PHP is pile of shit");

This way you can actually know where the error happens in the code.

Which leads to me functions.php is_blog_installed() where after adding in new die() statement I get:

Cannot find wp_users

Hmmm. I wonder why… Especially this comment is useful:

        // Loop over the WP tables.  If none exist, then scratch install is allowed.
        // If one or more exist, suggest table repair since we got here because the options
        // table could not be accessed.

So it thinks the blog is not installed, but still finds some tables and dies with useless “Error establish database connection” error.

More closely examination with print_r() shows that $alloptions variable used in is_blog_installed() is empty.

It tries to load options from a table called wp_2_options which hints that it is a multisite enabled table which has not yet been created. This is strange as we clearly say the site id is 1 in wp-config.php. So looks like WordPress think the site id is wrong.

Since there is no way to trace back where $wpdb array containing this bad is populated, the next grep goes with SITE_ID_CURRENT_SITE.

wpmu_current_site() seems to return correct id 1. False lead.

More careful examination of  print_r() $wpdb tells blog id is 2 (note: blog id differs from site id…).

Grepping again tells there exist $wpdb function called set_blog_id(). It seems to be called in a function called switch_to_blog(). However, putting exception there tells that it is never called.

Finally finally MySQL dump of wp_blogs table shows that it is damaged due to earlier run of multisite migration with bad config:

mysql> select * from wp_blogs;
+---------+---------+----------------------+--------------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
| blog_id | site_id | domain               | path         | registered          | last_updated        | public | archived | mature | spam | deleted | lang_id |
+---------+---------+----------------------+--------------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
|       1 |       1 | site1.fi             | /xxxx-wp/    | 2011-08-22 08:36:46 | 0000-00-00 00:00:00 |      1 | 0        |      0 |    0 |       0 |       0 |
|       2 |       1 | site2.com            | /            | 2011-08-22 10:47:51 | 0000-00-00 00:00:00 |      1 | 0        |      0 |    0 |       0 |       0 |
+---------+---------+----------------------+--------------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
2 rows in set (0.00 sec)

Purge this table from bad data

delete from wp_blogs;

Also drop other multi-site config;

drop table wp_site;
drop table wp_sitemeta;

Now when you go to http://site.com/wp-admin/network.php it should be in the pristine state and you can create multi-site configuration from the scratch, in a working manner.

Make sure that Server name field show the correct value on the WordPress Network creation screen.

Press Install.

Re-enable multi-site in wp-config.php:

define( 'MULTISITE', true );

Well.. still failure. But this finally seemed to fix the site when we force the blog id (looks like it is incremental column which did not reset properly):

 update wp_blogs set blog_id=1 where blog_id=3;

Still it is looking for bad URL fix we can fix by forcing WordPress RELOCATE. RELOCATE setting in wp-config.php will update site URL in the database for the current wp-login.php page URL.

define('RELOCATE',true);

Now go to login page. I was finally able to login to migrated multisite instance.

Post-mortem: Error establishing database connection error message was misleading. The real error was along the lines “Multi-site configuration for site id X is broken.” However this is not very usual error and WordPress does not correctly check this error situation.

9. Relogin (now with success)

Now you should see Network Adminin your username name in the top right corner. Start adding those sites!

10. Fail #3: Not Found

So far so good. However, after creating a new site and trying to access the site through path (e.g. yoursite.com/subsite) it gives 404 Not Found. Lovely.

The error instantly hints that maybe something is from with Apache and .htaccess configuration.

So it seems – changes from WordPress Network installation didn’t end up correctly to .htaccess. Just to refresh the memory the correct .htaccess settings are:

# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]

# uploaded files
RewriteRule ^([_0-9a-zA-Z-]+/)?files/(.+) wp-includes/ms-files.php?file=$2 [L]

# add a trailing slash to /wp-admin
RewriteRule ^([_0-9a-zA-Z-]+/)?wp-admin$ $1wp-admin/ [R=301,L]

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule  ^[_0-9a-zA-Z-]+/(wp-(content|admin|includes).*) $1 [L]
RewriteRule  ^[_0-9a-zA-Z-]+/(.*\.php)$ $1 [L]
RewriteRule . index.php [L]
# END WordPress

Even after fixing .htaccess it stills gives 404. Looks like .htaccess files were not properly enabled in the virtual host config /etc/apache2/sites-enabled directory. This fixed the problem:

<VirtualHost *>
    ServerName site.com

    <Directory /var/www/site>
    AllowOverride All
    </Directory>

</VirtualHost>

11. Fail #4: Error establishing database connection for new multi-site

Ok. This time we can already guess the problem. Blog ids and site ids don’t match somewhere.

MySQL command prompt reveals what we suspect:

mysql> select * from wp_blogs;
+---------+---------+-------------+------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
| blog_id | site_id | domain      | path | registered          | last_updated        | public | archived | mature | spam | deleted | lang_id |
+---------+---------+-------------+------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
|       1 |       1 | site.com    | /    | 2011-08-22 14:57:12 | 0000-00-00 00:00:00 |      1 | 0        |      0 |    0 |       0 |       0 |
|       4 |       1 | site.com    | /de/ | 2011-08-22 15:10:34 | 0000-00-00 00:00:00 |      1 | 0        |      0 |    0 |       0 |       0 |
+---------+---------+-------------+------+---------------------+---------------------+--------+----------+--------+------+---------+---------+
2 rows in set (0.00 sec)

blog_id is currently having its own life.

Delete the new site (/de/) through WordPress dashboard.

When checking wp_blogs table more closely we notice that blog_id is auto-increment column:

describe wp_blogs;
+--------------+---------------+------+-----+---------------------+----------------+
| Field        | Type          | Null | Key | Default             | Extra          |
+--------------+---------------+------+-----+---------------------+----------------+
| blog_id      | bigint(20)    | NO   | PRI | NULL                | auto_increment |
| site_id      | bigint(20)    | NO   |     | 0                   |                |
| domain       | varchar(200)  | NO   | MUL |                     |                |
| path         | varchar(100)  | NO   |     |                     |                |
| registered   | datetime      | NO   |     | 0000-00-00 00:00:00 |                |
| last_updated | datetime      | NO   |     | 0000-00-00 00:00:00 |                |
| public       | tinyint(2)    | NO   |     | 1                   |                |
| archived     | enum('0','1') | NO   |     | 0                   |                |
| mature       | tinyint(2)    | NO   |     | 0                   |                |
| spam         | tinyint(2)    | NO   |     | 0                   |                |
| deleted      | tinyint(2)    | NO   |     | 0                   |                |
| lang_id      | int(11)       | NO   | MUL | 0                   |                |
+--------------+---------------+------+-----+---------------------+----------------+
12 rows in set (0.00 sec)

We can reset the counter via MySQL command:

ALTER TABLE wp_blogs AUTO_INCREMENT=2

12. Migrating multi-sites to new domain name

As now we are familiar with wp_site and wp_blogs tables we know how to manipulate these tables if we want to migrate our test site to the production domain name:

update wp_site set domain="new.site.com";
update wp_blogs set domain="new.site.com";

Change value of DOMAIN_CURRENT_SITE in wp-config.php

define( 'DOMAIN_CURRENT_SITE', 'new.site.com' );

After this you can access the primary site at new.site.com. Then you can access each subsite settings and toggle on Update siteurl and home. Save site settings to force new site URL for the subsite.

Note: It was still hiding the old URL value for the primary site somewhere and I didn’t track down it yet.

13. More info

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

One-liner to copy remote MySQL database to local computer

The following commands dump a MySQL database from a remote server and create a corresponding database on the local computer.

The instructions have been tested on OSX and Linux (Ubuntu/Debian). On-line SSH compression is applied, so transferring SQL files, which are text content and compress well, should be around 6x faster than normal.

(Well… actually the script is six lines, but because this is my blog I’ll decide it doesn’t count)

The script

  • Remotely runs mysqldump and puts the result to a local file
  • Creates a MySQL database and corresponding user with full access to this database
  • Reads the content of mysqldump to the newly created database
 ssh [email protected] -C -o CompressionLevel=9 mysqldump -u YOURDATABASEUSER --password=YOURDATABASEPASSWORD --skip-lock-tables --add-drop-table YOURDATABASENAME > YOURDATABASENAME.sql
mysql -uroot -p
create database YOURDATABASENAME;
connect YOURDATABASENAME;
source YOURDATABASENAME.sql
GRANT ALL ON YOURDATABASENAME.* TO 'YOURDATABASEUSER'@'localhost' identified by 'YOURDATABASEPASSWORD';

Leave out create database and GRANT for the subsequent runs – all data on the local computer will be replaced.

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+