Tag Archives: bash

Linux server ghetto duplication

Posted on by
4

This blog post is about how to duplicate your Linux services and configuration from one server to another. We use simple and hackable SSH, rsync and shell scripting to copy the necessary files to make a cold spare from your existing server installation.

Screen Shot 2014-03-26 at 00.17.09

Table Of Content

1. Preface

2. More powerful duplication tools

3. Building and running the duplication script

4. Testing the spare server

5. mirror-ubuntu.bash

1. Preface

The method describes here is quite crude and most suitable for making a spare installation of your existing server. In the case you lose your production server, you can boot your cold spare, point your (fail-over) IP address to the spare server and keep business going – something you might want to do if you run mom’n’pop web hosting business. Because of the simplicity of the method it works on default Linux installations, bare metal servers and old-fashioned file systems like Ext4.

The instructions here have been tested with Ubuntu Linux 12.04, but should work with other versions with minor modifications. I have used this method successfully  with Heztner hosting (highly recommended for all the cheapskates out there) by having one production machine and one spare machine. The spare is mirrored weekly. Daily Duplicity backups can be restored on the spare if week is too long data loss period. Though in my case the servers are bare metal, the method works for VPSes as well.

The duplication script might be also useful for setting up a staging server from your production server for software developer and testing.

2. More powerful duplication tools

More fail safe, more engineer-oriented duplication approaches exist, but usually require additional preparation and tuning on the top of the default Linux installation. Applying these to existing running Linux installation might be tricky.

3. Building and running the duplication script

This script is run on the target server (cold spare) and it will clone the content of the source server (actual production machine) on itself. It uses SSH keys and SSH agent to create the initial connection, so make sure you are familiar with them.

Assumptions

  • The target server must be clean Linux installation, the same exact version as on your source server.
  • Your server has standard /etc/passwd user account management. This is copied first so that we correctly preserve file ownership (related ServerFault discussion).
  • Services you run (PHP, Django, Plone, Node.js, you-name-it) are under /srv as recommended by Linux filesystem hierarchy standard
  • Source and target MySQL servers must have initialy same password for the root. Set this when the script runs apt-get install mysql-server for the first time.

Limitations

  • The first run is interactive, as apt-get install asks bunch of stuff for packages like MySQL and Postfix.
  • MySQL dumping and loading the dump does not guarantee all of your data survives intact, especially when skip-lock option is used for the performance reason. For ordinary CMS operations this isn’t a big issue.
  • If you other services beside MySQL needing special flush or lock handling follow the related instructions.

For MySQL duplication make sure you have the following /root/.my.cnf file on the source server, as it allows you to interact with MySQL:

[mysqldump]
user=root
password=YOUR PASSWORD HERE

[client]
user=root
password=YOUR PASSWORD HERE

Run the script inside a screen, because the duplication process may take a while and you do not want to risk losing the process because your local internet connectivity issue.

scp mirror-ubuntu.bash targetserver:/tmp
ssh targetserver
screen 
cd /tmp
bash mirror-ubuntu.bash

4. Testing the spare server

After the duplication script has successfully finished mirroring the server you want to check if the services can be successful started and interacted on the cold spare.

Change internet-facing IP addresses in the related services to be the public IP address of the spare server. E.g. the following files might need changes:

  • /etc/default/varnish
  • /etc/apache2/ports.conf
  • /etc/apache2/sites-available/*

Spoof your local DNS to point the spare server on the tested sites. Edit your local /etc/hosts and add spoofed IP addresses like:

1.2.3.4 www.service1.example.com www.service2.example.com opensourcehacker.com

Access the sites from your web browser, see that database interaction works (login) and file interaction works (upload and download files).

5. mirror-ubuntu.bash

#!/bin/bash
#
# Linux server ghetto duplication script
# Copyright 2014 Mikko Ohtamaa http://opensourcehacker.com
# Licensed under MIT
#

# Everything is copied from this server
SOURCE=root@myserv.example.com

# Our network-traffic and speed optimized rsync command
RSYNC="rsync -a --inplace --delete --compress-level=9"

# Which marker string we use to detect custom init.d scripts
INIT_SCRIPT_MARKER="### BEGIN INIT INFO"

# As we will run in screen we need to detach
# from the forwarded SSH agent session and we use a local
# SSH key to perform the operations.
# Also overriding /root destroys our key.
# Create a key we use for the remaining operations.
TEMP_SSH_KEY=/tmp/mirror_rsa

# The software stack might have touched the following places.
# This list is compliled through trial-and-error,
# sweat and cursing.
# We cannot take /etc as a whole, because it contains
# some computer specific stuff (hard disk ids, etc.)
# and copying it as a whole leads to unbootable system.
CHERRYPICKED_ETC_TARGETS=(\
    "/etc/default" \
    "/etc/varnish" \
    "/etc/apache2" \
    "/etc/ssl" \
    "/etc/nginx" \
    "/etc/postfix" \
    "/etc/php5" \
    "/etc/cron.d" \
    "/etc/cron.daily" \
    "/etc/cron.monthly" \
    "/etc/cron.weekly" \
    "/etc/init.d")

# Create a key without a passphrase
# and put the public key on the source server
rm $TEMP_SSH_KEY 2>&1 > /dev/null
ssh-keygen -N '' -f $TEMP_SSH_KEY
ssh-copy-id -i $TEMP_SSH_KEY $SOURCE
# Detach from the currently used SSH agent
# by starting a session specific to this shell
eval `ssh-agent`
ssh-add $TEMP_SSH_KEY

# Assume the system have same Ubuntu base installation and no extra repositories configured.
# Bring target system up to date.
apt-get update -y
apt-get upgrade -y

# TODO: check that the kernel uname is same
# on the source and the target systems

# This is somewhat crude method to try to install all the packages on the source server.
# If the package is missing or replaced this command will happily churn over it
# (apt-get may fail). This MAY cause user interaction with packages
# like Postfix and MySQL which prompt for initial password. Not sure
# what would be the best way to handle this?
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1|while read pkg
do
    apt-get install -y $pkg
done

# As some packages might have changed due to version upgrades and
# deb renames the following step needs interaction
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1

# Copy user credentials first to make sure we
# get the user permissions and ownership correctly.
# http://serverfault.com/a/583336/74975
echo "Copying users"
$RSYNC $SOURCE:/etc/passwd /etc
$RSYNC $SOURCE:/etc/shadow /etc
$RSYNC $SOURCE:/etc/group /etc
$RSYNC $SOURCE:/etc/gshadow /etc

# Copy home so we have user home folders available
# Skip duplicity backup signatures
echo "Copying root"
$RSYNC $SOURCE:/root / --exclude=/root/.cache
# lost+found content is generated by fsck, uninteresting
echo "Copying home"
$RSYNC $SOURCE:/home / --exclude=/home/lost+found

echo "Copying /etc targets"
for i in "${CHERRYPICKED_ETC_TARGETS[@]}"
do
   $RSYNC $SOURCE:$i /etc
done

# Most of your service specific stuff should come here
echo "Copying /srv"
$RSYNC $SOURCE:/srv /

# Make sure stuff which went to /etc/init.d gets correctly reflecte across runlevels,
# as our /srv stuff has placed its own init scripts
for i in  /etc/init.d/*
do
    service=`basename $i`
    # Separate from upstart etc. jobs
    if grep --quiet "$INIT_SCRIPT_MARKER" $i ; then
        update-rc.d $service defaults
    fi
done

# Copy MySQL databases.
# Assume source and target root can connect to MySQL without a password.
# You need to set up /root/.my.cnf file on the source server first
# for the passwordless action.
# http://stackoverflow.com/a/9293090/315168
echo "Copying MySQL databases"
ssh -C $SOURCE mysqldump \
    --skip-lock-tables \
    --add-drop-table \
    --add-drop-database \
    --compact \
    --all-databases \
    > /root/all-mysql.sql

# MySQL dump restore woes
# http://stackoverflow.com/a/21087946/315168
mysql -u root -e "SET FOREIGN_KEY_CHECKS = 0; source /root/all-mysql.sql ; SET FOREIGN_KEY_CHECKS = 1;"

# Remove the key we used for the duplication
rm $TEMP_SSH_KEY

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Sevabot – Skype bot with UNIX scripting and HTTP webhooks integration

Posted on by
1

Skype is the most popular IM client with over 200 million monthly users. It’s easy and reliable: anyone can use it anywhere, even on mobile. When you are working with virtual teams, Skype group chats become vital tool of communication.

Skype lacks good built-in support for external service messages, or robots, sending and receiving automated messages. Sevabot, a friendly Skypebot, addresses this situation by allowing you to script Skype easily, and even remotely, over simple HTTP interface and with any UNIX scripting language. You can even send Skype messages to your group chat directly from a web browser Javascript.

Image

Sevabot loves you.

1. Use cases

Sevabot is geared to be an AI support person for a virtual development and operations teams.

  • Receive development status information to Skype chat (continuous integration like Jenkins, Travis, Github and Subversion commits)
  • Get operational alerts like overloaded servers, service down (e.g. Zabbix monitoring alerts)
  • Add your own group chat commands to make Sevabot perform automatized tasks for you (you can write scripts in *any* UNIX supported programming language)

Image

The scope of Sevabot does not need to be limited to software development: You could, e.g., make Sevabot interact with Google Docs or Salesforce.

2. Installation

Sevabot can be run on Linux server, Linux desktop and OSX desktop. Due to how Skype is built you need to run a Skype GUI client inside a virtual X server to deploy Skype on a production server. Vagrant automatized server configuration is supported for automatically creating and deploying a virtual machine running Sevabot.

Sevabot can be run Windows, in theory, but the authors have not had inspiration to explore this cunning option.

3. Programming

Sevabot is written in Python. Scripting Sevabot does not, however, need Python knowledge as

  • Sevabot modules are normal UNIX scripts which you can write in Bash, Python, Perl, Ruby, Javascript, Haskel or whatever is your daily drug
  • External services call Sevabot over HTTP webhooks and any programming language can do HTTP requets

You can even script Sevabot without programming knowledge, as there exist HTTP webhook middleman services like Zapier, which allow you to connect event sources (e..g Github API) and sinks (Sevabot HTTP interface).

In fact Sevabot is a middleman between Flask web server framework (enables HTTP interface) and Skype4Py API (control Skype GUI client with Python).

Here is a simple example how to send a message to a Skype chat from a Bash shell script. It’s a Subversion post-commit hook which displays the commit message in a specific Skype group chat (full example):

#!/bin/bash

repo="$1"

rev="$2"

svnlook="/usr/bin/svnlook"

# Get last commit author
author=`$svnlook author $repo`

# Get last commit message
commit_message=`$svnlook log $repo`

# List of changed files
changed=`$svnlook changed $repo`

# Chat id
chat="YOUR-CHAT-ID-HERE"

# Sevabot endpoint informaiton
# Shared secret
secret="YOUR-SHARED-SECRET-HERE"
msgaddress="http://YOURSERVER.COM:5000/msg/"

# Create chat message
msg="★ $author - $commit_message $changed"

# Sign the message with MD5
md5=`echo -n "$chat$msg$secret" | md5sum`

#md5sum command prints a '-' to the end. Let's get rid of that.
for m in $md5; do
    break
done

# Call Sevabot's HTTP interface
curl $msgaddress --data-urlencode chat="$chat" --data-urlencode msg="$msg" --data-urlencode md5="$m"

exit 0

4. You want it – come for us

Go to Sevabot Github project page for more information. Also check the community information; we have around five active contributors currently.

ImageFeliz Natal e boas festas, Mikko and Sevabot

 

 

 

 

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Configuring your Python application using environment variables

Status

2
Image

This blog post is a short explanation how to give and consume environment variables in your (Python) code. It’s a nice little trick, but if you are not well-versed in UNIX systems you might not know it.

Table Of Content

1. Configuring your application

2. Using environment variables in UNIX

3. Another trick: socket.gethostname()

Please see the original blog post for syntax color examples.

1. Configuring your application

An application can be configured usually in three ways. Through

  • command-line arguments (I prefer simplistic plac library with Python)
  • configuration files (I prefer YAML or Python standard library ConfigParser for INI style files)
  • environment variables

The two first ones are the most well-known methods. However, the last option is the easiest for quick and dirty hacks.

2. Using environment variables in UNIX

Note: Environment variables work alike in UNIX and Windows. However, I have no longer valid experience about Windows and how to set environment variables in the latest cmd.exe incarnations. (I used to write AUTOEXEC.bats for DOS, but that’s like long time ago). So if there are Windows gurus around please leave a comment.

Environment variables are

  • Easy way to make deployment specific changes in your application: run application differently on different computers by starting the app with a different command
  • Pass parameters for your application when you cannot parse command line yourself. E.g. your application is using Django or Zope launcher script and poking the launch script arguments is not easy.
  • Can be consumed everywhere inside your application, whether it is module level code (run on import), inside function, inside class and so on…
  • Needs only two lines to set up (import os ; os.environ.get())
  • Environment variables are especially useful if you want to have, in your scripts, some secret variables (username, password) which must not be committed on a public code repositories like Github (example). Note: the command lines are public for the all users of the UNIX system, so don’t do this on a shared server if you cannot expose the information to other server users.

Whether and how using environment variables is a good practice is debatable. But it definitely saves your butt when you need to make something quick and dirty.

You can give environment variables to your application on UNIX command line by simply prefixing your command with them (bash, sh, etc. shells):

DEMO_MODE=true python myapp.py

Or if you want to make the effect persistent for the current shell session use export (bash style)

export DEMO_MODE=true
python myapp.py
# Again
python myapp.py

Then you can read out for this environment variable easily using os.environ dictionary (pseudo code)

import os

# Just to check for the existing of DEMO_MODE environment variable,
# but you could also compare its value, pass it forward and so on
DEMO_MODE = os.environ.get("DEMO_MODE", None)

# We make some of the class members conditional 
# by given environment vaiables
class MyForm(object):

    name = StringField()

    if not DEMO_MODE:
       secret = PasswordField()

If you are using Python as the configuration language of your application (Django’s settings.py, Pyramid’s Configurator) you can also use this trick there to make some settings conditional which you normally hardcode in Python. Example settings.py:

import os

# Set Django debug mode based on environment variable
DEBUG = "DEBUG_MODE" in os.environ

Then, on the staging server, you would launch your Django application as

DEBUG_MODE=xxx python manage.py runserver

You can also give several environment variables once:

DEBUG_MODE=xxx API_SECRET=yyy python manage.py runserver

3. Another trick: socket.gethostname()

This is a way to bind certain settings to certain computers in your Python code by checking the name of the computer as returned by gethostname() call. This is useful if you don’t want to forget giving a specific launcher command on a specific server.

import socket

# Temporary hack to run hidden fields on a demo server
if socket.gethostname() in ['mikko-laptop', 'demoserver.example.com']:
     import special_config as config 
else:
     import normal_config as config

The only downside is that sometimes the hostname is not stable: I have noticed this behavior on OSX when connecting different networks on my laptop and apparently the name is given by the network.

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Timeouting commands in shell scripts

Posted on by
Reply

Often you want to automatize something using shell scripting. In a perfect world your script robot works for you without getting tired, without hick-ups, and you can just sit at the front of your desk and sip coffee.

Image

Then we enter the real world: Your network is disconnected. DNS goes downs. Your HTTP hooks and downloads stall. Interprocess communication hangs. Effectively this means that even if your script is running correctly from the point of operating system it won’t finish its work before you finish your cup of coffee.

Below is an example how to create timeouts and notifications in a shell script.

1. Never gonna give you up

(c) Rick Astley

Automation must work 100% and you must always know if it doesn’t do that. Otherwise if you cannot trust the automated systems you could have this glorious moment of “oh it has been broken for three months now”. No amount of coffee makes your day after that. Then you spend your nights pondering whether your scripting is running instead of playing Borderlands 2.

Some safety guards around your coffee cup include

  • Sane timeout thresholds for commands to avoid hang situations. E.g. your automated “make smallapp” should probably not run for 50 hours straight.
  • Get a notification if a fully automatized process does not end up as expected. So that you find out the problem right away, not three months later. Do not use email as the communication channel, it is unreliable and has awful signal-to-noise ratio. Better solutions include instant messengers (Skype), SMS (check twilio.com)
  • Get a notification also when a service, which should be running all the time, is restarted.

2. Timeout.bash

The shell scripting (sh, zsh, bash)  does not offer built-in tools for timeouting commands by default (please correct me if I am wrong, but stackoverflow.com et. al folks did not know any better). In Bash cookbook, there exist a timeout wrapper script example (The orignal SO.com answer).

The trick is that when the timeout wrapper terminates the command, you’ll get the exit code of SIGTERM (143) or SIGKILL (137) to the parent script. This might not be entirely clear from reading the script.

3. Using timeout wrapper and sending failure notifications to Skype

Below is an example how you can hook timeout to your own script. Here we have a simple continous integration script (ghetto-ci) which polls version control repositories and on any change executes the test suite. A test failure is reported back to Skype by ghetto-ci using Sevabot Skype bot. The loop script uses a server specific (Ubuntu) way to set up a headless X server, so that the tests can run Firefox on the server (can one call a headless Firefox server “mittens”?) The continous integration loop is deployed simply as leaving it running on the screen‘ed terminal on a server.

Some protection we do: The script has extra checks to protect against hung processes by killing them using pkill before starting each test run. Selenium’s WebDriver seem to often cause situations where the Python tests don’t quit cleanly, leaving around all kind of zombie processes. In this case, the test command timeouts using the timeout wrapper and we get a notification to Skype. Based on this we can refine the test running logic, timeout delay and such to make the script more robust allowing us to focus more on drinking the coffee and less watching a looping process running in a UNIX screen for potential failures.

Pardon me for possibly ugly shell scripting code. SH is not my primary language. The example code is also on Github. Please see the orignal blog post for syntax colored example.

#!/bin/bash
#
# Run CI check for every 5 minutes and
# execute tests if svn has been updated.
#
# The script sets up xvfb (X window framebuffer)
# which is used to run the headless Firefox.
#
# We will post a Skype message if the Selenium WebDriver
# has some issues (it often hangs)
#
# We also signal the tests to use a special static Firefox build
# and do not rely (auto-updated) system Firefox
#
# NOTE: This script is NOT sh compliant (echo -n),
# bash needed

# Timeouting commands shell script helper
# http://stackoverflow.com/a/687994/315168
TIMEOUT=timeout.sh

# Which FF binary we use to run the tests
FIXED_FIREFOX=$HOME/ff16/firefox/firefox

# It's me, Mariooo!
SELF=$(readlink -f "$0")

# Skype endpoint information
SKYPE_CHAT_ID="1234567890"

SKYPE_SHARED_SECRET="toholampi"

SEVABOT_SERVER="http://yourserver.com:5000/msg/"

#
#  Helper function to send Skype messages from help scripts.
#  The messages are signed with a shared seceret.
#
#  Parameter 1 is the message
#
function send_skype_message() {
    msg="$1"
    md5=`echo -n "$SKYPE_CHAT_ID$msg$SKYPE_SHARED_SECRET" | md5sum`

    #md5sum pads a '-' to the end of the string. We need to get rid of that.
    for m in $md5; do
        break
    done

    result=`curl --silent --data-urlencode chat="$SKYPE_CHAT_ID" --data-urlencode msg="$msg" --data-urlencode md5="$m" $SEVABOT_SERVER`
    if [ "$result" != "OK" ] ; then
        echo "Error in HTTP communicating to Sevabot: $result"
    fi
}

# Tell the tests to use downgraded FF!6
# which actually works with Selenium
if [ -e $FIXED_FIREFOX ] ; then
    FIREFOX_PATH=$FIREFOX_PATH
    export FIREFOX_PATH
    echo "Using static Firefox 16 build to run the tests"
fi

send_skype_message "♫ ci-loop.sh restarted at $SELF"

while true
do
    # Kill hung testing processes (it might happen)
    pkill -f "bin/test"

    # Purge existing xvfb just in case
    pkill Xvfb

    sleep 5

    echo "Opening virtual X11"

    # Start headless X
    Xvfb :15 -ac -screen 0 1024x768x24 &

    # Tell FF to use this X server
    export DISPLAY=localhost:15.0

    # Run one cycle of continous integration,
    # give it 15 minutes to finish
    echo "Starting test run"
    $TIMEOUT -t 900 continous-integration.sh
    result=$?

    if [ "$result" == "143" ] ; then
        echo "------- CI TIMEOUT OOPS --------"
        send_skype_message "⚡ Continuous integration tests timed out - check the ci-loop.sh screen for problems"
    fi

    sleep 300
done

Voila. Back to the coffee.

 

 

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+