Timeouting commands in shell scripts

Often you want to automatize something using shell scripting. In a perfect world your script robot works for you without getting tired, without hick-ups, and you can just sit at the front of your desk and sip coffee.

Image

Then we enter the real world: Your network is disconnected. DNS goes downs. Your HTTP hooks and downloads stall. Interprocess communication hangs. Effectively this means that even if your script is running correctly from the point of operating system it won’t finish its work before you finish your cup of coffee.

Below is an example how to create timeouts and notifications in a shell script.

1. Never gonna give you up

(c) Rick Astley

Automation must work 100% and you must always know if it doesn’t do that. Otherwise if you cannot trust the automated systems you could have this glorious moment of “oh it has been broken for three months now”. No amount of coffee makes your day after that. Then you spend your nights pondering whether your scripting is running instead of playing Borderlands 2.

Some safety guards around your coffee cup include

  • Sane timeout thresholds for commands to avoid hang situations. E.g. your automated “make smallapp” should probably not run for 50 hours straight.
  • Get a notification if a fully automatized process does not end up as expected. So that you find out the problem right away, not three months later. Do not use email as the communication channel, it is unreliable and has awful signal-to-noise ratio. Better solutions include instant messengers (Skype), SMS (check twilio.com)
  • Get a notification also when a service, which should be running all the time, is restarted.

2. Timeout.bash

The shell scripting (sh, zsh, bash)  does not offer built-in tools for timeouting commands by default (please correct me if I am wrong, but stackoverflow.com et. al folks did not know any better). In Bash cookbook, there exist a timeout wrapper script example (The orignal SO.com answer).

The trick is that when the timeout wrapper terminates the command, you’ll get the exit code of SIGTERM (143) or SIGKILL (137) to the parent script. This might not be entirely clear from reading the script.

3. Using timeout wrapper and sending failure notifications to Skype

Below is an example how you can hook timeout to your own script. Here we have a simple continous integration script (ghetto-ci) which polls version control repositories and on any change executes the test suite. A test failure is reported back to Skype by ghetto-ci using Sevabot Skype bot. The loop script uses a server specific (Ubuntu) way to set up a headless X server, so that the tests can run Firefox on the server (can one call a headless Firefox server “mittens”?) The continous integration loop is deployed simply as leaving it running on the screen‘ed terminal on a server.

Some protection we do: The script has extra checks to protect against hung processes by killing them using pkill before starting each test run. Selenium’s WebDriver seem to often cause situations where the Python tests don’t quit cleanly, leaving around all kind of zombie processes. In this case, the test command timeouts using the timeout wrapper and we get a notification to Skype. Based on this we can refine the test running logic, timeout delay and such to make the script more robust allowing us to focus more on drinking the coffee and less watching a looping process running in a UNIX screen for potential failures.

Pardon me for possibly ugly shell scripting code. SH is not my primary language. The example code is also on Github. Please see the orignal blog post for syntax colored example.

#!/bin/bash
#
# Run CI check for every 5 minutes and
# execute tests if svn has been updated.
#
# The script sets up xvfb (X window framebuffer)
# which is used to run the headless Firefox.
#
# We will post a Skype message if the Selenium WebDriver
# has some issues (it often hangs)
#
# We also signal the tests to use a special static Firefox build
# and do not rely (auto-updated) system Firefox
#
# NOTE: This script is NOT sh compliant (echo -n),
# bash needed

# Timeouting commands shell script helper
# http://stackoverflow.com/a/687994/315168
TIMEOUT=timeout.sh

# Which FF binary we use to run the tests
FIXED_FIREFOX=$HOME/ff16/firefox/firefox

# It's me, Mariooo!
SELF=$(readlink -f "$0")

# Skype endpoint information
SKYPE_CHAT_ID="1234567890"

SKYPE_SHARED_SECRET="toholampi"

SEVABOT_SERVER="http://yourserver.com:5000/msg/"

#
#  Helper function to send Skype messages from help scripts.
#  The messages are signed with a shared seceret.
#
#  Parameter 1 is the message
#
function send_skype_message() {
    msg="$1"
    md5=`echo -n "$SKYPE_CHAT_ID$msg$SKYPE_SHARED_SECRET" | md5sum`

    #md5sum pads a '-' to the end of the string. We need to get rid of that.
    for m in $md5; do
        break
    done

    result=`curl --silent --data-urlencode chat="$SKYPE_CHAT_ID" --data-urlencode msg="$msg" --data-urlencode md5="$m" $SEVABOT_SERVER`
    if [ "$result" != "OK" ] ; then
        echo "Error in HTTP communicating to Sevabot: $result"
    fi
}

# Tell the tests to use downgraded FF!6
# which actually works with Selenium
if [ -e $FIXED_FIREFOX ] ; then
    FIREFOX_PATH=$FIREFOX_PATH
    export FIREFOX_PATH
    echo "Using static Firefox 16 build to run the tests"
fi

send_skype_message "♫ ci-loop.sh restarted at $SELF"

while true
do
    # Kill hung testing processes (it might happen)
    pkill -f "bin/test"

    # Purge existing xvfb just in case
    pkill Xvfb

    sleep 5

    echo "Opening virtual X11"

    # Start headless X
    Xvfb :15 -ac -screen 0 1024x768x24 &

    # Tell FF to use this X server
    export DISPLAY=localhost:15.0

    # Run one cycle of continous integration,
    # give it 15 minutes to finish
    echo "Starting test run"
    $TIMEOUT -t 900 continous-integration.sh
    result=$?

    if [ "$result" == "143" ] ; then
        echo "------- CI TIMEOUT OOPS --------"
        send_skype_message "⚡ Continuous integration tests timed out - check the ci-loop.sh screen for problems"
    fi

    sleep 300
done

Voila. Back to the coffee.

 

 

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Recommended way for sudo-free installation of Python software with virtualenv

When installing Python software, sudo easy_install and sudo pip are something you should do very seldom. sudo means you are messing with your operating system files. easy_install means that it is easy to install, but impossible to uninstall. You most likely step on the toes your operating system package manager and make your Python installation damaged – damaged in a way you cannot reliably use OS package manager to install or upgrade the software in the future. There is also a high chance that two software depends on the different version of a library X and easy_install will happily overwrite anything with different version in your system.

Even though being badly broken, still plenty of Python software documentation recommends sudo easy_install as an installation method.

virtualenv is a Python tool for creating an isolated Python environment for a normal user. They are isolated in a sense, that only the software you install there will see and mess with the environment. When you are running virtualenv’ed python, pip and easy_install can freely pull in any libraries from pypi without the worry that you break some other software on your computer.

Here is a recipe for installing virtualenv’ed Python software which should across UNIXes and Windows. It will totally run as your normal user privileges, no sudo or admin needed. Only an installed Python interpreter and a working python command are the requirements.

So fire up your console, go the folder where you wish to perform the installation and type the following (some commenting added to explain the process):

# First create the folder where you wish to install the software
# with mkdir command. Then go to this folder with cd command.

# Windows users: Please manually download virtualenv.py
# to the target folder  ith your web browser. UNIX users
# can use curl command line downloader as below.

# Note: Don't rely on operating system virtualenv command.
# It might be hassle to instruct virtualenv package installation
# due to distribution flavours.
# Old Ubuntus ship really old virtualenv.py and it has not worked
# on all cases.
# Github virtualenv.py is the msot reliable method.
curl -L -o virtualenv.py https://raw.github.com/pypa/virtualenv/master/virtualenv.py

# Create a virtualenv environment
# where the software and its dependencies
# will be pulled from PyPi. In our case
# we call the created virtualenv folder "venv"
python virtualenv.py venv

# Activate the virtualenv environment.
# This will set your PATH environment
# variable so that following "python"
# command executes from under the virtualenv,
# not from your global system setup.
#

# Windows equivalent: .\venv\Scripts\activate
. venv/bin/activate

# Now when virtualenv is activated,
# pip and easy_install will install any software
# under this virtualenv environment, not on your operating system files
pip install YOUR_PACKAGE_NAME_ON_PYPI.PYTHON.ORG

# Usually, if you install Python command line software,
# new launcher scripts get created in venv/bin
# folder. When venv environment is active,
# this folder takes precedence in PATH environment
# variable. Meaning, when you have virtualenv activated
# you can simply type in the installed command name
# without full path to execute it.

And again with a real life example:

curl -L -o virtualenv.py https://raw.github.com/pypa/virtualenv/master/virtualenv.py
python virtualenv.py vvv-venv
. vvv-venv/bin/activate
pip install vvv

I have tested this recipe with vvv and Skype sevabot and have found it working. However, I wish to get some feedback and ideas how this could be further enhanced, so please send in your ideas.

Some notes

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Inserting leads to Zoho CRM using PHP

Here is a simple PHP class (one  file) which you can use to insert leads to Zoho CRM. It is designed to be hooked up with your site contact form. Instead of emailing the contact form results, they go directly to the CRM and are available for the sales person to call. The code based on the orignal Python based mfabrik.zoho package. PHP curl needed as a requirement.

<?php
/**
 * Simple Zoho CRM inserter.
 *
 * MIT licensed. Copyright 2011 Pete Sevander and Mikko Ohtamaa.
 *
 */

class ZohoException extends Exception { }

class Zoho {

    public function __construct($username, $password, $apikey, $extra_auth_params = array(), $auth_url="https://accounts.zoho.com/login") {
        $this->username = $username;
        $this->password = $password;
        $this->apikey = $apikey;

        $this->ticket = null;
    }

    public function open() {
        $this->ticket = $this->_createTicket();
    }

    public function _createTicket() {

        $params = array(
            "servicename" => "ZohoCRM",
            "FROM_AGENT" => "true",
            "LOGIN_ID" => $this->username,
            "PASSWORD" => $this->password
        );

        //$params = array_map('urlencode', $params);

        $url = "https://accounts.zoho.com/login";

        $body = openUrl($url, $params);

        $data = $this->_parse_ticket_response($body);
        $this->data = $data;

        if (isset($data["WARNING"]) || isset($data['CAUSE'])) {
            $warning = (isset($data["WARNING"])) ? $data["WARNING"] : $data["CAUSE"];
            if ($warning != "null") {
                throw new ZohoException("Could not auth: " . $warning);
            }
        }
        if ($data["RESULT"] != "TRUE") {
            throw new ZohoException("Ticket result was not valid");
        }

        return $data["TICKET"];
    }

    public function _parse_ticket_response($data) {

        $output = array();

        $lines = explode("\n", $data);

        foreach($lines as $line) {
            if (substr($line, 0,1) == "#") {
                continue;
            }
            if ($line == "") {
                continue;
            }
            if (!strstr($line, "=")) {
                continue;
            }
            $line = explode("=", $line);
            $output[$line[0]] = $line[1];
        }

        return $output;
    }

    public function ensure_opened() {
        if ($this->ticket == null) {
            throw new ZohoException("Login first");
        }
    }

    /**
    * https://crm.zoho.com/crm/private/xml/Leads/insertRecords?newFormat=1&apikey=APIkey&ticket=Ticket
    **/
    public function insertRecords($leads, $extra_post_parameters=array()) {
        $this->ensure_opened();

        $xmldata = $this->XMLfy($leads);

        $post = array(
            'newFormat' => 1,
            'ticket' => $this->ticket,
            'apikey' => $this->apikey,
            'version' => 2,
            'xmlData' => $xmldata,
            'duplicateCheck' => 2,
            'wfTrigger' => 'true'
        );

        array_merge($post, $extra_post_parameters);

        // We'll bump created time to make sure that duplicate data entry
        // gets bumped up on the salesdroids list

        // $created = strftime('%Y-%m-%d %H:%M');
        // $post['Created Time'] = $created;

        // XXX: Good idea but Zoho silently ignores changes to the creation time

        $q = http_build_query($post);

        //print_r($post);

        $response = openUrl("https://crm.zoho.com/crm/private/xml/Leads/insertRecords", $q);

        //print_r($response);
        //print_r($xmldata);
        $this->check_successful_xml($response);

        return true;

    }

    public function getRecords($columns ='leads(Name)') {
        $this->ensure_opened();

        $post = array(
            'newFormat' => 1,
            'ticket' => $this->ticket,
            'apikey' => $this->apikey,
            'version' => 2,
            'selectColumns' => $columns,
        );

        $q = http_build_query($post);
        $response = openUrl("https://crm.zoho.com/crm/private/json/Leads/getRecords", $q );

        echo $response;

    }

    public function check_successful_xml($response) {
        $html = new DOMDocument();
        $html->loadXML($response);

        if ($err = $html->getElementsByTagName('error')->item(0)) {
            throw new ZohoException($err->getElementsByTagName('message')->item(0)->nodeValue);
        }

        return true;
    }

    public function XMLfy ($arr) {
        $xml = "<Leads>";
        $no = 1;
        foreach ($arr as $a) {
            $xml .= "<row no=\"$no\">";
            foreach ($a as $key => $val) {
                $xml .= "<FL val=\"$key\">$val</FL>";
            }
            $xml .= "</row>";
            $no += 1;
        }
        $xml .= "</Leads>";
        return $xml;
    }
}

function openUrl($url, $data=null) {
    $ch = curl_init();
    $timeout = 5;

    if($data) {
        curl_setopt($ch,CURLOPT_POST,1);
        curl_setopt($ch,CURLOPT_POSTFIELDS, $data);
        curl_setopt($ch,CURLOPT_VERBOSE, true);
        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);

       // log output
       //$f = fopen("/tmp/zoho-curl.txt", "wt");
       //curl_setopt($ch,CURLOPT_STDERR, $f);

   }

    curl_setopt($ch,CURLOPT_URL,$url);
    curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
    curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
    $data = curl_exec($ch);
    curl_close($ch);
    return $data;
}

?>

And this is how you use it:

<?php

    $ZOHO_USER="yyyyy@xxxxxx.com";
    $ZOHO_PASSWORD="v3ryS3cr3t";
    $ZOHO_API_KEY='GET THIS FROM ZOHO CRM SETTINGS';

    require_once 'Zoho.php';

    $z = new Zoho($ZOHO_USER, $ZOHO_PASSWORD, $ZOHO_API_KEY);

    try {
        $z->open();

        $leads = array(
            'First Name' => 'Mikko',
            'Last Name' => 'Ohtamaa',
            'Company' => 'opensourcehacker.com',
            'Phone' => '+358 12 123 1234',
            'Email' => 'mikko @ foobar dot com',
            'Lead Owner' => 'yyyyy@xxxxxx.com',
        );

        try {
            $z->insertRecords(array($leads));
            $renderForm = false;
            echo '<h3>Contact information sent successfully. We will contact you soon.</h3><br /><strong>Data sent:</strong><dl>';
            foreach ($values as $key => $value) {
                echo '<dt><strong>' . $form->getElement($key)->getLabel() . '</strong></dt>';
                echo '<dd>' . $value . '</dd>';
            }
            echo '</dl>';

        } catch (ZohoException $e) {
            echo '<span>Error inserting data: ' . $e->getMessage() . '</span>';
        } 

    } catch (ZohoException $e) {
        echo '<span>Can\'t connect to Zoho: ' . $e->getMessage() . '</span>';
    }

?>

Some things to note

  • Owner must be given and be valid Zoho user
  • Company and Last Name fields are the only required fields by default
  • Zoho CRM checks duplicates using email field and thus using the same email address for (test) inserts won’t yield to visible results in My Leads view when duplicateCheck=2
  • Creation Time field cannot be changed
  • You can customize fields in Zoho CRM settings

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+