Showing posts with label ruby. Show all posts
Showing posts with label ruby. Show all posts

December 3, 2021

Day 3 - Keeping Config Management Simple with Itamae

By: Paul Welch (@pwelch)
Edited by: Jennifer Davis (@sigje)

Our DevOps toolbox is filled with many tools with Configuration Management being an often neglected and overloaded workhorse. While many resources today are deployed with containers, you still use configuration management tools to manage the underlying servers. Whether you use an image-based approach and configure your systems with Packer or prefer configuring your systems manually after creation by something like Terraform, chances are you still want to continuously manage your hosts with infrastructure as code. To add to the list of potential tools to solve this, I’d like to introduce you to Itamae. Itamae is a simple tool that helps you manage your hosts with a straight-forward DSL while also giving you access to the Ruby ecosystem. Inspired by Chef, Itamae has a similar DSL but does not require a server, complex attributes, or data bags.

Managing Resources

Itamae is designed to be lightweight; it comes with an essential set of resource types to bring your hosts to the expected state. These resource types focus on the core parts of our host we want to manage like packages, templates, and services. The bundled `execute` resource can be used as an escape hatch to manage resources that might not have a builtin resource type. If you find yourself wanting to manage something often that does not have a built in resource, you can build your own resources if you are comfortable with Ruby.

All Itamae resource types have common attributes that include: actions, guards, and triggers for other resources.

Actions

Actions are the activities that you want to have occur with the resource. Each bundled resource has predefined actions that can be taken. A `service` resource, for example, can have both an `:enable` and `:start` action which tells Itamae to enable the service to start on system boot and also start the service if it is not currently running.


    # enable and start the fail2ban service
    service “fail2ban” do
      action [:enable, :start]
    end
    

Guards

Guards ensure a resource is idempotent by only invoking the interpreted code if the conditions pass. The common attributes that are available to use within your infracode are `only_if` and `not_if`.


    # create an empty file only if it does not exist
    execute "create an empty file" do
      command "touch /tmp/file.txt"
      not_if "test -e /tmp/file.txt"
    end
    

Triggers

Triggers allow you to define event driven notifications to other resources.

The `notifies` and `subscribes` attributes allow you to trigger other resources only if there is a change such as restarting a service when a new template is rendered. These are synonymous with Chef & Puppet’s `notifies` and `subscribes` or Ansible’s `handlers`.


    # define nginx service
    service 'nginx' do
      action [:enable, :start]
    end
    
    # render template and restart nginx if there are changes
    template "/etc/nginx/sites-available/main" do
      source "templates/etc/nginx/sites-available/main.erb"
      mode   "0644"
      action :create
      notifies :restart, "service[nginx]", :delayed
    end

Itamae code is normally organized in “cookbooks” much like Chef. You can include recipes to separate your code. Itamae also supports definitions to help DRY your code for resources.

Example

Now that we have an initial overview of the Itamae basics, let’s build a basic Nginx configuration for a host. This example will install Nginx from a PPA on Ubuntu and render a basic configuration that will return the requestor’s IP address. The cookbook resources will be organized as follows:


    ├── default.rb
    └── templates
        └── etc
              └── nginx
                └── sites-available
                   └── main.erb

We will keep it simple with a single `default.rb` recipe and single `main.erb` Nginx site configuration template. The recipe and site configuration template content can be found below.


    # default.rb
    # Add Nginx PPA
    execute "add-apt-repository-ppa-nginx-stable" do
      command "add-apt-repository ppa:nginx/stable --yes"
      not_if "test -e /usr/sbin/nginx"
    end
    
    # Update apt cache
    execute "update-apt-cache" do
      command "apt-get update"
    end
    
    # install nginx stable
    package "nginx" do
      action :install
    end
    
    # enable nginx service
    service 'nginx' do
      action [:enable, :start]
    end
    
    # configure nginx
    template "/etc/nginx/sites-available/main" do
      source "templates/etc/nginx/sites-available/main.erb"
      mode   "0644"
      action :create
      notifies :restart, "service[nginx]", :delayed
      variables()
    end
    
    # enable example site
    link '/etc/nginx/sites-enabled/main'  do
      to "/etc/nginx/sites-available/main"
      notifies :restart, "service[nginx]", :delayed
      not_if "test -e /etc/nginx/sites-enabled/main"
    end
    
    # disable default site
    execute "disable-nginx-default-site" do
      command "rm /etc/nginx/sites-enabled/default"
      notifies :restart, "service[nginx]", :delayed
      only_if "test -e /etc/nginx/sites-enabled/default"
    end

    # main.conf
server {
  listen 80 default_server;
  listen [::]:80 default_server;

  server_name _;

  location / {
    # Return the requestor's IP as plain text
    default_type text/html;
    return 200 $remote_addr;
  }
}

Deploying

*To deploy the above example, it is assumed that you have a temporary VPS instance available.

There are 3 different ways you can deploy your configurations with Itamae:

  • `itamae ssh` via the itamae gem.
  • `itamae local` also via the itamae gem.
  • `mitamae` locally on the host.

Mitamae is an alternative implementation of Itamae built with mruby. This post is focusing on Itamae in general but the Mitamae implementation is a notable option if you want to deploy your configuration using prebuilt binaries instead of using SSH or requiring Ruby.

With your configuration ready it’s just a single command to deploy over SSH. Itamae uses the SpecInfra library which is the same library that ServerSpec uses to test hosts. You can also access a host’s inventory in Itamae much like you can with Chef & Ohai. To deploy your configuration, run:


    itamae ssh --key=/path/to/ssh_key --host=<IP> --user=<USER> default.rb
    --log-level=DEBUG

Itamae will manage those packages and write out the template we specified, bringing the host to our desired state. Once the command is complete, you should be able to curl the host’s IP address and receive a response from Nginx.

Wrapping Up

Thank you for joining me in learning about this lightweight configuration management tool. Itamae gives you a set of bundled resource types to quickly configure your infrastructure in a repeatable and automated manner with three ways to deploy. Check out the Itamae Wiki for more information and best practices!

December 16, 2013

Day 16 - omnibus'ing your way to happiness

Written by: John Vincent (@lusis)
Edited by: Ben Cotton (@funnelfiasco)

We've all been there.

You find this awesome Python library or some cool new utility that you want to run.

You check your distribution's package repository. It's not there or worse it's an ancient version.

You check EPEL or a PPA for it. Again, it's either not available or it's an ancient version. Oh and it comes broken out into 20 sub-packages so you can't just grab the package and install it. And you have to add a third-party repository which may have other stuff you don't want pulled in.

You try to compile it yourself and realize that your version of OpenSSL isn't supported or you need a newer version of Python.

You Google furiously to see if someone has posted a spec file or some blog entry on building it. None of it works. Four hours later you still don't have it installed. You look all around and see nothing but yak hair on the ground. Oh and there's a yak with a full coat grinning at you.

My friend have I got a deal for you. Omnibus!

What is omnibus?

Omnibus is a toolchain of sorts. It was created by Chef (formerly Opscode) as a way to provide a full install of everything needed to run their products in a single native system package. While you may not have a need to build Chef, Omnibus is flexible enough to build pretty much anything you can throw at it.

Omnibus can be confusing to get started with so consider this your guidebook to getting started and hopefully having more time to do the fun things in your job. Omnibus works by leveraging two tools you may already be using - Vagrant and FPM. While it's written in Ruby, you don't have to have Ruby installed (unless you're creating your own software to bundle) and you don't even have to have FPM installed. All you need is Vagrant and two vagrant plugins.

The omnibus workflow

The general flow of an omnibus build is as follows:
  • Check out an omnibus project (or create your own)
  • Run vagrant up or vagrant up <basebox name>
  • Go get coffee
  • Come back to a shiny new native system package of whatever it was omnibus was building
Under the covers while you're drinking your coffee, omnibus is going through a lot of machinations to get you that package (all inside the vagrant vm):
  • Installing Chef
  • Checking out a few chef cookbooks to run for prepping the system as a build host
  • Building ruby with with chef-solo and the rbenv cookbook.
  • installing omnibus and fpm via bundler
  • Using the newly built ruby to run the omnibus tool itself against your project.
From here Omnibus is compiling everything above libc on the distro it's running under from source and installing it into /opt/<project-name>/. This includes basics such as OpenSSL, zlib, libxml2, pcre and pretty much everything else you might need for your final package. Every step of the build process sets your LDFLAGS, CFLAGS and everything else to point to the project directory in /opt.

After everything is compiled and it thinks it's done, it runs a sanity check (by running ldd against everything in that directory) to ensure that nothing has linked against anything on the system itself.
If that check passes, it calls FPM to package up that directory under /opt into the actual package and drops it off in a subdirectory of your vagrant shared folder.

The added benefit here is that the contents of the libraries included in the system package are almost exactly the same across every distro omnibus builds your project against. Gone are the days of having to worry about what version of Python is installed on your distro. It will be the same one everywhere.

A sample project

For the purposes of this post, we're going to work with a use case that I would consider fairly common - a Python application.

You can check out the repo used for this here: https://github.com/lusis/sample-python-omnibus-app

As I mentioned previously, the only thing you need installed is Vagrant and the following two plugins:
  • vagrant-omnibus
  • vagrant-berkshelf
If you have that done, you can simply change into the project directory and do a vagrant up. I would advise against that, however, as this project is configured to build packages for Ubuntu-10.04 through Ubuntu-12.04 as well as CentOS 5 and CentOS 6. Instead, I would run it against just a specific distribution such with vagrant up ubuntu-12.04.

Note that on my desktop (Quad-core i7/32GB of memory) this build took 13 minutes

While it's building, you should also check out the README in the project.

Let's also look at a few key files here.

The project configuration

The project definition file is the container that describes the software you're building. The project file is always located in the config/projects directory and is always named the same as the package you're building. This is important as Omnibus is pretty strict about names used aligning across the board.

Lets look at the definition for this project in config/projects/sample-python-app.rb. Here is an annotated version of that file:
annotated project def

The things you will largely be concerned with are the block of dependencies. Each of these corresponds to a file in one of two places (as noted):
  • a ruby file in <project root>/config/software
  • a file in the official omnibus-software repository on github (https://github.com/opscode/omnibus-software)
This dependency resolution issue is important and we'll address it below.

A software definition

The software definition has a structure similar to the project definition. Not every dependency you have listed needs to live in your repository but if it is not there, it will be resolved from the master branch of the opscode repository as mentioned.

This can obviously affect the determinism of your build. It's a best practice to copy any dependencies explicitly into your repository to ensure that Chef doesn't introduce a breaking change upstream. This is as simple as copying the appropriate file from the official repository into your software directory.

We're going to gloss over that for this article since we're focusing on writing our own software definitions. If you look at the dependencies we've defined, you'll see a few towards the end that are "custom". Let's focus on pyopenssl first as that's one that is always a pain from distro to distro:

annotated software def

The reason I chose pyopenssl was not only because it's a common need but because of it's sensitivity to the version of OpenSSL it builds against.

This shows the real value of Omnibus. You are not forced to only use a specific version of pyopenssl that matches your distro's OpenSSL library. You can use the same version on ALL distros because they all link against the same version of OpenSSL which Omnibus has kindly built for you.
This also shows how you can take something that has an external library dependency and ensure that it links against the version in your package.

Let's look at another software definition - config/software/virtualenv.rb
Note that this is nothing more than a standard pip install with some custom options passed. We're ensuring we're calling the right version of pip by using the one in our package directory - install_dir matches up with the install_path value in config/projects/sample-python-app.rb which is /opt/sample-python-app.

Some other things to note:
  • the embedded directory This is a "standard" best practice in omnibus. The idea is that all the bits of your package are installed into /opt/package-name/embedded. This directory contains the normal [bin|sbin|lib|include] directory structure you're familiar with. The intent of this is to signal to end-users that the stuff under embedded is internal and not something you should ever need to touch.
  • passing --install-option="--install-scripts=#{install_dir}/bin to the python package This ensures that pip will install the library binaries into /opt/package-name/bin. This is the public-facing side of your package if you will. The bits/binaries that users actually need to call should either be installed in a top-level hierarchy under /opt/package-name or symlinked from a file in the embedded directory to the top-level directory. You'll see a bit of this in the post-install file your package will call below.

postinstall/postrm

The final directory we want to look at is <project-root>/package-scripts/sample-python-app. These contain files that are passed to fpm as postinstall and postremove scripts for the package manager.
annotated postinstall

The biggest thing to note here is the chown. This is thing that bites folks the most. Since fpm simply creates tar files of directories, those files will always be owned by the user that runs fpm. With Omnibus, that's the vagrant user. What ends up happening is that your package will install but the files will all be owned by whatever uid/gid matches the one used to package the files. This isn't what you want. In this case we simply do a quick chown post-install to fix that.

As I mentioned above, we're also symlinking some of the embedded files into the top-level to signal to the user they're intended for use.

Installing the final artifact

At this point your build should be done and you should now have a pkg directory under your project root. Note that since this is using vagrant shared folders, you still haven't ever logged on to the actual system where the package is built. You don't even need to scp the file over.

If you want to "test" your package you can do the following: - vagrant destroy -f ubuntu-12.04 - vagrant up ubuntu-12.04 --no-provision - vagrant ssh ubuntu-12.04 -c 'PATH=/opt/sample-python-app/bin:\$PATH virtualenv --version'

Next steps

Obviously there's not always going to be an omnibus project waiting for everything you might need. The whole point of this is to create system packages of things that you need - maybe even your company's own application codebase.

If you want to make your own omnibus project, you'll need Ruby and the omnibus gem installed. This is only to create the skeleton project. Once you have those installed somewhere/anywhere, just run:
omnibus project <project-name>

This will create a skeleton project called omnibus-<project-name>. As I mentioned earlier, naming is important. The name you pass to the project command will define the package name and all values in the config/projects/ directory.

You'll likely want to customize the Vagrantfile a bit and you'll probably need to write some of your own software definitions. You'll have a few examples created for you but they're only marginally helpful. Your best bet is to remove them and learn from examples online in other repos. Also don't forget that there's a plethora of predefined software in the Opscode omnibus-software repository.
Using the above walk-through, you should be able to easily navigate any omnibus project you come across and leverage it to help you write your own project.

If I could give one word of advice here about the generated skeleton - ignore the generated README file. It's confusing and doesn't provide much information about how you should use omnibus. The vagrant process I've described above is the way to go. This is mentioned in the README but it's not the focus of it.

Going beyond vagrant

Obviously, this workflow is great for your internal development flow. There might come a time when you want to make this part of the official release process as part of a build step in something like Jenkins. You can do that as well but you'll probably not be using vagrant up for that.
You have a few options:
  • Duplicate the steps performed by the Vagrantfile on your build slaves:
  • install chef from opscode packages
  • use the omnibus cookbook to prep the system
  • run bundle install
  • run omnibus build
Obviously you can also prep some of this up front in your build slaves using prebuilt images - Use omnibus-omnibus-omnibus This will build you a system package with steps 1-3 above taken care of. You can then just clone your project and run /opt/omnibus/bin/omnibus build against it.

Getting help

There's not an official community around omnibus as of yet. The best place to get help right now is twitter (feel free to /cc @lusis), the #chef channel on Freenode IRC and the chef-users mailing list.
Here are a few links and sample projects that can hopefully also help:

December 11, 2010

Day 11 - A Journey to NoSQL

Written by Michael Stahnke (@stahnma)

The N00b

When I was first learning about being a Unix Admin, I just wanted to know what systems my team supported, so that when I got called at 2 AM, I could either make some weak attempt at getting online and fixing a problem (I was new...very new), or promptly help that application analyst find the correct support team pager number. It was the week before I first went into our pager rotation that I realized something was very wrong. I had no idea what systems we actually supported. I wasn't the only one.

There had recently been some form of reorganization right before I hired in at this company. What was once four teams (IBM AIX, HP-UX, Sun Solaris and Red Hat Linux), was becoming three teams (Capacity Planning, Systems Implementation and Systems operations). However, there were still other server teams at other sites, plus Unix workstation support, and some IRIX somewhere out there. The fundamental problem, though, was, "Do I have the ability to help the person who has paged my team?"

A solution...sort of

I found this state to be extremely non-desired, so I started writing a Unix server tracking system. It started out as a basic web application utilizing a MySQL back-end. It worked great. The teams loved it. They knew what we supported and what we didn't. Then, the requests for enhancement came in. I needed to add MAC addresses, world wide port names, cluster licensing terms, customer information, out-of-band management URLs, etc. This quickly grew, but I was still happy with it. We designed several workflow automations through the tool as well. However, as the tool grew larger, and less maintainable, I was starting to get extremely frustrated with it.

While problems for this application were abundant, there were two issues that made it less of an operational platform than I desired. The first problem was that in order to do any type of CRUD actions, you have to have database drivers on the client. This was a big challenge. We had an extremely heterogenous environment, multiple firewalls, and some ancient operating systems that probably couldn't have had a MySQL driver loaded on them without sacrificing some type of domesticated animal and praying to a deity that was anything but righteous.

The other problem was flexibility of schema. Each time we added a new piece of data to track, it had to be analyzed, and then added into the schema. Normalization was great for one:many and many:many relationships, but then made the SQL queries much more complex with joins or sub-queries, especially for unix admins without much or any SQL background. In short, the relational portion of the RDBMS system was in the way.

Another solution...getting warmer

I left that shop before that problem was really solved, but since I had an opportunity again at my next assignment to solve a similar problem, I decided I would try some things in a different way. My first thoughts were around putting some form of web-services infrastructure in front of a basic RDBMS backed web application. I thought that speaking HTTP would be easier than MySQL, Oracle or even DBI for most clients. I toyed with it and did some mock-ups, but I still felt like the data model was complicated and required many calls and client-side parsing to really get the data into usable formats for automation, updates, or to generate Nagios configuration, etc. It was time for something completely different.

NoSQL. It was obvious. Of course, at this time (2006) I had never heard of the term NoSQL, but looking back on it, that was the epiphany I had. If relationships are difficult to model and manage, maybe some other model would work. Then it hit me: LDAP. The LDAP container is designed for easy replication, extremely granular security controls, and availability. On top of that, those features were all there out of the box. Schemas could be programatically deployed, and many of the data model questions were things like 'should this be single-valued or multi-valued'. Those questions were quite simple when compared to joining 17 tables to see a complete system configuration in the old RDBMS I had authored. As an added bonus, using LDAP didn't introduce a new source of truth for the environment since it was in use for account management.

LDAP also had a good solution for the driver problem. We were using LDAP for user authentication, so our systems already had LDAP client libraries loaded. Even the few that didn't, the client-side libraries were readily available, even on my less-than-favorite flavors of Unix.

We modified schema, populated data by hand, and then with some simple scripts. Life was good...at least for a while. After a couple years operating in this mode, the schema became a bit more problematic. Extending schema at will was not the greatest idea I've ever had. We also had a problem where some admins would make new objectClasses rather than extend one, or inherit from one. This led to conflicts in schema and some data integrity issues. None of it was absolutely horrible, but in the end it smelled like a chilli dog left in a desk drawer overnight.

The search continues

I had a lot of discussion about this problem with a group of my friends (and eventual business partners). We spent hours going back and forth on how to model host information and metadata and expose that information to our configuration management, monitoring, accounting, chargeback, and provisioning systems. It always came back to a discussion on discrete math: use Set Theory. The best, and possibly only sane way, to keep this data organized was to use set theory.

Luckily, we had a greenfield to play with as we forming a new company. We tried it out. We tried to not extend or customize schema for host information beyond loading in well-known IANA referenced schemas. The basic premise, obviously, is that everything can be grouped into sets. We created an OU=Sets at the top level our LDAP directory. Under OU=Sets, we created a DN of of 'set name' for example dn: cn=centos5,ou=Sets,dc=websages,dc=com is an entry in our directory. It is setup as a groupOfUniqueNames and contains the DN of each host that is in fact a CentOS 5 host. The nice thing about OU=Sets is you can just keep adding things into it, without extending schema.

It may seem a bit backward at first to have the attribute as the set name and then the host dn as the entry, but it seems to work. LDAP also allows groups within groups, so nesting works perfectly. As an example, if cn=ldap_servers,ou=sets exists, it may contain cn=ldap_write_servers,ou=sets and cn=ldap_replicas,ou=sets. Grouping in this manner allows one change to cascade through the directory.

Of course, with every good solution, there are more problems to be solved. In this case it's recursion. OpenLDAP and 389/RHDS/Fedora-DS/SunOne/iPlanet/et all don't seem to automatically recurse nested groups, though I have heard that some LDAP implementations do. Luckily, it's not that big of a problem.

Recursion

In this example, I'll be looking for all LDAP servers. Our directory information tree is setup such that we have three groups:

  • ldap_write_servers
  • ldap_replicas
  • ldap_servers

The ldap_servers entry is a groupOfUniqueNames whose uniqueMembers are the other two groups. To traverse this, we'll need some recursion.

Sample Code

In my code, I most often use ruby. When working with LDAP, I've used the classic ldap bindings heavily, but recently I've really taken a liking to activeldap. Activeldap borrows heavily from the Active Record design pattern and applies it to LDAP. It is not a perfect translation of active record, but it is quite nice for most operations on a directory server.

Activeldap requires some minimal setup to be useful. You can install it with gems or your favorite package manager.

require 'rubygems'
require 'active_ldap'

class Entry < ActiveLdap::Base
end

ActiveLdap::Base.setup_connection(
  :host => 'ldap.websages.com', :port => 636, :method => :ssl,
  :base => 'dc=websages,dc=com',
  :bind_dn => "uid=stahnma,ou=people,dc=websages,dc=com",
  :password => ENV['LDAP_PASSWORD'], :allow_anonymous => false)

This is a simple setup section for some code using activeldap. Require the library (and rubygems unless your environment will load them, or you installed activeldap in some other method). Then you run setup_connection. The Websages directory server requires SSL and does not allow anonymous bind, so a few more parameters are used than you might see on a clear-text, anonymous setup.

From there, it's really not very difficult to recurse through groups and find the entries.

# Returns the members of a ldap groupOfUniqueNames
def find_members(search, members = [])
  Entry.find(:all , search).each do |ent|
    # Ensure the search result is a group
    if ent.classes.include?('groupOfUniqueNames')
       # Check to see if each member is a group
       ent.uniqueMember.each do |dn|
         members << find_members(dn, members)
       end
    else
    # Add the results to the members array
     members <<  search
    end
  end
  # clean up the array before returning
  members.flatten.uniq
end

The above code will find all members of a groupOfUniqueNames including entries of groups within groups.

My calling function is just:

puts find_members('cn=ldap_servers')

Another excellent feature of activeldap is that if you simple puts an activeldap object, the LIDF text for the object is displayed on standard out.

Entry.find(:all , "cn=ldap_servers").each do |h|
  puts h
end

Produces a simple LDIF output:

version: 1
dn: cn=ldap_servers,ou=Sets,dc=websages,dc=com
cn: ldap_servers
description: Hosts acting as LDAP Servers
objectClass: groupOfUniqueNames
objectClass: top
uniqueMember: cn=ldap_replicas,ou=sets,dc=websages,dc=com
uniqueMember: cn=ldap_write_servers,ou=sets,dc=websages,dc=com

LDAP is a good answer

Now I can basically apply set theory for system management of meta data and configuration information. At Websages, we use our LDAP directory for nearly everything and integrate it into our fact generation for puppet, our backup schedules, our controlling IRC bots, and our broadcast SMSing while acting like idiots at the bar.

So next time you're faced with storing a bunch of host information or meta-data, you might turn back to a technology that is non-relational, scales horizontally, offers extensive ACL options, and is lightweight and fast. LDAP was NoSQL before the term was coined and often loses out on today's NoSQL discussions, but it's track record is proven.

When I see the term NoSQL, I am reminded of a classic Dilbert, "I assure you, it has a totally different name."

Dilbert

December 10, 2008

Day 10 - Config Generation

A few days ago we covered using a yaml file to label machines based on desired configuration. Sometimes part of this desired configuration includes using a config file that needs modification based on attributes of the machine it is running on: labels, hostname, ip, etc.

Using the same idea presented in Day 7, what can we do about generating configuration files? Your 'mysql-slave' label could cause your my.cnf (mysql's config file) to include settings that enable slaving off of a master mysql server. You could also use this machine:labels mapping to automatically generate monitoring configurations for whatever tool you use; nagios, cacti, etc.

The older ways of doing config generation included using tools like sed, m4, and others, to modify a base configuration file inline or writing a script that had lots of print statements to generate your config. These are both bad with respect to present-day technology: templating systems. Most (all?) major language have templating systems: ruby, python, perl, C, etc. I'll limit today's coverage, for the sake of providing an example, to ruby and ERB.

ERB is a ruby templating tool that supports conditionals, in-line code, in-line variable expansion, and other things you'll find in other systems. It gets bonus points because it comes standard with ruby installations. That one bonus means that most people (using ruby) will use ERB as their templating tool (Ruby on Rails does, for example), and this manifests itself in the form of good documentation and examples.

Let's generate a sample nagios config using ruby, ERB and yaml. Before that, we'll need another yaml file to describe what checks are run for each label. After all, the 'frontend' label might include checks for process status, page fetch tests, etc, and we don't want a single 'check-frontend' check since mashing all checks into a single script can mask problems.

You can view the hostlabels.yaml and lablechecks.yaml to get an idea of the simple formatting. Using this data we can see that 'host2.prod.yourdomain' has the 'frontend' label and should be monitored using the 'check-apache' and 'check-frontend-page-test' checks.

The ruby code and ERB are about 70 lines total, perhaps too much to write here, so here are the files:

Running 'configgen.rb' with all the files above in the same directory produces this output. Here's a small piece of it:
define hostgroup {
  hostgroup_name frontend
  members host2.prod.yourdomain
}

define service {
  hostgroup_name frontend
  service_description frontend.check-http-getroot
  check_command check-http-getroot
}

define service {
  hostgroup_name frontend
  service_description frontend.check-https-certificate-age
  check_command check-https-certificate-age
}

define service {
  hostgroup_name frontend
  service_description frontend.check-https-getroot
  check_command check-https-getroot
}
I'm not totally certain this generates valid nagios configurations, but I did my best to make it close.

If you add a new 'frontend' server to hostlabels.yaml, you can regenerate the nagios config trivially and see that the 'frontend' hostgroup now contains a new host:

define hostgroup {
  hostgroup_name frontend
  members host3.prod.yourdomain, host2.prod.yourdomain
}
(There's also a new host {} block declaring the new host3.prod.yourdomain not shown in this post)

Automatically generating config files moves you into a whole new world of sysadmin zen. You can regenerate any configuration file if it is corrupt or lost. No domain knowledge is required to add a new host or label. Knowing the nagios (or other tools) config language is only required when modifying the config template, not the label or host definitions (a time/mistake saver). You could swap nagios out for another monitoring tool and still make sure the underlying concepts (frontend has http monitoring, etc) are consistent. Being able to automatically generate configs means that you probably have both the templates and the source data (our yaml files here) stored in revision control, which is a whole other best practice to focus on.

Further reading:

December 5, 2008

Day 5 - Capistrano

Do you store and deploy configuration files from a revision control system? You should. If you don't, yet, this article aims to show you how to make that happen with very little effort using Capistrano.

Capistrano is a ruby-powered tool that acts like make (or rake, or any build tool), but it is designed with deploying data and running commands on remote machines. You can write tasks (like make targets) and even nest them in namespaces. Hosts can be grouped together in roles and you can have a task affect any number of hosts and/or roles. Capistrano, like Rake, uses only Ruby for configuration. Capistrano files are named 'Capfile'.

Much of the documentation and buzz about Capistrano deals with deployment of Ruby on Rails, but it's not at all limited to Rails.

For a simple example, lets ask a few servers what kernel version they are running:

# in 'Capfile'
role :linux, "jls", "mywebserver"

namespace :query do
  task :kernelversion, :roles => "linux" do
    run "uname -r"
  end
end
Output:
% cap query:kernelversion
  * executing `query:kernelversion'
  * executing "uname -r"
    servers: ["jls", "mywebserver"]
    [jls] executing command
 ** [out :: jls] 2.6.25.11-97.fc9.x86_64
    [mywebserver] executing command
 ** [out :: mywebserver] 2.6.18-53.el5
    command finished
Back at the original problem being solved, we want to download configuration files for any service on any host we care about and store it revision control. For now, let's just grab apache configs from one server.

Learning how to do this in Capistrano proved to be a great exercise in learning a boatload of Capistrano's features. The Capfile is short, but too long to paste here, so click here to view.

If I run "cap pull:apache", Capistrano dutifully downloads my apache configs from 'mywebserver' and pushes them into a local svn repository. Here's what it looks like (I removed some output):

% cap pull:apache
    triggering start callbacks for `pull:apache'
  * executing `ensure:workdir'
At revision 8.
  * executing `pull:apache'
    triggering after callbacks for `pull:apache'
  * executing `pull:sync'
  * executing "echo -n $CAPISTRANO:HOST$"
    servers: ["mywebserver"]
    [mywebserver] executing command
    servers: ["mywebserver"]
 ** sftp download /etc/httpd/conf -> /home/configs/work/mywebserver
    [mywebserver] /etc/httpd/conf/httpd.conf
    [mywebserver] /etc/httpd/conf/magic
    [mywebserver] done
  * sftp download complete
    servers: ["mywebserver"]
 ** sftp download /etc/httpd/conf.d -> /home/configs/work/mywebserver
    [mywebserver] /etc/httpd/conf.d/README
    [mywebserver] /etc/httpd/conf.d/welcome.conf
    [mywebserver] /etc/httpd/conf.d/proxy_ajp.conf
    [mywebserver] done
  * sftp download complete
A         /home/configs/work/mywebserver/README
A         /home/configs/work/mywebserver/httpd.conf
A         /home/configs/work/mywebserver/magic
A         /home/configs/work/mywebserver/welcome.conf
A         /home/configs/work/mywebserver/proxy_ajp.conf
    command finished
Adding         configs/work/mywebserver/README
Adding         configs/work/mywebserver/httpd.conf
Adding         configs/work/mywebserver/magic
Adding         configs/work/mywebserver/proxy_ajp.conf
Adding         configs/work/mywebserver/welcome.conf
Transmitting file data .....
Committed revision 9.
If I then modify 'httpd.conf' on the webserver, and rerun 'cap pull:apache':
<output edited for content>
% cap pull:apache
 ** sftp download /etc/httpd/conf -> /home/configs/work/mywebserver
    [mywebserver] /etc/httpd/conf/httpd.conf
    [mywebserver] /etc/httpd/conf/magic
    [mywebserver] done
  * sftp download complete
Sending        configs/work/mywebserver/httpd.conf
Transmitting file data .
Committed revision 10.
Now if I want to see the diff against the latest two revisions, to see what we changed on the server:
% svn diff -r9:10 file:///home/configs/svn/mywebserver/httpd.conf
Index: httpd.conf
===================================================================
--- httpd.conf  (revision 9)
+++ httpd.conf  (revision 10)
@@ -1,3 +1,4 @@
+# Hurray for revision control!
 #
 # This is the main Apache server configuration file.  It contains the
 # configuration directives that give the server its instructions.
This kind of solution is not necessarily ideal, it's a good and simple way to get history tracking on your config files right now until you have the time, energy and need to improve the way you do config management.

Capistrano might just help you with deployment and other common, remote access tasks.

Further reading:

Capistrano homepage
Capistrano cheat-sheet
RANCID
A similar idea presented here (download config files and put them in revision control) but for network gear.
Coverage for this was suggested by Jon Heise, who helpfully provided me with a intro to Capistrano. <3