CI/CD Life

Update from Matt

2022-06-28T00:20:00-04:00

Brief update from Matt Bajor / Unauthorized

Image of Moonrock Mountain by Matt Bajor

Hi everyone! As you may have noticed, there has not been much movement here on CICD 4 Life. I’ve been working on various other projects, including a venture into building and touring large scale art. While I am still working in DevOps (at Workday), my general direction involves moving towards art and sculpture.

However, until we get there I intend to keep updating here and queue up a few new posts on Kubernetes, GKE, and Jenkins.

Until then please check out my new website Winterworks: Fine Art Firepits and Dichroic Sculpture

Update from Matt was originally published by Matt Bajor at CI/CD Life on June 28, 2022.

Containerizing local commands

2018-08-29T00:20:00-04:00

Running `yarn` from a container

Image from https://yarnpkg.com/en/

Within a continuous integration / continuous delivery system one of the hardest problems to deal with as the system and teams grow is build time dependency management. At first, one version of a few dependencies is totally manageable in traditional ways. yum install or apk add is the perfect solution for a single team or project.

Over time as the CI system begins to grow and support more and more teams this solution starts to show weaknesses. Anyone who has had to install rvm, nvm, or any of the other environment based tool version managers within a CI system knows the pain of getting things working right in a stable way. In addition, distributing the configuration for these tools can be just as challenging, especially when dealing with multiple versions of a tool.

One way to help address these complications (in addition to a few others) is to contain your tooling in Docker containers. This gives a few advantages over traditional package managers:

The software is entirely contained within Docker and can be installed or removed without fuss by any job that needs it
Containers can (and should) be built in house as to be reproducibly built at a minimum and fully audited and tracked within the change management tool if required
Multiple versions and configurations can exist side by side without any interference
Containers can be shared between teams and shell wrappers distributed via Homebrew

How can we run yarn in a container for a local project?

TL;DR: Docker volumes

The way the process works is similar to other Docker workflows:

We build a Docker image that has whatever tool we need
Then any configuration (no secrets!) are layered in to the image
When run the container, we mount our current working directory into the container
The command is executed in the running container, but acting on the local directory that is volume mounted
The command finishes and all output is captured in our local directory
The container exits and since we’ve used --rm, is completely gone

This gives us the effects of the tool without modifying other parts of the filesystem as one does in normal software installation. One issue that this process has as stated is that the command to run the container can be a bit unwieldy. In order to simplify things in this regard we must sacrifice the ‘no changes to the filesystem’ benefit by shipping a shell wrapper. Nonetheless, shipping a shell script or allowing others to create their own is a bit easier than full package / config management.

Anyways, now that we have an idea of how it will work, let’s take a look at how it does work.

Step 1: Build a Docker image for `yarn`

The first item we need is a Docker image that contains our tooling. In this case we’re going to install yarn, a common dependency manager for Node, similar to npm. Since Yarn depends upon Node, we can create a container that has the specific version of each that is needed by the team using it. In this case we will install the latest Node and Yarn packages, but pinning them to other versions would be a fairly simple task.

Let’s take a look at our Dockerfile here:

# We're using alpine for simplicity. We could make it smaller
# by downloading the tarball and adding to scratch with a multi-stage
# Docker build
FROM alpine

# Install yarn which should pull in node as a dependency
RUN apk add --update yarn

# We will configure the cache to use our volume mounted workspace
RUN yarn config set cache-folder /workspace/.yarn-cache

# Using an entrypoint allows us to pass in any args we need
ENTRYPOINT ["yarn"]

This super simple Dockerfile will give us a container that has yarn as an entrypoint. We can now build the image using a command like so:

docker build -t technolog/run-yarn .

This will give us the container named technolog/run-yarn that we can test with a command like this:

docker run --rm -ti technolog/run-yarn --version
# 1.3.2

Excellent, yarn works! However, in this configuration we have no way to have it operate on a local package.json or node_modules. Everything is still in the container. We will fix that by using a Docker volume mount.

Step 2: Volume mount the current directory into the Docker container

If we were to just run the container with the example above, nothing is going to happen outside of the container. What we need to do is make the current directory accessible inside the container with a Docker volume. This is a simple task and looks something like:

docker run --rm -ti -v "$(pwd):/workspace" technolog/run-yarn init
# yarn init v1.3.2
# question name: left-left-pad
# question version (1.0.0): 10.0.0
# question description: A left pad for the left pad
# question entry point (index.js):
# question repository url: https://github.com/technolo-g/left-left-pad
# question author: Not me!
# question license (MIT):
# question private: no
# success Saved package.json
# Done in 76.15s.
cat ./package.json
# {
#   "name": "left-left-pad",
#   "version": "10.0.0",
#   "description": "A left pad for the left pad",
#   "main": "index.js",
#   "repository": "https://github.com/technolo-g/left-left-pad",
#   "author": "Not me!",
#   "license": "MIT"
# }

Here we can see that the file was created locally (./) and contains all of the info we provided to yarn running in the Docker container. Pretty neat! One thing you may notice is that the Docker command is growing a bit. This exact command (or the one you create) doesn’t roll off of the fingers and so can be hard to have everyone typing the same thing. There are a few solutions to this minor issue and one of them is using bash aliases like so:

alias yarny="docker run --rm -ti -v \"\$(pwd):/workspace\" technolog/run-yarn"
which yarny
# yarny: aliased to docker run --rm -ti -v "$(pwd):/workspace" technolog/run-yarn
yarny --version
# 1.3.2

If we are using this command in a lot of places and especially within the build system, a slightly more robust wrapper may be required. Sometimes dealing with the shell and it’s intricacies is best left to ZSH developers and a script is a more unambiguous approach. What I mean by that is a script that encapsulates the Docker command and is then installed on the user or machine’s path. Let’s take a look at one for yarn:

#!/bin/bash -el
# -e Makes sure we exit on failures
# -l Gives us a login shell

# Define a function that replaces our command
yarn() {
    # docker run
    # Remove the container when it exits
    # Mount ./ into the container at /workspace
    # Allow interactive shell
    # Provision a TTY
    # Specify the container and pass any args
  docker run \
    --rm \
    --volume "$(pwd):/workspace" \
    --interactive \
    --tty \
    technolog/run-yarn $@
}

# Run the function, passing in all args
yarn "$@"

Now if we make this file executable and run it, we should have a fully working yarn installation within a container:

chmod +x yarn
./yarn --version
# 1.3.2

Step 3: Distribute the software

The final step is getting these commands to be available. The beauty of this solution is that there are many ways to distribute this command. The script can live in a variety of places, depending on your needs:

In the code repo itself for a smaller team
Included and added to the path of the build runner
Distributed locally with Homebrew
Kept in a separate repo that is added to the path of builds

It depends on your environment, but I prefer to make the scripts available by keeping them all in one repo, cloning that repo, and adding it to the path on a build. This allows the scripts to change with the build and versions to be pinned via git tags if needed. Every team can include the scripts they need and use the version that works for them if they have to pin.

Step 4: ….

Run the build with whatever tools are required

Step 5: Cleanup

Now that we’re done with the tools, let’s wipe them out completely. We will do that using Docker’s prune command:

docker rm -fv $(docker ps -qa) || echo "INFO: No containers to remove"
docker system prune --all --force --volumes

This will kill any running containers and then prune (delete):

Any stopped containers
Any unused networks
Any unused images
Any build cache
Any dangling images

Pretty much anything we would worry about interfering with the next build. If there are containers (such as the drone itself) that must be kept alive, the command is a bit different, but more or less the same.

Enhancements

build.sh

Building the Docker image repeatably and consistently is key to this whole approach. Changing how the container works depending on who builds it will lead to the same pitfalls of bad dependency management: mainly broken builds.

Here is an example build.sh that I would use for the above container:

#!/bin/bash -el

# If PUSH is set to true, push the image
push="$PUSH"

# Set our image name here for consistency
image="technolog/run-yarn"

# Run the build, adding any passed in params like --no-cache
docker build "$@" -t "$image" $(dirname -- "$0")
if [ "$push" == "true" ]; then
  docker push "$image"
fi

Versioning

Once teams begin using this framework, you’ll find each develops a set of version requirements that may not match all the rest of the teams. When you find yourself in this scenario, it is time to begin versioning the images as well. While :latest should probably always point at the newest version, it’s also reasonable to create :vX.X tags as well so teams can pin to specific versions if desired.

In order to do this, you can add a Docker build argument or environment variable to install a specific version of a piece of software and use that version to tag the image as well. I am going to leave this as an exercise for the user, but the steps would be:

Read the version in build.sh
Pass that version as a build arg to docker build
In the Dockerfile, read that ARG and install a specific version of the software

This becomes a bit more complex when sharing between teams and requiring different versions of both node and yarn, but it can be managed with a smart versioning scheme.

Disclaimer!

This methodology does not encourage just pulling random images from Docker hub and running them! You must always use your good judgement when deciding what software to run in your environment. As you see here, we have used the trusted Alpine Docker image and then installed yarn from trusted Alpine packages ourselves. We did not rely on a random Docker image found on the hub, nor did we install extra software that was not required or executed untrusted commands (curl | sudo bash). This means as long as we trust Alpine, we should be able to trust this image, within reason. As my Mum would say: Downloading unknown or unsigned binaries from the Internet will kill you!

Conclusion

This is a powerful and flexible technique for managing build time dependencies within your continuous integration / continuous delivery system. It is a bit overkill if you have a single dependency and can change it without affecting anything unintended. However, if you, like me, run many versions of software to support many teams’ builds, I think you’ll find this to be a pretty simple and potentially elegant solution.

Containerizing local commands was originally published by Matt Bajor at CI/CD Life on August 29, 2018.

Making HTTP GET and POST Requests to Docker Hub in a Jenkinsfile

2017-08-13T00:20:00-04:00

NOTE: If you intend on using techniques such as these and allowing such wide open functionality in Jenkins, I recommend that you run your entire Jenkins build system without outbound internet access by default. Allow access from the build system network segment only to approved endpoints while dropping the rest of the traffic. This will allow you to use potentially dangerous, but extremely powerful scripts while maintaining a high level of security in the system.

API Calls

Making HTTP calls in a Jenkinsfile can prove tricky as everything has to remain serializable else Jenkins cannot keep track of the state when needing to restart a pipeline. If you’ve ever received Jenkins’ java.io.NotSerializableException error in your console out, you know what I mean. There are not too many clear-cut examples of remote API calls from a Jenkinsfile so I’ve created an example Jenkinsfile that will talk to the Docker Hub API using GET and POST to perform authenticated API calls.

Explicit Versioning

The Docker image build process leaves a good amount to be desired when it comes to versioning. Most workflows depend on the :latest tag which is very ambiguous and can lead to problems being swallowed within your build system. In order to maintain a higher level of determinism and auditability, you may consider creating your own versioning scheme for Docker images.

For instance, a version of 2017.2.1929 #<year>-<week>-<build #> can express much more information than a simple latest. Having this information available for audits or tracking down when a failure was introduced can be invaluable, but there is no built-in way to do Docker versioning in Jenkins. One must rely on an external system (such as Docker Hub or their internal registry) to keep track of versions and use this system of record when promoting builds.

This versioning scheme we are using is not based on Semver¹, but it does encode within it the information we need to keep versions in lock and also will always increase in value. Even if the build number is reset, the date + week will keep the versions from ever being lower that the day previously. Version your artifacts however works for your release, but please make sure of these two things:

The version string never duplicates
The version number never decreases

Interacting with the Docker Hub API in a Jenkinsfile

For this example we are going to connect to the Docker Hub REST API in order to retrieve some tags and promote a build to RC. This type of workflow would be implemented in a release job in which a previously built Docker image is being promoted to a release candidate. The steps we take in the Jenkinsfile are:

Provision a node
- Stage 1
  - Make an HTTP POST request to the Docker Hub to get an auth token
  - Use the token to fetch the list of tags on an image
  - Filter through those tags to find a tag for the given build #
- Stage 2
  - Promote (pull, tag, and push) the tag found previously as ${version}-rc
  - Push that tag to latest to make it generally available

This is a fairly complex looking Jenkinsfile as it stands, but these functions can be pulled out into a shared library² to simplify the Jenkinsfile. We’ll talk about that in another post.

Jenkinsfile

#!groovy
/*
 NOTE: This Jenkinsfile has the following pre-requisites:
   - SECRET (id: docker-hub-user-pass): Username / Password secret containing your
     Docker Hub username and password.
   - ENVIRONMENT: Docker commands should work meaning DOCKER_HOST is set or there
     is access to the socket.
*/
import groovy.json.JsonSlurperClassic; // Required for parseJSON()

// These vars would most likely be set as parameters
imageName = "technolog/serviceone"
build = "103"

// Begin our Scripted Pipeline definition by provisioning a node
node() {

    // First stage sets up version info
    stage('Get Docker Tag from Build Number') {

        // Expose our user/pass credential as vars
        withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
            // Generate our auth token
            token = getAuthTokenDockerHub(user, pass)
        }

        // Use our auth token to get the tag
        tag = getTagFromDockerHub(imageName, build, token)
    }

    // Example second stage tags version as -release and pushes to latest
    stage('Promote build to RC') {
        // Enclose in try/catch for cleanup
        try {
            // Define our versions
            def versionImg = "${imageName}:${tag}"
            def latestImg = "${imageName}:latest"

            // Login with our Docker credentials
            withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
                sh "docker login -u${user} -p${pass}"
            }

            // Pull, tag, + push the RC
            sh "docker pull ${versionImg}"
            sh "docker tag ${versionImg} ${versionImg}-rc"
            sh "docker push ${versionImg}-rc"

            // Push the RC to latest as well
            sh "docker tag ${versionImg} ${latestImg}"
            sh "docker push ${latestImg}"
        } catch(err) {
            // Display errors and set status to failure
            echo "FAILURE: Caught error: ${err}"
            currentBuild.result = "FAILURE"
        } finally {
            // Finally perform cleanup
            sh 'docker system prune -af'
        }
    }
}

// NOTE: Everything below here could be put into a shared library

// GET Example
// Get a tag from Docker Hub for a given build number
def getTagFromDockerHub(imgName, build, authToken) {

    // Generate our URL. Auth is required for private repos
    def url = new URL("https://hub.docker.com/v2/repositories/${imgName}/tags")
    def parsedJSON = parseJSON(url.getText(requestProperties:["Authorization":"JWT ${authToken}"]))

    // We want to find the tag associated with a build
    // EX: 2017.2.103 or 2016.33.23945
    def regexp = "^\\d{4}.\\d{1,2}.${build}\$"

    // Iterate over the tags and return the one we want
    for (result in parsedJSON.results) {
        if (result.name.findAll(regexp)) {
            return result.name
        }
    }
}

// POST Example
// Get an Authentication token from Docker Hub
def getAuthTokenDockerHub(user, pass) {

    // Define our URL and make the connection
    def url = new URL("https://hub.docker.com/v2/users/login/")
    def conn = url.openConnection()
    // Set the connection verb and headers
    conn.setRequestMethod("POST")
    conn.setRequestProperty("Content-Type", "application/json")
    // Required to send the request body of our POST
    conn.doOutput = true

    // Create our JSON Authentication string
    def authString = "{\"username\": \"${user}\", \"password\": \"${pass}\"}"

    // Send our request
    def writer = new OutputStreamWriter(conn.outputStream)
    writer.write(authString)
    writer.flush()
    writer.close()
    conn.connect()

    // Parse and return the token
    def result = parseJSON(conn.content.text)
    return result.token

}

// Contain our JsonSlurper in a function to maintain CPS
def parseJSON(json) {
    return new groovy.json.JsonSlurperClassic().parseText(json)
}

Script Security

Due to the nature of this type of script, there is definitely a lot of trust assumed when allowing something like this to run. If you follow the process we are doing in Modern Jenkins nothing is getting into the build system without peer review and nobody but administrators have access to run scripts like this. With the environment locked down, it can be safe to use something of this nature.

Jenkins has two ways in which Jenkinsfiles (and Groovy in general) can be run: sandboxed or un-sandboxed. After reading Do not disable the Groovy Sandbox by rtyler (@agentdero on Twitter), I will never disable sandbox again. What we are going to do instead is whitelist all of the required signatures automatically with Groovy. The script we are going to use is adapted from my friend Brandon Fryslie and will basically pre-authorize all of the required methods that the pipeline will use to make the API calls.

Pre-authorizing Jenkins Signatures with Groovy

URL: http://localhost:8080/script

import org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval

println("INFO: Whitelisting requirements for Jenkinsfile API Calls")

// Create a list of the required signatures
def requiredSigs = [
    'method groovy.json.JsonSlurperClassic parseText java.lang.String',
    'method java.io.Flushable flush',
    'method java.io.Writer write java.lang.String',
    'method java.lang.AutoCloseable close',
    'method java.net.HttpURLConnection setRequestMethod java.lang.String',
    'method java.net.URL openConnection',
    'method java.net.URLConnection connect',
    'method java.net.URLConnection getContent',
    'method java.net.URLConnection getOutputStream',
    'method java.net.URLConnection setDoOutput boolean',
    'method java.net.URLConnection setRequestProperty java.lang.String java.lang.String',
    'new groovy.json.JsonSlurperClassic',
    'new java.io.OutputStreamWriter java.io.OutputStream',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods findAll java.lang.String java.lang.String',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.io.InputStream',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.net.URL java.util.Map',

    // Signatures already approved which may have introduced a security vulnerability (recommend clearing):
    'method java.net.URL openConnection',
]

// Get a handle on our approval object
approver = ScriptApproval.get()

// Aprove each of them
requiredSigs.each {
    approver.approveSignature(it)
}

println("INFO: Jenkinsfile API calls signatures approved")

After running this script, you can browse to Manage Jenkins -> In-process Script Approval and see that there is a list of pre-whitelisted signatures that will allow our Jenkinsfile to make the necessary calls to interact with the Docker Hub API. You’ll notice there is one method in there that they mark in red as you can see the potential security issues with it. java.net.URL openConnection can be an extremely dangerous method to allow in an unrestricted environment so be careful and make sure you have things locked down in other ways.

Making HTTP GET and POST Requests to Docker Hub in a Jenkinsfile was originally published by Matt Bajor at CI/CD Life on August 13, 2017.

Modern Jenkins Unit 3 / Part 4: Configuring the GitHub Jenkins Plugin

2017-07-31T00:20:00-04:00

Setting up the Jenkins <–> Git Interactions

One task that just about every build system needs to complete before doing anything is getting the code that it will be testing and packaging. For most software development shops in the world, this code is kept in a source code repository in a version control system (VCS). The most common implementation that I have been exposed to is Git. Linus Torvalds with a few kernel developers wrote the initial version of git while working on the kernel after BitKeeper’s copyright holder revoked free use¹. Git is what powers GitHub (surprise, surprise) and is just about the most popular VCS out there. We are even using it to develop our Jenkins CI system!

In order to interact with GitHub, we are going to need a couple of things setup:

A machine user that can be managed independently of your customers
A SSH key secret in the credentials store
Configuration that says to use the credentials in the store for cloning

Once we have these in place we should be able to create a job that clones a repo and does something.

Creating a machine user

Teams of developers change constantly with people being added and removed from the roster quite frequently. To use credentials that are tied to a person is not good practice as it does not separate the actions of the user and the actions of the system. For this reason we will create our own machine user in GitHub and configure her to do the builds.

Note: Do not stage or commit anything during this step. We will be encrypting the sensitive files in the next step.

Browse to http://github.com/ in an Incognito window (we need to create a new account and you’re probably logged in).
Create an account for your machine user. It is not against GitHub’s T.O.S.² to create a machine user, but it is to script this process. You can’t reuse your existing email, so you must have a secondary email setup for this user.
Verify the email and then browse to Settings in the upper right hand corner.
Fill out the info that makes sense and then head over to SSH and GPG Keys
We now need to create an ssh-key for the machine user. We will use a new type of key based on this article: https://blog.g3rt.nl/upgrade-your-ssh-keys.html

PWD: ~/code/modern-jenkins/

# cd into our secrets dir
cd secure

# Generate a ed25519 key in the new RFC4716 format with a random good passphrase.
# Something like: 't0$VQki3RWVim!K*rzA1' and then make sure to note the pw. We
# will be using it in a moment
ssh-keygen -o -a 100 -t ed25519 -f git-ssh-key.priv

# Rename the pubkey
mv git-ssh-key.priv.pub git-ssh-key.pub

# Record that passphrase we used to generate the key in secure/git-ssh-key.pw
vi git-ssh-key.pw

# Record the GitHub username for the user in git-ssh-key.user
echo "CICDLifeBuildBot" > git-ssh-key.user

Now add the contents of git-ssh-key.pub to the SSH Keys in your GitHub Settings page with a good name. We now have a machine user and need to configure Jenkins to use it.

Add the secrets to the credential store

Now that we have the Git credentials that should in theory work for cloning and pushing we need to get them added into the Jenkins credential store. How we will do this is via the init system Groovy and some Docker volume mounts.

Mount the secrets into the master container

Since we are still in development, let’s just add our secure dir to the compose file as a mount. When we get to prod we will change this up.

deploy/master/docker-compose.yml

...
#  # Jenkins master's configuration
#  master:
#    image: mastering-jenkins/jenkins-master
#    ports:
#      - "8080:8080"
#    volumes:
#      - plugins:/usr/share/jenkins/ref/plugins
#      - warfile:/usr/share/jenkins/ref/warfile
       - ${PWD}/../../secure:/secure:ro
#  
#  # Jenkins plugins configuration
#  plugins:
...

Restart the service using our script

PWD: ~/code/modern-jenkins/deploy/master

# Restart
./start.sh

# Confirm that the files have been mounted where we expect
docker exec -t master_master_1 ls /secure
#README.md  git-ssh-key.priv  git-ssh-key.pub  git-ssh-key.pw  git-ssh-key.user

Restarting the master will mount the directory ~/code/modern-jenkins/secure at /secure inside the container. Since we specified ro at the end of the VOLUME definition, the secrets themselves will be read-only. With the secrets loaded into the environment, we can begin developing a script that configures GitHub.

Writing Groovy to configure GitHub

Configuring Jenkins plugins with Groovy can seem to be more like an art than a science. Each plugin operates a little bit differently and since Jenkins has been so popular for so long, plugins are in all different states of repair. The basic gist of how to do any plugin is:

Read the values in from the filesystem or ENV that you need to configure
Get a handle on the configuration object for the plugin
Create a new instance of the configuration objects with the values you want
Update the main configuration with your newly created one
Save the config object

Any plugin can be configured this way and it’s normally a matter of familiarizing yourself with the plugin’s data models + classes enough to create what you need. Let’s take a look at the (well documented) Groovy script that configures our GitHub plugin.

URL: http://localhost:8080/script

images/jenkins-plugins/files/init.groovy.d/02-configure-github-client.groovy

// 02-configure-git-client.groovy
// Thanks to chrish:
// https://stackoverflow.com/questions/33613868/how-to-store-secret-text-or-file-using-groovy
import jenkins.model.*
import com.cloudbees.jenkins.plugins.sshcredentials.impl.*
import com.cloudbees.plugins.credentials.domains.*;
import com.cloudbees.plugins.credentials.*;


// Read our values into strings from the volume mount
privKeyText = new File('/secure/git-ssh-key.priv').text.trim()
passPhraseText = new File('/secure/git-ssh-key.pw').text.trim()
sshUserText = new File('/secure/git-ssh-key.user').text.trim()

// Get a handle on our Jenkins instance
def jenkins = Jenkins.getInstance()

// Define the security domain. We're making these global but they can also
// be configured in a more restrictive manner. More on that later
def domain = Domain.global()

// Get our existing Credentials Store
def store = jenkins.getExtensionList(
  'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
  )[0].getStore()

// Create a new BasicSSHUserPrivateKey object with our values
gitHubSSHKey = new BasicSSHUserPrivateKey(
  CredentialsScope.GLOBAL,
  "git-ssh-key",
  sshUserText,
  new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(privKeyText),
  passPhraseText,
  "GitHub Machine User SSH Creds"
)

// Add the new object to the credentials store
store.addCredentials(domain, gitHubSSHKey)

// Get the config descriptor for the overall Git config
def desc = jenkins.getDescriptor("hudson.plugins.git.GitSCM")

// Set the username and email for git interactions
desc.setGlobalConfigName("${sshUserText}")
desc.setGlobalConfigEmail("${sshUserText}@cicd.life")

// Save the descriptor
desc.save()

// Echo out (or log if you like) that we've done something
println("INFO: GitHub Credentials and Configuration complete!")

If things are setup and working right, you should see our INFO statement printed out below the console. To confirm it worked, browse to the credentials store confirm that we now have a SSH Key secret with the id of git-ssh-key. If you click on “Update” you should see the SSH key (don’t worry, it is armored and you can’t show the passphrase from here). While this is not 100% secure, it is still much better than baking the credentials into the image or a lot of other methods people use to expose secrets. Incremental improvements are what I always say!

Add the groovy to the Docker image

After confirming that the script works, add it to the plugins Docker image by dropping it in images/jenkins-plugins/files/init.groovy.d with the name of 02-configure-github-client.groovy. Then you can rebuild the image and restart the service to verify that everything is working on boot.

PWD: ~/code/modern-jenkins/

# Create images/jenkins-plugins/files/init.groovy.d/02-configure-github-client.groovy
# with the content from the script console.

# Build the image
./images/jenkins-plugins/build.sh

# Restart the service
cd deploy/master
./start.sh
docker-compose logs -f

Watch the log output from the containers to confirm that the scripts are running (the name of the scripts is output in the console out) and that there are no stack traces or other unexpected errors. When the system is fully up and running, browse to the GUI and check to see if the credential has been installed. I think you’ll be pleasantly surprised!

Testing our changes

We should in theory be able to test our changes by creating a tiny little job that just clones a repo. Follow me and we’ll give it a run for it’s money:

Browse to http://localhost:8080
Click New Item in the top left
Name it what you like, select freestyle project, and click OK. This will drop you at the configuration screen.
Under Source Code Management, select Git
For the Repository URL enter: git@github.com:technolo-g/modern-jenkins.git
For the Credentials choose GitHub Machine User SSH Creds
Scroll down to Build and add a shell step
Within the shell step type echo "INFO: dir contents are" && ls
Save the job and run it
The job should be able to clone the repo and list the directory contents. If this is not the case, you must debug what is happening here and try to get it working again. It is critical that the changes we make in each part are working as expected.

Congratulations! You have just performed your first and certainly not last programmatic configuration of Jenkins! Give yourself a pat on the back because that was no easy feat.

Commit, Push, PR

Take extra care when pushing branches that are supposed to have encrypted secrets. Sometimes accidents happen and you want to know about them immediately. Since we have added a bunch of secrets in this branch, pay attention to make sure that all of them are encrypted and none are plaintext.

If you got wicked stuck (or even a little bit off track), take a look here to see the repo’s state at the end of this post. https://github.com/technolo-g/modern-jenkins/tree/unit3-part4

Well, we can definitely run a job now so that’s good news. The bad news is that we created it by hand (yuck!). The next installment will introduce you to a way that eliminates use of the GUI for creating jobs. I personally think it has completely changed the way CI systems are built and I would never hand create a job EVER now.

Modern Jenkins Unit 3 / Part 4: Configuring the GitHub Jenkins Plugin was originally published by Matt Bajor at CI/CD Life on July 31, 2017.

Modern Jenkins Unit 3 / Part 3: Managing Secrets in GitHub

2017-07-30T00:20:00-04:00

Encrypting files in the repo with Transcrypt

In production we will hopefully have a secrets management system, or at a very minimum private repos to store encrypted secrets in. For the purpose of this demo though I will be using a public repo and do not want to expose any sensitive data that we may need to add to this repo. WARNING: To be clear, storing any kind of secrets in a git repo, encrypted or not, may not be a good idea. Consult your local security team for advice. We however, don’t have any state secrets for this demo and very few options.

Transcrypt is a shell script that uses OpenSSL to encrypt and decrypt files in your git repo that are noted in the .gitattributes file. Let’s initialize our repo and confirm that it is working.

PWD: ~/code/modern-jenkins/

# On MacOS you can use brew
brew install transcrypt

# On Linux you have to install OpenSSL then place the script in your path
yum install openssl
  
# Download the script to our PATH
wget -O /usr/local/sbin/transcrypt https://raw.githubusercontent.com/elasticdog/transcrypt/master/transcrypt
  
# Make it executable
sudo chmod +x /usr/local/sbin/transcrypt

# Confirm it works
trancrypt --help

NOTE: More information on the software can be found on the README here: https://github.com/elasticdog/transcrypt

Initialize the repo

We will initialize Transcrypt within the repo on a clean branch. This will hopefully ensure that we can configure Transcrypt prior to adding any secrets to the repo.

PWD: ~/code/modern-jenkins/

# Check out a branch and initialize transcrypt
git checkout -b chore-install_transcrypt

transcrypt
# Encrypt using which cipher? [aes-256-cbc]
# Generate a random password? [Y/n]
# 
# Repository metadata:
# 
#   GIT_WORK_TREE:  /Users/matt.bajor/code/modern-jenkins
#   GIT_DIR:        /Users/matt.bajor/code/modern-jenkins/.git
#   GIT_ATTRIBUTES: /Users/matt.bajor/code/modern-jenkins/.gitattributes
# 
# The following configuration will be saved:
# 
#   CIPHER:   aes-256-cbc
#   PASSWORD: <a really good password, trust me>
# 
# Does this look correct? [Y/n]
# 
# The repository has been successfully configured by transcrypt.

Add the generated PASSWORD to your LastPass, keychain, text document on your desktop, or wherever you store secure passwords. It is very important to not lose that passphrase.

We now want to get the .gitattributes file added up and merged in so we can test if secrets encryption is working as expected so let’s branch, commit, push and PR.

PWD: ~/code/modern-jenkins/

git add .
git commit -m "Initialize Transcrypt in the repo"
git push origin chore-install_transcrypt

# PR, review, merge and then catchup master
git checkout master
git pull
git checkout -b test-secrets

Let’s create a secrets dir and initialize it with a README to confirm that transcrypt is working as expected without potentially exposing secrets on the internet. The way transcrypt works is that it looks at the .gitattributes in the repo root for files to encrypt. If it sees that there are paths to encrypt, it will encrypt them before adding to the tree. It does still allow you to view the changes as plaintext locally which sometimes can lead to confusion as to whether or not its working.

PWD: ~/code/modern-jenkins/

# Look at the .gitattributes
cat .gitattributes
# #pattern  filter=crypt diff=crypt
  
# Echo our new pattern in there
echo 'secure/* filter=crypt diff=crypt' >> .gitattributes
  
# Confirm that the file looks good:
cat .gitattributes
# #pattern  filter=crypt diff=crypt
# secure/* filter=crypt diff=crypt 

# Add a README with a bit of info
mkdir -p secure/
echo "# Transcrypt Secrets\n\n This repo is encrypted with transcrypt" > secure/README.md

# Commit and push
git add .
git commit -m "Add test secrets"
git push origin test-secrets

When opening the PR, you should notice that there are two files changed:

.gitattributes
secure/README.md

You should also notice that the contents of README.md is encrypted. If this is the case, give yourself a thumbs up, merge the PR, and update your local branch.

NOTE: If you see that README.md is not encrypted, do not open the PR. Instead, delete your branch and start over. Follow the instructions on the Transcrypt site if you need more help or information. It is important to confirm that the encryption mechanism is working the way we expect before we put secrets into the repo.

Let’s take this time to add a line to the PULL_REQUEST_TEMPLATE.md file reminding us to check for unencrypted secrets:

./PULL_REQUEST_TEMPLATE.MD

...
# #### Contribution checklist
#
  - [ ] THERE ARE NO UNENCRYPTED SECRETS HERE
# - [ ] The branch is named something meaningful
# - [ ] The branch is rebased off of current master}
...

Now that the repo has been initialized with Transcrypt, we can begin adding some configuration that depends on credentials. In the next segment, we will create a machine user for GitHub and configure the Git plugin to use those credentials when cloning.

The repo from this section can be found under the unit3-part3 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit3-part3

Next Post: Configuring the Jenkins GitHub plugin programmatically with Groovy

Modern Jenkins Unit 3 / Part 3: Managing Secrets in GitHub was originally published by Matt Bajor at CI/CD Life on July 30, 2017.

Modern Jenkins Unit 3 / Part 2: Configure Jenkins URL

2017-07-29T00:20:00-04:00

Configuring the Jenkins URL

Currently it just so happens that the Jenkins URL in the management console does have the proper config as it is set to http://localhost:8080. This is merely a coincidence that the default value and the current address match though. Once we start moving this thing around, it will be very important that it is set to the right value else we’ll have all kinds of strange issues. Since it will definitely have to be configured, let’s start here. It doesn’t hurt that it is a fairly simple example of configuring Jenkins with an environment variable passed by Docker Compose.

Passing from Docker Compose

This is a setting that will change from environment to environment and so I think the best place to set it is in the compose file. Let’s create a variable in there that we can read it in during init and make the right configuration. Edit the compose file and create a JENKINS_UI_URL env variable as well as volumes for the configs themselves.

deploy/master/docker-compose.yml

#---
## deploy/master/docker-compose.yml
## Define the version of the compose file we're using
#version: '3.3'
#
## Define our services
#services:
#  # Jenkins master's configuration
#  master:
#    image: modernjenkins/jenkins-master
#    ports:
#      - "8080:8080"
     environment:
        - JENKINS_UI_URL=http://localhost:8080
#    volumes:
#      - plugins:/usr/share/jenkins/ref/plugins
#      - warfile:/usr/share/jenkins/ref/warfile
       - groovy:/var/jenkins_home/init.groovy.d
#
#  # Jenkins plugins' configuration
#  plugins:
#    image: modernjenkins/jenkins-plugins
#    volumes:
#      - plugins:/usr/share/jenkins/ref/plugins
#      - warfile:/usr/share/jenkins/ref/warfile
       - groovy:/usr/share/jenkins/ref/init.groovy.d
#
## Define named volumes. These are what we use to share the data from one
## container to another, thereby making our jenkins.war and plugins available
#volumes:
#  plugins:
#  warfile:
   groovy:
#

Now that it is set, we should be able to write a groovy init script to read it. But first, we will have to restart Jenkins to pick up the new environment:

PWD: ~/code/modern-jenkins

cd deploy/master
docker-compose down -v
docker-compose up -d

# Confirm that the variable is there:
docker inspect master_master_1 | grep JENKINS_UI

# Should output
# "JENKINS_UI_URL=http://localhost:8080",

With the environment variable now being available to us, we can use the script console at localhost to develop our script that will set the URL of our Jenkins instance.

URL: http://localhost:8080/script

images/jenkins-plugins/files/init.groovy.d/01-cofigure-jenkins-url.groovy

import jenkins.model.*

// Read the environment variable
url = System.env.JENKINS_UI_URL

// Get the config from our running instance
urlConfig = JenkinsLocationConfiguration.get()

// Set the config to be the value of the env var
urlConfig.setUrl(url)

// Save the configuration
urlConfig.save()

// Print the results
println("Jenkins URL Set to " + url)

This should output:

Deploying our Groovy Jenkins Configs

We now have a working script that will manage the URL of Jenkins in any environment that we would be deploying into simply by setting a variable in the compose file. Now we need to make this available to the master so that it can be executed on startup. To do that, we will add a director to the plugins image and then mount it into the master in a similar way to how the war and plugins work.

PWD: ~/code/modern-jenkins/

cd images/jenkins-plugins
mkdir -p files/init.groovy.d/

# Add the file above to this directory as
# files/init.groovy.d/01-cofigure-jenkins-url.groovy

Add the full directory (instead of the individual scripts) of Groovy configuration to the Docker image and then export it as a volume.

images/jenkins-plugins/Dockerfile

...
#  # Install our base set of plugins and their dependencies that are listed in
#  # plugins.txt
#  ADD files/plugins.txt /tmp/plugins-main.txt
#  RUN install-plugins.sh `cat /tmp/plugins-main.txt`
#
   # Add our groovy init files
   ADD files/init.groovy.d /usr/share/jenkins/ref/init.groovy.d
#
...

#  # Export our war and plugin set as volumes
#  VOLUME /usr/share/jenkins/ref/plugins
#  VOLUME /usr/share/jenkins/ref/warfile
   VOLUME /usr/share/jenkins/ref/init.groovy.d
#
#  ...

We can now rebuild the image and pick up our new config that should (hopefully) configure the URL of our Jenkins on boot!

PWD: ~/code/modern-jenkins

./images/jenkins-plugins/build.sh

cd deploy/master/
docker-compose down -v
docker system prune -f
docker-compose up -d
docker-compose logs -f

Ok you should now see that the URL in the Jenkins management console is set to http://localhost:8080! Ahhhh, it was like that before? Hmm. Ok well then let’s break it to confirm it is working. Modify the value in the compose file to something different and restart Jenkins:

PWD: ~/code/modern-jenkins/deploy/master

# Change the JENKINS_UI_URL to something different in docker-compose.yml
perl -pi -e 's/JENKINS_UI_URL=.*/JENKINS_UI_URL=http:\/\/derpyhooves/g' docker-compose.yml

# Restart the stack
docker-compose down -v
docker-compose up -d
docker-compos logs

Did that work? A typo you say? I can’t imagine it. I’ve typed docker-compose over 1000 times today, it’s not possible for me to misspell it. In addition, if you notice the difference between this set of commands and the one earlier, we seem to be drifting to chaos. Let’s take note of that, but address it after we confirm that this current change is working as expected.

Gooood, it does work :) The URL in the management console has been updated to the new, wrong, custom value so we can confirm it works. Let’s commit our code now, but attend to that tiny little pile of tech debt we found (starting and stopping the system differently every time is definitely tech debt). NOTE: Don’t forget to set the JENKINS_UI_URL back to http://localhost:8080

PWD: ~/code/modern-jenkins/

git checkout -b feat-configure_jenkins_url
git add .
git commit -m "Configure the Jenkins URL with Groovy

This change adds an environment variable to the compose file
that sets the URL of the Jenkins instance upon boot. This is
done via the script added to init.groovy.d"

Squirrel!

So we’ve noticed something a bit stinky in our code as we’ve been going about our business. This happens very often in our work lives and attending to little tech debts like this is critical to having quality software. I certainly encourage everyone to leave the code better than they found it and to refactor things when they see something turning into a turd like object.

I also encourage you to make note of these things and take care of them after you are in a place to save game. Switching context between one problem and another can be very expensive mind and time wise so feel free to take a note, create a ticket or something, then finish what you are doing (unless there is an actual issue that needs to be addressed before your code will work). When you submit your PR for review, then jump on that Jira and refactor your heart away.

We need to do everything, but we can only do one thing at a time. Try to be aware of time management.

Let’s get that squirrel

What we noticed was that we were beginning to type the command differently every time we did it. That seems like it should be replaced by a shell script. Let’s create a start script so that this thing starts consistently every time:

deploy/master/start.sh

#!/bin/bash -el

echo "INFO: (re)Starting Jenkins"
docker-compose down -v
docker-compose up -d

echo "INFO: Use the following command to watch the logs: "
echo "docker-compose logs -f"

Write that guy out to deploy/master/start.sh and make it executable with chmod +x deploy/master/start.sh and we’ve now got ourselves a script that will consistently restart our app.

Add that onto the branch with a good message and push, PR, merge, etc.. See, you’re getting the hang of it!

Next let’s get ready to handle some secrets. Shhhhhh!

The repo from this section can be found under the unit3-part2 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit3-part2

Next Post: Managing Jenkins secrets in GitHub with Transcrypt

Modern Jenkins Unit 3 / Part 2: Configure Jenkins URL was originally published by Matt Bajor at CI/CD Life on July 29, 2017.

Modern Jenkins Unit 3 / Part 1: Intro to the Jenkins Groovy Init System

2017-07-28T00:20:00-04:00

Image from Simpsons Mathematics

Configuring the Jenkins Master on Boot with Groovy

One of the greatest strengths of Jenkins is it’s ability to do almost anything. This comes from it’s extremely customizable nature. I think of it as a scheduler that can do any given task on any given trigger with enough configuration. What we are building specifically though is a system to build, test, and deploy our piece of custom software that most likely is at least a little bit different than anything else out there requiring the same tasks. We will need to use Jenkins’ ultra powerful customization toolkit, but do it in a way that strives to be:

Deterministic: Given a set of inputs, there should be only one output. If that output is not as we expect, there is a problem that we should spend time fixing. ie: removing “flaky” tests that fail for “no reason”.
Reliable: The system should have high availability to the users who depend on it. Having a system that is sometimes down and sometimes up does not encourage teams to inject automated builds into their workflows.
Repeatable: This system should be able to be recreated without persistent data from the repo.
Agile: The system should evolve to meet the needs of it’s consumers in a sustainable way. If one team’s jobs or configs are breaking another team’s pipelines, it is a good indication that it is time to split the monolith into two independent systems.
Scalable: As the system becomes more popular, more people are going to utilize it. When this happens, it’s critical to be able to support the increased capacity in not only job runners, but also collaboration from more teams.

Luckily we can treat the code that configures the system in the same way we treat the code the builds and runs the system :)

Intro to the Jenkins init system

Jenkins has a not-much-talked about feature that I have yet to see much information on. It is the Jenkins Groovy Init system and really the only documentation I have been able to find are two articles on the Jenkins wiki: https://wiki.jenkins.io/display/JENKINS/Post-initialization+script

  Post-initialization script
  
  Created by Kohsuke Kawaguchi, last modified by Daniel Beck on Dec 10, 2015
  
  You can create a Groovy script file $JENKINS_HOME/init.groovy, or any .groovy
  file in the directory $JENKINS_HOME/init.groovy.d/, (See Configuring Jenkins
  upon start up for more info) to run some additional things right after Jenkins
  starts up. This script can access classes in Jenkins and all the plugins. So for
  example, you can write something like:
  import jenkins.model.*;
  
  // start in the state that doesn't do any build.
  Jenkins.instance.doQuietDown();
  
  
  Output is logged to the Jenkins log file. For Debian based users, this is
  /var/log/jenkins/jenkins.log

which points to this: https://wiki.jenkins.io/display/JENKINS/Configuring+Jenkins+upon+start+up

  Jenkins can execute initialization scripts written in Groovy if they are present
  during start up. See Groovy Hook Script for details. The hook name for this
  event is "init". Those executions happen at the very end of the initialization,
  and therefore this can be used to pre-configure Jenkins for a particular OEM
  situation. 
  
  While one can always write a plugin to participate in the initialization of
  Jenkins, this script-based approach can be useful as it doesn't require any
  compilation and packaging.

Not super impressive documentation considering how powerful this mechanism is. Using this init system you are able to configure any aspect of the master that you are able to using “Manage Jenkins”. This includes (but is not limited to):

The URL and name of this instance
Authentication and security settings
Secrets in the credential store
Global plugin configuration
Global tool configuration
Adding and removing nodes
Creation of jobs (though we’ll only use it to create one special job)

The Groovy Script Console

Not only does it support configuring so much of the system, it has a direct REPL like environment to code in. This is called the “Script Console” (available at http://localhost:8080/script on our instance) and can be considered a shell into the Jenkins system. This shell has the same abilities of the init system’s shell so it makes for super quick and easy development of the scripts that we will use.

Jenkins Groovy Hello World

Let’s kill two stones with one bird. We will do a quick little Hello World that will introduce you to bot the syntax of groovy as well as how to use the script console.

Stand up your development Jenkins (cd deploy/master && docker-compose up -d)
Browse to the script console at http://localhost:8080/script
Enter the following into the box in front of you:

URL: http://localhost:8080/script

  import jenkins.model.*

  def jenkins = Jenkins.getInstance()
  jenkins.setSystemMessage("I'm Bender, baby! Oh god, please insert liquor!")
  // You can change the message if you please. I'm not at the office :)

Browse back to the main Jenkins interface
Check out the cool message for all the users of your system to see. I bet your boss will love it!

Nothing too crazy, but this should give you a good idea of how we are going to configure our master to build our particular brand of software. Inside Old Mr. Jenkins is just a series of objects (I think of them as his organs) that we can grab and squeeze and modify to fit our needs. I hope Janky is ready to play “Operation”!

Next Post: Configure Jenkins URL with Groovy on boot

Modern Jenkins Unit 3 / Part 1: Intro to the Jenkins Groovy Init System was originally published by Matt Bajor at CI/CD Life on July 28, 2017.

Modern Jenkins Unit 2 / Part 5: Starting Jenkins with Docker Compose

2017-07-27T00:20:00-04:00

“The Good Ole’ Days”

Back in aught eight when I was a kid, the way we deployed complex services was a 1000 line shell script that was neither idempotent nor checked into SCM. It just sat there at an http endpoint, ready for sudo | bashing (I guess normally sudo wasn’t an issue as we ran as root :P). If it needed a tweak, you could just ssh to the server, fire up pico, make the change, deploy your stuff, sit back and relax while wondering why the rest of the team is complaining about deploys not working. After all, it Works on My Machine :)

While I look back with fondness at the days of yore, I can only imagine it is the fresh Colorado air that is making me forget how crappy it is to have one deploy work and then literally the same thing 5 minutes later fails because someone was mucking with the script. So we’re not going to do that.

Instead, we are going to use something called Docker Compose. Docker Compose is a project by Docker that was based on something called fig a long time ago. Unlike the rest of their toolkit, docker-compose is a Python application that uses YAML to describe a service or set of services. It allows you to define pretty much every aspect of how the services are run, what the networking and storage systems will look like, and to fine tune how your app will work via environment variables.

There is a ton of info out there on Docker Compose ¹ so please do take a peek. For now, let’s roll forward into the unknown and create our first compose file.

deploy/master/docker-compose.yml

---
# deploy/master/docker-compose.yml
# Define the version of the compose file we're using
version: '3.3'

# Define our services
services:
  # Jenkins master's configuration
  master:
    image: modernjenkins/jenkins-master
    ports:
      - "8080:8080"
    volumes:
      - plugins:/usr/share/jenkins/ref/plugins
      - warfile:/usr/share/jenkins/ref/warfile

  # Jenkins plugins' configuration
  plugins:
    image: modernjenkins/jenkins-plugins
    volumes:
      - plugins:/usr/share/jenkins/ref/plugins
      - warfile:/usr/share/jenkins/ref/warfile

# Define named volumes. These are what we use to share the data from one
# container to another, thereby making our jenkins.war and plugins available
volumes:
  plugins:
  warfile:

A compose file is made up of a few sections as in the example above. Here the ones we’re using:

version ²: Define what version of compose file this is
services ³: This is where we list out all of the services that we need running. This example is fairly straightforward, but it is possible to include any service your app needs in this section. You’re basically describing the full system and it’s interactions.
volumes ⁴: This is where data storage is described. We’re using it to define two volumes, one for plugins and one for the warfile. Upon creating this volume, data from the container will be copied in. Since the first container does not have anything at that path, the data from the second container is what we get which is exactly what we want.
networks ⁵: Not used here, but a way to define all container networking.

This is a fairly simple example of a compose file so it should be fairly straightforward to understand. You may also notice that it’s very succinct and to the point while still being super readable. This is why I like Docker Compose. We can describe something extremely complex (not so much in this case) as an easy to read YAML file.

Test it out

Ok, here’ we go girls and boys. The big reveal. Our rocket-powered-motorcycle is fueled up and we’re ready to jump the Snake river!

PWD: ~/code/modern-jenkins

# Compose up
cd deploy/master
docker-compose up -d
docker-compose logs -f

The Jenkins app should be starting up now and once it says “Jenkins is fully up and running” you should be able to browse to the UI at http://localhost:8080 and bask in its Janky glory.

Now that we know how to start / stop it, we should add this to the documentation. It is important to keep these docs up to date so that anyone can jump in and start using it without having to do a massive amount of research. Let’s add this to the README:

deploy/README.md

# Deployment Scripts

Jenkins is deployed via Docker Compose. In order to run the service locally, use
the following commands:

```
# Get into the deploy directory
cd deploy/master

# Start the service as a daemon
docker-compose up -d

# View logs
docker-compose logs -f

# Stop Jenkins
docker-compose down -v

# Pull new images
docker-compose pull
```

WTF Matt, a 6 part blog series to replace “java -jar jenkins.war” and 6 clicks?

hahaha, well you got me there! JK. While java -jar jenkins.war and a few mouse clicks could get us to the same place, it would not have been wasting nearly enough bits :trollface:

Obviously there are two potential reasons why we did this:

I am a crazy person
I am a just-crazy-enough person

Luckily for you, the answer is the latter. If you’ll recall, the whole reason I’m writing this is because I’m tired of people showing me their ugly Jenkins and encouraging me to tell them how cute it is.

The problem with most of these monstrosities is not that they don’t build the software. If it didn’t do that it wouldn’t exist at all. The problem is that they are extremely hard, time consuming, dangerous, and unfun to work on.

That’s fine for something that never changes, but as it turns out we’re making things for the internet which is growing and changing constantly meaning that we constantly need to change and adapt in order to move forward. This applies very much so to our build system. It is a system that eventually everyone in the company begins to rely on, from C levels that need profits to PMs and POs who need features, to Engineers who need to do the actual work.

When a CI system turns into a bowl of spaghetti each little necessary change becomes a nerve-racking-afterhours-signed-out-of-slack maintenance that gives us all PTSD after the 30th time it goes south. What we are doing here is implementing a semi-rigid structure for our system to basically manage change effectively while still moving fast.

Let’s walk through some of the common Crappy Times at Jenkins High:

A plugin with a broken dependency: Instead of finding out after checking the ‘restart Jenkins when done’ that a plugin can’t fully resolve it’s dependencies, we will see it when we try to build the Docker image. It is still non-optimal that it’s happening, but it is not a prod outage and builds are still running, preventing a Cincinnati Time Waste tournament.
Rolling credentials for the CI Git user: In the old days, this required a ton of coordination in addition to an outage. We have not yet showed it, but when your secrets are tied to the container we are able to modify all the required credentials, roll the system, and get back at it.
A job that broke for “no reason”: It’s always unpleasant to be surprised by a job that is just no longer going green. When we version all of our jobs, plugins, and master configuration, bisecting what caused a failure (or behavior change) becomes much simpler. We just go back to the point in which the job was running and diff the environment to what is running today. Since we’re able to run everything locally it should be a fairly straightforward process to replicate the problem on your laptop and lower your MTTR.

All of these problems we are talking about are still going to occur, but what we’re doing is pushing the problems down to build time from runtime. We want to find these issues in advance where they are not causing outages. We want to be able to treat our pipelines, configuration, and infrastructure as code to avoid the bowl of spaghetti that is fragile and unkown in nature. The teams should not be called “10ft Pole” (my old team) that help with the build system, they should be called “Golden Retriever Puppies” because everyone wants to play with us.

In conclusion

In conclusion, I hope you are able to see how the beginnings of our system are going to lend themselves to being a fully scalable solution that can scale to hundreds of builds, thousands of developers, and at least 10s of different companies you’re going to work at :)

If you don’t see it quite yet then you’re going to have to trust me that we are indeed doing this work for something and not for nothing. Anyways, no skin off of my nose if you don’t. Just keep typing code monkey.

In the next unit of this series we will begin configuring Jenkins. This will allow you to begin making Jenkins do the stuff you need it to do. Stay tuned for Unit 3 of Modern Jenkins: Programmatically Configuring Jenkins for Immutability with Groovy.

The repo from this section can be found under the unit2-part5 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part5

Next Post: The Jenkins Groovy init system (init.groovy.d)

Modern Jenkins Unit 2 / Part 5: Starting Jenkins with Docker Compose was originally published by Matt Bajor at CI/CD Life on July 27, 2017.

Modern Jenkins Unit 2 / Part 4: The Jenkins Plugin Image

2017-07-26T00:20:00-04:00

The plugins image

You may have noticed that while we called the previous image jenkins-master, we never did drop the war in it. In fact, the only reference to that war we’ve seen is the very base image which sets a version, path, and a checksum. What’s the reason for this madness?

The answer is that the images we have built up until now are only a runtime environment for this image. The master image (the one we just built) will almost never change. When doing an upgrade the war never has new system requirements and rarely changes the directory structure or anything like that.

What does change from deployment to deployment is the set of plugins, version of the Jenkins war, and the configuration that interacts with those things. For this reason I choose to run a vanilla Jenkins master container (with a few environment variable configs passed in) and a highly customized plugin container. This plugin container is where the binaries live and is volume mounted by the master to provide the software itself.

Let’s create it now and we can talk more about it after.

images/jenkins-plugins/Dockerfile

# images/jenkins-plugins/Dockerfile
FROM modernjenkins/jenkins-base
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location=https://github.com/technolo-g/modern-jenkins/tree/master/images/jenkins-plugins/Dockerfile \
      image_name=modernjenkins/jenkins-plugins \
      base_image=modernjenkins/jenkins-base

# Add our plugin installation tool. Can be found here and is modified from the
# upstream version.
# https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-plugins/files/install-plugins.sh
ADD files/install-plugins.sh /usr/local/bin/

# Download the Jenkins war
# JENKINS_URL, JENKINS_ROOT, JENKINS_WAR, and JENKINS_SHA are set in the parent
RUN mkdir -p ${JENKINS_ROOT}/ref/warfile \
  && curl -fsSL ${JENKINS_URL} -o ${JENKINS_WAR} \
  && echo "${JENKINS_SHA}  ${JENKINS_WAR}" | sha256sum -c - \
  && chown -R ${user}:${user} ${JENKINS_ROOT}

# We will run all of this as the jenkins user as is dictated by the base imge
USER ${user}

# Install our base set of plugins and their depdendencies that are listed in
# plugins.txt
ADD files/plugins.txt /tmp/plugins-main.txt
RUN install-plugins.sh `cat /tmp/plugins-main.txt`

# Export our war and plugin set as volumes
VOLUME /usr/share/jenkins/ref/plugins
VOLUME /usr/share/jenkins/ref/warfile

# It's easy to get confused when just a volume is being used, so let's just keep
# the container alive for clarity. This entrypoint will keep the container
# running for... infinity!
ENTRYPOINT ["sleep", "infinity"]

You can see from the Dockerfile that this image is where the action is. We have a similar set of metadata at the top like the other images, then we add a file named install-plugins.sh This file is from the upstream Jenkins Docker image and it’s purpose is to install a set of plugins as well as any depdendencies they have. It can be downloaded from the link provided in the Dockerfile ¹.

Then we go on to download the jenkins war and check it’s SHA. If the SHA does not match what we have in the base image, this step will fail and you know that something amiss is going on. Since the version and the SHA are both set in the very base image they should always match. There is never a scenario in which those two do not match up.

Once the war and tools are installed we can install our set of plugins. The install-plugins.sh script needs the war to run so now we should be ready. What this script is doing in the background is interacting with the Jenkins Update Center to attempt to install each plugin that is listed in plugins.txt. It will reach out to download the plugin and check for any depdendencies the plugin may have. If there are any, it will download those, resolve transitive deps and so on until the full set of plugins defined by us are installed along with any deps they need to function.

NOTE: This is different than the plugins.sh file that is out there. That script will not resolve dependencies and makes it very hard to audit which plugins you actually need.

PWD: ~/code/modern-jenkins/

# Add plugin resolver
cd images/jenkins-plugins
mkdir -p files/
wget -O files/install-plugins.sh \
  https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-plugins/files/install-plugins.sh
chmod +x files/install-plugins.sh

# Add a very base set of plugins to plugins.txt
# Add some credential storage
echo "credentials" >> files/plugins.txt
# Enable GitHub interactions
echo "github" >> files/plugins.txt
# Make our blue balls green
echo "greenballs" >> files/plugins.txt
# Give us Groovy capabilities
echo "groovy" >> files/plugins.txt

If you recall we discussed that this image is only going to provide the software itself and the Jenkins master image will provde the runtime. How that works is that we will export our plugins and warfile via the VOLUME statements at the bottom of this Dockerfile and mount them into the master via --volumes-from. This makes our plugins image an fully contained and versionable bundle of the master war and any plugins we need. A little later on, we will talk about how to include your configuration as well.

Finally we have the ENTRYPOINT. This version is farily simple: sleep infinity. What this does is keeps the container running even though we do not have a running process in it. Since this is only a container for our wars and JPIs, it doesn’t need to run the JVM or anything like that. It only needs to provide it’s exported volumes. If we were to omit the ENTRYPOINT, everything would still work as expected but for the fact that the jenkins-plugin container would not be running.

It would appear to be in a stopped state which for me is very confusing. The container is being used by the master (by way of volumes) and so it is indeed in use. The fact that Docker shows it as stopped is misleading IMO and so this just props up the container for clarity.

Building the image

Well, we’ve got another image to build and I think by this time you know what we’re going to do and it’s not DRY out our builders :P

PWD: ~/code/modern-jenkins/

# Warm up the copy machine...
cd images/jenkins-plugins
cp -rp ../jenkins-master/build.sh .
perl -pi -e 's~jenkins-master~jenkins-plugins~g' build.sh

# Build the image
./build.sh
# yay! I worked on the first time :trollface:

Testing the image

As us rafters say, the proof is at the put-in. Let’s give it a whirl!

PWD: ~/code/modern-jenkins

# Start the plugins container first
docker container run --name plugins -d modernjenkins/jenkins-plugins

# Now fire up the master
docker container run --rm -ti --volumes-from=plugins -p 8080:8080 \
  modernjenkins/jenkins-master
  
# Open the GUI
open http://localhost:8080

Would you look at that? Jenkins seems to be starting up swimmingly! If it is not for you, try to debug what exactly is going wrong. There are a lot of moving parts and this is a fairly complex system so don’t feel bad. It happens to all of us. Except us writing blog tutorials. We are always 100% right and our instructions work 11/10 times so you’re wrong and you should feel bad :P Seriously though, if something is jacked up in these instructions please use your super PR skills and help a brotha out by submitting a PR to the repo.

Unicorn from: http://sperlingsmaedchen.deviantart.com/art/unicorns-fart-rainbows-381339815

Cleaning up

After running tests like these, we definitely need to begin thinking about cleanup. What would happen if we tried to run the same tests again right now? Feel free to try it, but the secret is that it won’t work. We need to delete the remnants from the previous test before starting another so I make it a habit to ensure a clean environmnet before I run a test and attempt to cleanup afterwards. The command I normally use for this is the equivalent of “nuke it ‘till it glows”: docker container rm -fv $(docker ps -qa). This little gem will remove all containers, running or not as well as any volumes they may have created (you may want to read more about that, volumes not in the state you thought they were can ruin your day in lots of ways).

One other thing you may be noticing is that no matter how diligent you are, you’re developing a stack of <none> images, weirdly named volumes, and orphaned networks. This is normal cruft left behind while doing Docker development and it can be removed by using docker system prune . This will remove:

all stopped containers
all volumes not used by at least one container
all networks not used by at least one container
all dangling images ('s)

NOTE: If you really want to clean up, add a -a and it will also remove images not attached to a running container. I find that to be annoying except when we’re in prod, but it is handy there.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
(~) ------------------------------------------------------------------------- 🐳  unset (matt.bajor)
% docker rm -fv $(docker ps -qa)
34b9692447f6
59b24f290270
(~) ------------------------------------------------------------------------- 🐳  unset (matt.bajor)
% docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
Deleted Networks:
master_default

Deleted Images:
untagged: modernjenkins/jenkins-master@sha256:8f4b3bcad8f8aa3a26da394ce0075c631d311ece10cf7c23ce60058a9e47f6ed
deleted: sha256:96c78f549467f8b4697b73eddd9da299d8fd686696b45190a2bba24ad810529a
deleted: sha256:d1f38cb683287825bbf1856efdfaa87e2a7c279ceb793f9831b88b850ae1c9a0
deleted: sha256:5371c45cef2d3c5c468aae4fd5e93c335e8e681f2aa366f6122902c45e8ec9cb
deleted: sha256:079be452ec3e99b51a35b76e67b1bb3af649c3357e3ba05d2b6bd2a8127804b4
deleted: sha256:87baad26b39521ddd0d7b12ac46b2f92344f2f8ad34f0f35c524d5c0c566b409
deleted: sha256:c348763948964e1f63c427bea6b4d38c3a34403b61aee5e7b32059a3c095af32
deleted: sha256:6f92439bdac179e8c980dc6a7eb4f9647545e9c6d34d28edbba3c922efa9ea1e
deleted: sha256:edd5cbd4dc3cb3e9ab54bb1d7f446d5638c4543f04f2b63ae1a3e87a661be7a2
deleted: sha256:7890def677cf6649567c4355ef8f10c359f71c0ac9ca6ab94d8f359a5d57f84d
deleted: sha256:2704ec820811576ee2c60b8a660753939457f88fbe6938c2039489a6047ec59c
deleted: sha256:202acc3c794ce58a5e0b0e6b3285ab5ae27c641804c905a50b9ca7d5c601b2b3
deleted: sha256:70e19603643ce03f9cbff3a8837f1ebfb33fe13df7fba66c2501be96d9a2fb93
deleted: sha256:8e757cb858613c81e5fa8fb2426d22584539c163ce4ab66d6b77bd378ee2817a
deleted: sha256:18d1a064d790f3be371fef00813efe1c78996eab042977b952f4cbf067b846e8
deleted: sha256:bddcbf75436ff49e435fe3c371337b6b12ae125e68e0d833ac6180ffd82f34d9
deleted: sha256:f4dae60dcb2542e532eb05c94abba2da00d5a36360cb1d79cb32f87bf9b9c909
deleted: sha256:12f7c2589fdbb6e8b9ac78983511df70e9613c8da42edf23ee1cdb3599437233
deleted: sha256:26b155d41fabd6881f871945586c623a485688fc67f08223df288522f7aeed87
deleted: sha256:3a7c393698419b8f4f7a1464264459d2662d9015b3d577ad8cb12e0b4ae069a5
deleted: sha256:53794a3680b75ae98f70ab567db86a1b0e262615a8194bad534edfd5d8acc2f2
deleted: sha256:13449dedb3ec5df1f1b969aa2f1f93bb2a3bed2fb3ebe7279cce52b750696031
deleted: sha256:55aae84cda94b4611f73ec70b4cc1ea7ce4bbb77a1999b585fcc46c6239fe2a5
deleted: sha256:b41674288931c4e4bcd43e9fcc0d0af8d9ddd9a31f04506050ce0f0dfc59e3e3

Total reclaimed space: 313.9MB

Commit, push, PR

You know the drill. Integrate early, integrate often. Make sure you actually are looking at the work you’re merging. Afterall, it has your public name on it twice.

If you did get lost (I know I had to make a minor change to my base image), take a look at the unit2-part4 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part4

Next Post: Starting Jenkins with Docker Compose

https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-plugins/files/install-plugins.sh ↩

Modern Jenkins Unit 2 / Part 4: The Jenkins Plugin Image was originally published by Matt Bajor at CI/CD Life on July 26, 2017.

Modern Jenkins Unit 2 / Part 3: Building the Jenkins Master Image

2017-07-25T00:20:00-04:00

NOTE: Make sure you’re checking out a branch at the beginning of each section!

Building our master image

Now that we have a good base to inherit from, we can begin building out the rest of our images inheriting from that one. The next image we need is for the master. This image won’t contain too much other than generic configuration and a couple tools because we want our master image itself to be as generic as possible. The customization of each provisioned Jenkins master consists of configuration and plugins which we will package in a separate image. We will talk more about why it’s broken down this way later on. For now, let’s take a look at what we have for a Jenkins master image (modernjenkins/jenkins-master):

images/jenkins-master/Dockerfile

# images/jenkins-master/Dockerfile
FROM modernjenkins/jenkins-base
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location=https://github.com/technolo-g/modern-jenkins/tree/master/images/jenkins-master/Dockerfile \
      image_name=modernjenkins/jenkins-master \
      base_image=modernjenkins/jenkins-base

# Jenkins' Environment
ENV COPY_REFERENCE_FILE_LOG $JENKINS_HOME/copy_reference_file.log

# `/usr/share/jenkins/ref/` contains all reference configuration we want 
# to set on a fresh new installation. Use it to bundle additional plugins 
# or config file with your custom jenkins Docker image.
RUN mkdir -p /usr/share/jenkins/ref/init.groovy.d

# # Disable the upgrade banner & admin pw (we will add one later)
RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state \
    && echo 2.0 > ${JENKINS_HOME}/jenkins.install.InstallUtil.lastExecVersion

# Fix up permissions
RUN chown -R ${user} "$JENKINS_HOME" /usr/share/jenkins/ref

# Install our start script and make it executable
# This script can be downloaded from
# https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-master/files/jenkins.sh
COPY files/jenkins.sh /usr/local/bin/jenkins.sh
RUN chown jenkins /usr/local/bin/* && chmod +x /usr/local/bin/*

# Make our jobs dir ready for a volume. This is where job histories
# are stored and we are going to use volumes to persist them
RUN mkdir -p ${JENKINS_HOME}/jobs && chown ${user}:${group} ${JENKINS_HOME}/jobs

# Install Docker (for docker-slaves plugin)
RUN yum-config-manager --add-repo \
      https://download.docker.com/linux/centos/docker-ce.repo \
    && yum makecache fast \
    && yum install -y docker-ce \
    && yum clean all -y

# Switch to the Jenkins user from now own
USER ${user}

# Configure Git
RUN git config --global user.email "jenkins@cicd.life" \
    && git config --global user.name "CI/CD LIfe Jenkins"

# Main web interface and JNLP slaves
EXPOSE 8080 50000
ENTRYPOINT ["/usr/local/bin/jenkins.sh"]

Looking at this Dockerfile, you may see a few new things like USER (will run the commands after this declaration as the defined user) and EXPOSE (exposes defined ports for binding to an outside port), but for the most part it’s very similar to the previous one. Set a few ENV vars, RUN a few commands etc.

We need a build script so we’ll do the same thing that we did before (except now we have the script in our repo) by creating a build.sh that can also push. Let’s just duplicate this now:

PWD: ~/code/modern-jenkins/

cd images/jenkins-master
cp -rp ../jenkins-base/build.sh .
perl -pi -e 's~jenkins-base~jenkins-master~g' build.sh

Now we have a nice little build script for this image too. While a puppy might have died when we copy/pasta’d I didn’t hear it whimper.

There is one more file that we need for this image and it’s the startup script. Since the internet was generous enough to provide one, we should just use it. This is the script that powers the official image and I’ve got a copy of it just for you in my repo. To retrieve it, use wget:

PWD: ~/code/modern-jenkins/

cd images/jenkins-master
mkdir files
wget -O files/jenkins.sh \
  https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-master/files/jenkins.sh
chmod +x files/jenkins.sh

Build the image and test it out

Now that we’ve got all the files created that our image depends on, let’s build and test it a bit.

PWD: ~/code/modern-jenkins/

# Build it
cd images/jenkins-master
./build.sh

# Run it
docker container run --rm -ti modernjenkins/jenkins-master bash
docker version

# You should see the Docker client version only

Commit, push, PR

The master image seems to be gtg so let’s get it integrated. You may now be seeing what we mean by ‘continuous integration’. Every time we have a small chunk of usable work, we integrate it into the master branch. This keeps change sets small and makes it easier for everyone to incorporate the steady stream of changes into their work without spending days in Git hell.

You can compare your git tree to mine at state at the unit2-part3 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part3 The Docker images are also available to pull if you don’t feel like building them for some reason.

Our next move will be to build the meat of our system: the plugins container.

Next Post: Building the Jenkins Plugin image

Modern Jenkins Unit 2 / Part 3: Building the Jenkins Master Image was originally published by Matt Bajor at CI/CD Life on July 25, 2017.

Modern Jenkins Unit 2 / Part 2: Building the Base Jenkins Image (and Intro to Docker)

2017-07-24T00:20:00-04:00

Note: I am assuming familiarity with Docker for this tutorial, but even if you’ve never used it I think it should still be possible to follow along. It always helps to know your tools though so if you’re unfamiliar take some time to do a Docker Hello World or the like. It will be worth your investment in time as we will be using this technology throughout the tutorial. Everything we will do is based on Docker for Mac which you can download here: https://www.docker.com/docker-mac Linux users should be able to follow along without much adjustment too.

Building our Images

Well I think we’re fully setup with a great foundation at this point. We have a general spec for the system we would like to create, we have a nicely organized code repository hosted in GitHub and setup with a Grade A PR template that will ensure we’re thinking about what we’re doing, and we have a workflow that works for us and is reusable in any situation (nobody hates receiving PRs). It is time to actually begin writing some code!

Nearly every software vendor provides a Docker image for their piece of software which is super awesome when spiking things out or researching a new technology. The reality of it is though that a lot of companies have a security requirement that all software is vetted by the security team and then consumed from internal repositories. These repositories are served up by tools such as Artifactory ¹ and feature built-in security scanning via Black Duck ², permission models that allow only certain users to publish, and promotion mechanisms for getting only verifiable software into the environment. Pulling Docker images straight off of the Hub does not fit into that model at all.

For that reason, we are going to develop a set of our own images with a common base. This gives us commonality between images which has many benefits including flexibility to add only the software that we want to. Now our examples will use public servers for all of this activity, but you can substitute those URLs for the URLs of your own internal artifact repository.

While we don’t want to trust every Docker image that has been published, we do have to start our chain of trust somewhere. In our case we will start with the CentOS 7 base image from the Docker Hub. There are lots of other great options out there, such as Alpine and Ubuntu, but I think CentOS is perfectly fine for this application and is what I use on a daily basis due to certain requirements.

modernjenkins/jenkins-base

This image contains purely the JDK. Since we decided to base this image on CentOS (for security, support, compatibility, and reliability to name a few reasons) that is where our chain of trust begins. I personally have been trusting CentOS DVDs for an extremely long time so I feel confident they are a good place to start. On top of the Centos 7 base we will install the OpenJDK and setup a few environment vars. Let’s show the whole file and then talk about what each of the sections are.

images/jenkins-base/Dockerfile

# images/jenkins-base/Dockerfile
FROM centos:7
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location=https://github.com/technolo-g/modern-jenkins/tree/master/images/jenkins-base/Dockerfile \
      image_name=modernjenkins/jenkins-base \
      base_image=centos:7


# Jenkins' Environment
ENV JENKINS_HOME /var/jenkins_home
ENV JENKINS_ROOT /usr/share/jenkins
ENV JENKINS_WAR /usr/share/jenkins/ref/warfile/jenkins.war
ENV JENKINS_SLAVE_AGENT_PORT 50000
ENV user=jenkins
ENV group=jenkins
ENV uid=1000
ENV gid=1000

# Jenkins Version info
ENV JENKINS_VERSION 2.69
ENV JENKINS_SHA d1ad00f4677a053388113020cf860e05a72cef6ee64f63b830479c6ac5520056

# These URLs can be swapped out for internal repos if needed. Secrets required may vary :)
ENV JENKINS_UC https://updates.jenkins.io
ENV JENKINS_URL http://mirrors.jenkins.io/war/${JENKINS_VERSION}/jenkins.war

# Jenkins is run with user `jenkins`, uid = 1000
# If you bind mount a volume from the host or a data container,
# ensure you use the same uid
RUN groupadd -g ${gid} ${group} \
    && useradd -d "$JENKINS_HOME" -u ${uid} -g ${group} -s /bin/bash ${user}

# Install our tools and make them executable
COPY files/jenkins-support /usr/local/bin/jenkins-support
RUN mkdir -p ${JENKINS_ROOT}
RUN chown jenkins /usr/local/bin/* ${JENKINS_ROOT} \
    && chmod +x /usr/local/bin/*

# Configure to Denver timezone. I dislike debugging failures in UTC
RUN unlink /etc/localtime && ln -s /usr/share/zoneinfo/America/Denver /etc/localtime

# Install Java, Git, and Unzip
RUN yum install -y java-1.8.0-openjdk-devel tzdata-java git unzip \
    && yum clean all

The above Dockerfile will be our base image that everything else will inherit from. While we are initially only creating a single Jenkins master, you may find that others in your organization would like their own Jenkins instance and this pattern ensures you’re ready for it without sacrificing readability. Now let’s talk about what is in this Dockerfile.

Metadata

# images/jenkins-base/Dockerfile
FROM centos:7
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location=https://github.com/technolo-g/modern-jenkins/tree/master/images/jenkins-base/Dockerfile \
      image_name=modernjenkins/jenkins-base \
      base_image=centos:7

This information is critical when tracking down a source in the supply chain as well as for new contributors who want to change how the container works.

# comment at the top is just the path within the repo to the file itself
FROM defines the image that we are building on top of
MAINTAINER tells who the maintainer of this image is
LABEL section provides labels that can be accessed with docker inspect

Environment

# Jenkins' Environment
ENV JENKINS_HOME /var/jenkins_home
ENV JENKINS_ROOT /usr/share/jenkins
...

These environment variables values that we want to have permanently baked into the image. They will be available in any container that is instantiated from this image or any other that inherits it. These types of variables make it easy to bring consistency across the environment.

Files & Commands (Actually doing the work)

RUN groupadd -g ${gid} ${group} \
    && useradd -d "$JENKINS_HOME" -u ${uid} -g ${group} -s /bin/bash ${user}

# Install our tools and make them executable
COPY files/jenkins-support /usr/local/bin/jenkins-support
...

These steps actually modify our image by installing software, modifying the filesystem, adding files from the build context, etc. They can use the ENV vars set above or arguments passed in as well as all other kinds of manipulations. You can see all the possible commands here: https://docs.docker.com/engine/reference/builder/

Adding the jenkins-support file to the repo

We depend on a file called jenkins-support to make things work correctly. It is basically a shim to get Jenkins working within a Docker container properly. It cant be downloaded from my repo like so:

PWD: ~/code/modern-jenkins

cd images/jenkins-base
mkdir files
wget -O files/jenkins-support https://raw.githubusercontent.com/technolo-g/modern-jenkins/master/images/jenkins-base/files/jenkins-support
chmod +x files/jenkins-support

Notes about images

Each line in a Dockerfile creates a layer and then all of these layers are mushed together (or was it squished?) to make our root fs. This mushing process is only ever additive so what that means is if you create a big file in one RUN step but then remove it in another RUN step, you’re not actually going to see any difference in image size. The key is finding the right balance between number of layers and size of layers. If we can keep layers under 50mb but still split up our logical process into easily understood and intuitive blocks (ie: doing a full yum transaction in one RUN block) then we’re sittin’ pretty.

From the Docker website

There is so much more I would like to tell you about best practices that I’ve found around image creation that I will have to save it for another post. Just know for now, we can never delete data that was created in a previous layer. That will directly translate into cleaning up after yourself in an atomic action. A real example is this:

# Install Java, Git, and Unzip then cleanup
RUN yum install -y java-1.8.0-openjdk-devel tzdata-java git unzip \
    && yum clean all

Building the image

Now that we have a super awesome Dockerfile, we need to build it. Normally I would have you do docker image build -t blah/boring . etc., but today I’m going to set your future self up for a win. We’re going to write a script right off the bat to build this thing. I promise you that you will be rebuilding this image at least 2 more times so let’s just go ahead and script it from the get go.

images/jenkins-base/build.sh

#!/bin/bash -el
# images/jenkins-base/build.sh

# Define our image name
image_name=modernjenkins/jenkins-base:latest

# Accept any args passed and add them to the command
docker image build ${@} -t $image_name $(dirname -- "$0")

# If we add PUSH=true to the command, it will push to the hub
if [ "$PUSH" = true ] ; then
  docker image push $image_name
fi

This will not be the last time we see this lil’ guy as we will add it to all of the image repos. Some may say “That’s not DRY Matt!”, to which I say “Suck a lemon!”. This code will never change and know that you can cd images/blah && ./build.sh really makes it easy and convenient to work with these images. Now we run the script and out pops a baby Docker :)

PWD: ~/code/modern-jenkins

cd images/jenkins-base
chmod +x ./build.sh # Gotta set executable perms
./build.sh
# ...
# profit!

yey! You’ve built your first Docker image (for this project)!

Testing the image

We can now go ahead and give this image a quick spin. It won’t be too exciting, but we can probably run the standard test to see that Java is installed:

PWD: ~/code/modern-jenkins

# Run the container and pop yourself into a shell
docker run --rm -ti modernjenkins/jenkins-base bash

# Check for java
java --version
# damn
java --help
# ugh
java version
# wtf! oh right...
java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

Commit and push

OK, so now that we have a nice working image, this seems like a perfect place to call shippable increment. Let’s commit to our branch, push it to origin, clean up any erroneous commits by squashing. , and create a PR. We will then self review it, confirm everything looks up to snuff, and merge. Then a nice little git pull should get us all up to date locally and we can begin work on the next increment.

PWD: ~/code/modern-jenkins/

git checkout -b feat-add_jenkins-base_image
git add .
git commit -m "Add a base image containing OpenJDK 8"
git push origin feat-add_jenkins-base_image

Moving on

Now we’ll begin to build on top of our base images. If you need to see the repo’s state at the end of this section, please rever to the unit2-part2 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part2

Next Post: Building the Jenkins Master Docker image

Modern Jenkins Unit 2 / Part 2: Building the Base Jenkins Image (and Intro to Docker) was originally published by Matt Bajor at CI/CD Life on July 24, 2017.

Modern Jenkins Unit 2 / Part 1: GitHub + GitHub Flow

2017-07-23T00:20:00-04:00

Storage + Process

Now that we have a good idea of what the desired traits and abilities of our system should be, we can begin to lay down the foundation. The very beginnings of all projects start with a SCM (source control management) repository.

Multi or single repo?

We want our repo to be intuitively laid out as there is the chance that almost every engineer in the company will need to interact with it at some point. This means that we should separate concerns within the tree and make things easy to find. In the past we tended to use a lot of separate repos and to treat them as if they were fully independent and could be shared with any other team as is. I am a fan of this idea for certain use cases, but what I have found is that it makes things very hard to find on a system that you contribute to infrequently. I prefer instead to start out with a mono-repo and if there comes a time in which we need to break it up, do it then. Making a bunch of different repos for a project like this right off the bat is a premature optimization IMO.

NOTE:

I’m going to treat this as a tutorial so you should be able to follow along by running what’s in the code blocks

Flesh out the repo structure

First let’s make sure that we have a repo to work on setup in GitHub. You should be able to do this for free as GitHub allows unlimited open source projects. Once we get to the point of configuring our deployment, we will add some encryption to cover the secret data.

Create the repo and populate the master branch

Create an account on GitHub ¹ if you do not have one already. GitHub is the worlds largest hub for sharing code and lots of companies use it as their primary SCM host with either their public or on-premise offering. We will be storing everything we do in this tutorial on GitHub. You can sign up for free (no credit card needed) here: https://github.com/join

https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/

and the creating your first public repo:

https://help.github.com/articles/create-a-repo/

NOTE: Don’t check the “Initialize this repo with a README” button.

You can name it anything you want, but I have named mine modern-jenkins. Clone this somewhere locally (I use ~/code for all of my repos) and let’s begin initializing the repo. I prefer to start with an empty commit when creating a new repo as we can then PR everything that ever hits the master branch. You can do this like so:

# Setup your Git author information first:
git config --global user.email matt@notevenremotelydorky.com
git config --global user.name "Matt Bajor"

# Navigate to your code directory
cd ~/code

# I have created a Github project here:
# https://github.com/technolo-g/modern-jenkins
git clone git@github.com:technolo-g/modern-jenkins.git
cd modern-jenkins

# Populate the master branch. This will be our first and ONLY commit directly to
# master. Everything else will go through a Pull Request
git commit --allow-empty -m "Initial Commit"
git push origin master

Begin with our GitHub Flow workflow

https://guides.github.com/introduction/flow/

We will be using GitHub Flow ² to work with our repo. Visit the link if you want to get a good idea of how it works, but the gist of it is that the master branch is where stable code lives. It should always be releasable. When we want to make a change, we will create a feature branch, make changes, then Pull Request ³ the changes back into master. Since the branches only live until the change is merged into master, they are considered to be “short lived”. There are some other workflows such as Git Flow ⁴ that work differently with “long lived” branches and other workflows, but they can lead to more of a headache with large, long running projects in my experience.

Our workflow will be:

Cut a feature branch off of master
Make changes to the branch to implement the feature
Push your changes to the remote (GitHub)
Create a Pull Request to merge your changes into master
(self-)Review the changes and comment and/or approve
Merge changes into master and delete the feature branch
Pull the new master locally

Create repo skeleton

Let’s begin creating our repo!

PWD: ~/code/modern-jenkins/

# Starting from our empty master, checkout a feature branch
git checkout -b feat-repo_skeleton

# Create a base readme
echo '# Dockerized Jenkins Build System' > README.md

# Create a place for our images
mkdir -p images/ && echo '# Docker Images' > images/README.md

# Create a place for our deployment config
mkdir -p deploy/ && echo '# Deployment Scripts' > deploy/README.md

# A spot for the DSL
mkdir -p dsl/ && echo '# Jenkins Job DSL' > dsl/README.md

Add a PULL_REQUEST_TEMPLATE

Whenever we change the master branch we will be submitting a Pull Request against the repo. GitHub gives you the ability to template the contents of the Pull Request form. Since we’re probably working by ourselves here, you may wonder “WTH Matt, why are we setting up collaboration tools when I’m clearly doing this tutorial by myself?” There are a couple of reasons:

Get used to the process: As the system evolves and as it grows there will be more and more people adding to this repo. If we start off pushing to master it’s much easier to continue that tradition and end up with everyone pushing to master all willy nilly. If we start off with a robust process, it has a habit of sticking around.
Fighting complacency: Since we’ll be self-reviewing most of this code, it can be really easy to just click ‘Merge’ without thinking of what you’re doing. This PR template has a small checklist that will keep you honest with yourself. If you know you have a tendency to skip over something go go ahead and add that to the checklist too.
Change management: Going from working app to working app requires keeping an eye on what’s changing whenever they change. When things do go awry (and they will), PRs will help untangle the mess much quicker than a steady stream of commits to the master branch. In theory, it is much easier to tell when a PR breaks a repo instead of a single commit in the history.
Thinking about your work in chunks: We really will be adding just a set of commits to a repo to get to our final form, but if we treat our work like that it’s never done. Instead, we should think about small chunks of work that bring value and can be deployed as a whole. Agile calls this a releasable increment. These chunks should make it easier to reason about what impact the change may have.

Even if I haven’t convinced you that this is important, I’m going to put it in a code block which will force your hand anyways. Ha!

PULL_REQUEST_TEMPLATE.md

cat <<EOF > PULL_REQUEST_TEMPLATE.md
#### What does this PR do?

#### Why did you take this approach?

#### Contribution checklist

- [ ] The branch is named something meaningful
- [ ] The branch is rebased off of current master
- [ ] There is a single commit (or very few smaller ones) with a [Good commit message](https://github.com/torvalds/subsurface-for-dirk/blob/master/README#L92) that includes the issue if there was one
- [ ] You have tested this locally yourself


#### A picture of a cute animal (optional)
<img src="https://68.media.tumblr.com/7b36a31855ed619f91b8fc4416d0cafc/tumblr_inline_o6b4ngEE551sdwbtb_540.png" width="350"/>
EOF

Integrating our changes into master

Now that we have a layout we like and a fancy new PR template, let’s get them into the master branch. We will do this via PR.

PWD: ~/code/modern-jenkins/

# Add the changes to our branch
git add .
git commit -m "Create initial repo structure and boilerplate"
git push origin feat-repo_skeleton

Now that the changes are pushed up, browse to your repo in your favorite web browser. Mine is located at https://github.com/technolo-g/modern-jenkins. You should see a green button that says “Compare & Pull Request”:

Click that and it will take you to a form that allows you to set the title of the PR as well as a description. Enter both and click “Create Pull Request”. Feel free to describe exactly what you’re doing and why. It’ good practice ;)

Once it has been submitted, you can then view the changeset, comment on it if you like (don’t feel bad talking to yourself), and approve or modify the PR. Since this guy is pretty simple, take a quick look through your commit and make sure there are no typos etc. If all looks good, give er’ a ‘LGTM’ in the comments section and merge away.

NOTE: I always recommend disabling merge commits in a repo’s settings. These just muddy up the commit history and instead I prefer to use the “Squash Merging” setting instead. This will squash all commits in the PR down to one and will allow you to edit the commit message before doing so. This really makes rebases and other git surgery easier than when there are a ton of merge commits to wade through. You can also do this before creating the PR if you like. See my post here on cleaning up your git history.

Sweet! You have made your first change on a road of many and you have done it in a very sustainable way that gives everyone working on the project context of what you’re doing and why. Most importantly, you’ve given your future self some really good reminders as to what you were thinking at the time the change was made.

On to the next chapter!

Now that we have our skeleton there, let’s begin hashing out the actual stuff in the repo :) If you need it, the code from this part can be found under the unit2-part1 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part1

Next Post: Building Jenkins’ base Docker image (and brief Intro to Docker)

Modern Jenkins Unit 2 / Part 1: GitHub + GitHub Flow was originally published by Matt Bajor at CI/CD Life on July 23, 2017.

Modern Jenkins Unit 1 / Part 2: Architecting a Jenkins + Docker CI System

2017-07-22T00:20:00-04:00

Defining the architecture requirements

We have defined the characteristics of our system so we know how we want it to behave. In order to obtain that behavior though, we need to determine what architectural features will get us there. Let’s go through our business requirements one by one and come up with some technical requirements to guide us through the project.

Easy to develop

We need a way to make changes to the system in order to move it forward that is easy to work with and as simple as possible. Developing in production may seem easy at first, but over time this creates a lack of trust in the system as well as a lot of downtime. Instead, let’s implement these technical features to make changing this system sustainable:

New developers can get going by cloning the repo and running a command
Everything can be run locally for development purposes
Resetting to a known good state is quick and easy

Getting started workflow

PWD: ~/code

# Ok, let's get you setup FNG
git clone git@github.com:technolo-g/modern-jenkins.git

cd modern-jenkins/deploy/master/
./start.sh

# You should be all set! You can begin development now.

Safe to develop

The nature of a CI system is to verify code exhibits the qualities we want and expect it to then deliver it production. This makes it inherently dangerous as it is extremely powerful and programmed to deliver software to production. We need the ability to modify this system without interfering with production by accident. To accomplish this we will implement:

Splitting development and production credentials
Disabling potentially dangerous actions in development
Explicitly working with forks when possible

Consistently deployable

This system, when deployed to production, will be used by a large portion of your development team. What this means is that downtime in the build system == highly paid engineers playing chair hockey. We want our system to always be deployable in case there is an issue with the running instance and also to be able to quickly roll back if there is an issue with a new version or the like. To make this happen we will need:

A versioning system that keeps track of the last known good deployment
State persistence across deploys
A roll-forward process
A roll-back process

Easy to upgrade

The nature of software development involves large amounts of rapid change. Within our system this mostly consists of plugin updates, Jenkins war updates, and system level package updates. The smaller the change during an upgrade, the easier it is to pinpoint and fix any problems that occur and the way to do that is frequent upgrades. To make this a no-brainer we will:

Build a new image on every change, pulling in new updates each time
Automatically resolve plugin dependencies
Smoke test upgrades before deployment

Programmatically configurable

Hand configuring complex systems never leads to the same system twice. There are lots of intricacies of the config including: order of operations, getting the proper values, dealing with secrets, and wiring up many independent systems. Keeping track of all this in a spreadsheet that we then manually enter into the running instance will get it done, but becomes pure nightmare mode after the first person is done working on it. Luckily with Jenkins’ built in Groovy init system ¹, we will be able to configure Jenkins with code in a much more deterministic way. Most of this functionality is built in, but we will still need:

A process to develop, test, and deploy our Groovy configuration
A mechanism to deploy it along with the master
The ability to share these scripts across environments

Immutable

With a consistently deployable, programmatically configured system it is almost as easy to redeploy instead of modifying the running instance. It is indeed easier to just click a checkbox in the GUI to change a setting the first time you do it, but the second, third, and fourth times become increasingly more difficult to remember to go check that box. For this reason, we are NEVER* going to mutate our running instance. We will instead just deploy a new version of our instance. This will only be a process change as the architectural features listed above will enable this process. We should however, document this process so that it will be followed by all.

Documentation of how to get changes into production

100% in SCM

Everything we’re doing is ‘as code’ so we will store it in the source repository ‘as code’. This will include master configuration, secrets, job configuration, infrastructure provisioning and configuration, as well as anything else the system needs to operate. To support this we’ll need:

A SCM repo (we’re using a Git repo hosted by GitHub)
A layout that supports: - Docker containers - Deployment tooling - Job DSL - Secrets
A mechanism for encrypting secrets in the repo

Git Repo Diagram

.
├── README.md
├── ansible
│   ├── playbook.yml
│   └── roles
├── deploy
│   ├── master
│   │   └── docker-compose.yml
│   └── slaves
├── images
│   ├── image1
│   │   └── Dockerfile
│   └── image2
│       └── Dockerfile
└── secure
    ├── secret1
        └── secret2

Secure

This pipeline can be considered the production line at a factory. It is where the individual components of your application are built, tested, assembled, packaged, and deployed to your customers. Any system that does the above mentioned processes should be heavily locked down as it is integral to the functioning of the company. Specifically we should:

Keep secrets secure and unexposed
Segregate actions by user (both machine and local)
Create an audit trail for any changes to the production system
Implement least-privilege access where necessary
Apply patches as soon as is feasible

Scalable

If scalability is thought of from day 0 on a project it is much easier to implement than if it is bolted on later. These systems have a tend to grow over time and this growth should be able to happen freely. This is not always an easy task, but we will attempt to make it easier by:

Keeping individual components de-coupled for independent scaling
Not locking ourselves into a single platform or provider
Monitoring the system to point out bottlenecks

Scaling to infinity… and beyond!

Now let’s start making some stuff!

Next Post: Code Repository and Development Lifecycle for a Jenkins + Docker CI/CD System

https://wiki.jenkins.io/display/JENKINS/Post-initialization+script ↩

Modern Jenkins Unit 1 / Part 2: Architecting a Jenkins + Docker CI System was originally published by Matt Bajor at CI/CD Life on July 22, 2017.

Modern Jenkins Unit 1 / Part 1: Planning a Jenkins + Docker CI System

2017-07-21T00:20:00-04:00

What are we doing here?

I have been introduced to far more installations of Jenkins that suck than don’t. One of the issues with a system of this level of complexity that is shared between many teams is that they tend to grow organically over time. Without much “oversight” by a person or team with a holistic view of the system and a plan for growth, these build systems can turn into spaghetti infrastructure the same way a legacy codebase can turn into spaghetti code.

Eventually failures end up evenly dividing themselves amongst infrastructure, bad commits, and flaky tests. Once this happens and trust is eroded, the only plausible fix for most consumers is to re-run the build. This lack of trust really makes it less fun to develop quality, reliable software. Its very important to have trust that the systems you are using to verify your work and not the other way around. An error in a build should unambiguously point to the code, not the system itself.

While not all companies can afford to have a designated CI team supporting the build infrastructure, it is possible to design the system initially (or refactor) in a way that encourages good practice and stability thus elevating developers’ confidence in the system and speed with which they can deliver new features. This series of posts will hopefully be able to get you started in the right direction when having to build or refactor a CI / CD system using Jenkins.

Describing the characteristics of the end state

I am a big fan of Agile software development¹ myself. I don’t believe that there is one kind of ‘Agile’ that works for everyone or anything like that, but I do believe 100% in the iterative approach agile takes to solve complex problems. Breaking work into small deliverables, and using those chunks for planning at multiple intervals such as:

Yearly: Very high level direction
Quarterly: General feature requirements
Sprint: Feature implementation details

When you have multiple perspectives on the scope and movement of a project, it really gives you the ability to manage expectations and make the most of your time. Since you have a general idea of what the long term goals are, you (ideally) can then make trade-off decisions with accurate values on the scales. This leads to less rewrites and the ability to put a bit more into the code you write because you know it will become legacy and you know Future You will appreciate the effort.

This is perhaps in opposition to the idea of hacking your way to a solution which is also valuable process, but more for a problem with an undefined domain. Luckily shipping software has most of problem domain defined so we’re able to set long, medium, and short time goals for our CI / CD infrastructure.

Anyways, let’s state some of the properties we want from our build system:

Easy to develop: A common problem with complex systems is that the process of setting up a local development environment is just as complex, if not more so. We will be developing just about the entire system locally on our laptop so we should hopefully get this for minimal cost.
Safe to develop: Another property that goes hand in hand with easy to develop is safe to develop. If our local environment is always reaching out to prod to comment on PRs or perhaps pushing branches and sending Slack notifications, it can be misleading and confusing to figure out where exactly these things are coming from. We will attempt to null out any non-local side effects in our development environment to provide a safe place to work.
Consistently deployable: Hitting the button and getting an error during a deployment is one of the worst feelings ever. In most cases when we need to deploy, we need to deploy right now! While building this project we will always keep master releasable and work with our automation in a way that no matter when we need to deploy, it will go.
Easy to upgrade: The best way to mitigate dangerous upgrades is to do them more frequently. While this may sound counterintuitive, it works for a couple of reasons:
1. The delta is smaller the more frequently you upgrade (and deploy!)
2. Each time you upgrade and deploy, you get more practice at it
3. If you fix a small error each upgrade, eventually most errors are fixed :) As we will see, what enables easy upgrades is a rollback safety net and a good way to verify the changes are safe. Since we’ll never be able to catch all the bugs, having the rollback as a backup tool is clutch.
Programmatically configurable: This is a giant one. Humans are really terrible at doing or creating the same thing over and over. This is way worse once you have a group of humans trying to do the same thing over and over (like creating jobs or configuring the master). Since we cannot be trusted to click the right buttons in the right sequence, we need to make sure the system does not depend on us doing that. We will cover a variety of tools to configure our system including the Netflix Job DSL, Groovy configuration via the Jenkins Init system, and later on Jenkinsfiles. There should be nothing that is hand jammed!
Immutable: “Immutable Infrastructure” ² is a term that was coined by Chad Fowler back in 2013. He proposes that the benefits of immutability in software architecture (like what is brought by functional programming) would also apply to infrastructure. What this translates to is that instead of mutating a running server by applying updates, rebooting, and changing configuration on running machines, when changes need to be made we should just redeploy a new set of servers with the updated code instead. This makes it much easier to reason about what state a system is in. You will see that this is one of the main characteristics that makes this Jenkins infrastructure different (and I think much better) than a legacy installation.
100% in SCM: Since we are committing to programmatic configuration, it is crucial that we keep everything within source control management. We should apply the same rigor with our CI / CD system’s codebase that we do with the one it is testing. PRs and automated tests should be the norm with anything you are creating, even if it is a one man show (IMHO).
Secure: Security is constantly an afterthought which is a dangerous way to work in a build system. These systems are so complex and so critical to the company (they actually assemble and package the product) that we MUST make them secure by default. Sane secrets management, least access privileges, immutable systems, and forcing an audit trail all lead to a more secure and healthy environment.
Scalable: Build systems at successful companies only grow, they do not shrink. Teams are constantly adding more code, more tests, more jobs, and more artifacts. Normally this occurs in a polyglot environment leading to exponential growth of requirements on the machines that are actually doing the builds. It does not scale to have a single server with every single requirement installed. This pattern soon becomes unwieldly and hard to work with. We should containerize everything and separate the infrastructure from the system running on top of it to allow independent and customized scaling and maintenance.

If we can build a system that meets these requirements, we will get a lot of stuff for free as well including:

Easy disaster recovery
Portability to different service providers
Fun working environment
High levels of productivity
…
Profit!

An iterative approach

Each iteration will build a shippable increment on top of the iteration before it and eventually we will have a production Jenkins! Along the way we will learn a bunch of stuff, build some Docker containers, write a compose file, automate some configuration, get real groovy, and much, much more. I encourage you to stay tuned and work through solving this problem with me.

All code will be published to this blog’s git repo ³ so that you can verify your answers and fix inconsistencies if you get stuck along the way. The repo should be tagged to match up with this series.

Now that we have a general idea of how we want our system to behave and how we will build it out, let’s dig into the architectural concerns.

Next Post: Architecting a Jenkins + Docker CI System

Modern Jenkins Unit 1 / Part 1: Planning a Jenkins + Docker CI System was originally published by Matt Bajor at CI/CD Life on July 21, 2017.

Intro to Docker Swarm: Part 4 - Demo

2015-01-29T00:00:00-05:00

Vagrant up up and away!

The primary output of my hackweek endeavours was a Docker Swarm cluster in a Vagrant environment. This post will go over how to get it spun up and then how to interact with it.

What is it?

This is a fully functional Docker Swarm cluster contained within a Vagrant environment. The environment consists of 4 nodes:

dockerhost01
dockerhost02
dockerhost03
dockerswarm01

The Docker nodes (dockerhost01-3) are running the Docker daemon as well as a couple of supporting services. The main processes of interest on the Docker hosts are:

Docker daemon: Running with a set of tags
Registrator daemon: This daemon connects to Consul in order to register and de-register containers that have their ports exposed. The entries from this service can be seen under the /services path in Consul’s key/value store
Swarm client: The Swarm client is what maintains the list of Swarm nodes in Consul. This list is kept under /swarm and contains a list of <ip>:<ports> of the Swarm nodes participating in the cluster

The Docker Swarm node (dockerswarm01) is also running a few services. Since this is just an example a lot of services have been condensed into a single machine. for production, I would not recommend this exact layout.

Swarm daemon: Acting as master and listening on the network for Docker commands while proxying them to the Docker hosts
Consul: A single node Consul instance is running. It’s UI is available at http://dockerswarm01/ui/#/test/
Nginx: Proxying to Consul for the UI

How to provision the cluster

1. Setup pre-requirements

The GitHub Repo: https://github.com/technolo-g/docker-swarm-demo
Vagrant (latest): https://www.vagrantup.com/downloads.html
Vagrant hosts plugin: vagrant plugin install vagrant-hosts
VirtualBox: https://www.virtualbox.org/wiki/Downloads
Ansible: brew install ansible
Host entries: Add the following lines to /etc/hosts:

10.100.199.200 dockerswarm01
10.100.199.201 dockerhost01
10.100.199.202 dockerhost02
10.100.199.203 dockerhost03

2a. Clone && Vagrant up (No TLS)

This process may take a while and will download a few gigs of data. In this case we are not using any TLS. If you want to use TLS with Swarm, go to 2b.

# Clone our repo
git clone https://github.com/technolo-g/docker-swarm-demo.git
cd docker-swarm-demo

# Bring up the cluster with Vagrant
vagrant up

# Provision the host files on the vagrant hosts
vagrant provision --provision-with hosts

# Activate your enviornment
source bin/env

2b. Clone && Vagrant up (With TLS)

This will generate certificates and bring up the cluster with TLS enabled.

# Clone our repo
git clone https://github.com/technolo-g/docker-swarm-demo.git
cd docker-swarm-demo

# Generate Certs
./bin/gen_ssl.sh

# Enable TLS for the cluster
echo -e "use_tls: True\ndocker_port: 2376" > ansible/group_vars/all.yml

# Bring up the cluster with Vagrant
vagrant up

# Provision the host files on the vagrant hosts
vagrant provision --provision-with hosts

# Activate your TLS enabled enviornment
source bin/env_tls

3. Confirm it’s working

Now the cluster is provisioned and running, you should be able to confirm it. We’ll do that a few ways. First lets take a look with the Docker client:

$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.4
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: swarm/0.0.1
Server API version: 1.16
Go version (server): go1.2.1
Git commit (server): n/a

$ docker info
Containers: 0
Nodes: 3
 dockerhost02: 10.100.199.202:2376
 dockerhost01: 10.100.199.201:2376
 dockerhost03: 10.100.199.203:2376

Now browse to Consul at http://dockerswarm01/ui/#/test/kv/swarm/ and confirm that the Docker hosts are listed with their proper port like so:

The cluster seems to be alive, so let’s provision a (fake) app to it!

How to use it

You can now interact with the Swarm cluster to provision nodes. The images in this demo have been pulled down during the Vagrant provision so these commands should work in order to spin up 2x external proxy containers and 3x internal webapp containers. Two things to note about the commands:

The constraints need to match tags that were assigned when Docker was started. This is how Swarm’s filter knows what Docker hosts are available for scheduling.
The SERVICE_NAME variable is set for Registrator. Since we are using a generic container (nginx) we are instead specifying the service name in this manner.

# Primary load balancer
docker run -d \
  -e constraint:zone==external \
  -e constraint:status==master \
  -e SERVICE_NAME=proxy \
  -p 80:80 \
  nginx:latest

# Secondary load balancer
docker run -d \
  -e constraint:zone==external \
  -e constraint:status==non-master \
  -e SERVICE_NAME=proxy \
  -p 80:80 \
  nginx:latest

# 3 Instances of the webapp
docker run -d \
  -e constraint:zone==internal \
  -e SERVICE_NAME=webapp \
  -p 80 \
  nginx:latest

docker run -d \
  -e constraint:zone==internal \
  -e SERVICE_NAME=webapp \
  -p 80 \
  nginx:latest

docker run -d \
  -e constraint:zone==internal \
  -e SERVICE_NAME=webapp \
  -p 80 \
  nginx:latest

Now if you do a docker ps or browse to Consul here:

http://dockerswarm01/ui/#/test/kv/services/

You can see the two services registered! Since the routing and service discovery part is extra credit, this app will not actually work but I think you get the idea.

I hope you have enjoyed this series on Docker Swarm. What I have discovered is Docker Swarm is a very promising application developed by a very fast moving team of great developers. I believe that it will change the way we treat our Docker hosts and will simplify things greatly when running complex applications.

All of the research behind these blog posts was made possible due to the awesome company I work for: Rally Software in Boulder, CO. We get at least 1 hack week per quarter and it enables us to hack on awesome things like Docker Swarm. If you would like to cut to the chase and directly start playing with a Vagrant example, here is the repo that is the output of my Q1 2014 hack week efforts:

https://github.com/technolo-g/docker-swarm-demo

Intro to Docker Swarm: Part 4 - Demo was originally published by Matt Bajor at CI/CD Life on January 29, 2015.

Intro to Docker Swarm: Part 3 - Example Swarm SOA

2015-01-22T00:00:00-05:00

A Docker Swarm SOA

One of the most exciting things that Docker Swarm brings to the table is the ability to create modern, resilient, and flexible architectures with very little overhead. Being able to interact with a heterogenious cluster of Docker hosts as if it were a single host enables the existing toolchains in use today to build everything we need to create a beautifully simple SOA!

This article is going to attempt to describe a full SOA architecture built around Docker Swarm that has the following properties:

A hypervisor layer composed of individual Docker hosts (Docker/Registrator)
A cluster layer tying the Docker hosts together (Docker Swarm)
A service discovery layer (Consul)
A routing layer to direct traffic based off of the services in Consul (HAProxy / Nginx)

Hypervisor Layer

The hypervisor layer is made up of a group of discrete Docker hosts. Each host has the services running on it that allows it to participate in the cluster:

Docker daemon: The Docker daemon is configured to listen on the network port in addition to the local Linux socket so that the Swarm daemon can communicate with it. In addition, each Dockerhost is configured to run with a set of tags that work with Swarm’s scheduler to define where containers are placed. They help describe the Docker host and is where any identifying information can be associated with the Docker host. This is an example of a set of tags a Docker host would be started with:
- zone: application/database
- disk: ssd/hdd
- env: dev/prod
Swarm daemon: The Swarm client daemon is run alongside the Docker daemon in order to keep the node in the Swarm cluster. This Swarm daemon is running in join mode and basically heartbeats to Consul to keep its record updated in the /swarm location. This record is what the Swarm master uses to create the cluster. If the daemon were to die the list in Consul should be updated to automatically to remove the node. The Swarm client daemon would use a path in Consul like /swarm and it would contain a list of the docker hosts:
Registrator daemon: The Registrator app¹ is what will be updating Consul when a container is created or destroyed. It listens to the Docker socket and upon each event will update the Consul key/value store. For example, an app named deepthought that requires 3 instances on separate hosts and that is running on port 80 would create a structure in Consul like this:

The pattern being:

/services/<service>-<port>/<dhost>:<cname>:<cport> value: <ipaddress>:<cport>
- service: The name of the container’s image
- port: The container’s exposed port
- dhost: The Docker host that the container is running on
- cport: The Container’s exposed port
- ipaddress: The ipaddress of the Docker host running the container
The output of a docker ps for the above service looks like so:

  $ docker ps
  CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                                   NAMES
  097e142c1263        mbajor/deepthought:latest   "nginx -g 'daemon of   17 seconds ago      Up 13 seconds       10.100.199.203:49166->80/tcp   dockerhost03/grave_goldstine
  1f7f3bb944cc        mbajor/deepthought:latest   "nginx -g 'daemon of   18 seconds ago      Up 14 seconds       10.100.199.201:49164->80/tcp   dockerhost01/determined_hypatia
  127641ff7d37        mbajor/deepthought:latest   "nginx -g 'daemon of   20 seconds ago      Up 16 seconds       10.100.199.202:49158->80/tcp   dockerhost02/thirsty_babbage

This is the most basic way to record the services and locations. Registrator also supports passing metadata along with the container that includes key information about the service².

Another thing to mention is that it seems the author of Registrator intends the daemon to be run as a Docker container. Since a Docker Swarm cluster is meant to be treated as a single Docker host, I prefer the idea of running the Registrator app as a daemon on the Docker hosts themselves. This allows a state on the cluster in which 0 containers are running and the cluster is still alive. It seems like a very appropriate place to draw the line between platform and applications.

Cluster Layer

At this layer we have the Docker Swarm master running. It is configured to read from Consul’s key/value store under the /swarm prefix and it generates its list of nodes from that information. It also is what listens for client connections to Docker (create, delete, etc..) and routes those requests to the proper backend Docker host. This means that it has the following requirements:

Listening on the network
Able to communicate with Consul
Able to communicate with all of the Docker daemons

As of yet I have yet to see mention of making the Swarm daemon itself HA, but after working with it there really do not seem to be any reasons that it could not be. I expect that a load balancing proxy with TCP support (HAproxy) could be put in front of a few Swarm daemons with relative ease. Sticky sessions would have to be enabled and possibly an active/passive if there are state synchronization issues between multiple Swarm daemons, but it seems like it would be doable. Since the containers do continue to run and are accessible even in the case of a Swarm failure we are going to accept the risk of a non-ha Swarm node over the complexity and overhead of loadbalancing the nodes. Tradeoffs right?

Service Discovery Layer

The service discovery layer is run on a cluster of Consul nodes; specifically it’s key/value store. In order to maintain quorum (n/2 + 1 nodes) even in the case of a failure there should be an odd number of nodes. Consul has a very large feature set³ including auto service discovery, health checking, and a key/value store to name a few. We are only using the key/value store, but I would expect there are benefits to incorporating the other aspects of Consul into your architecture. For this example configuration, the following processes are acting on the key/value store:

The Swarm clients on the Docker hosts will be registering themselves in /swarm
The Swarm master will be reading /swarm in order to build its list of Docker hosts
The Registrator daemon will be taking nodes in and out of the /services prefix
Consul-template will be reading the key/value store to generate the configs for the routing layer

This is the central datastore for all of the clustering metadata. Consul is what ties the containers on the Docker hosts to the entries in the routing backend.

Consul also has a GUI that can be installed in addition to everything else and I highly recommend installing it for development work. It makes figuring out what has been registered and where much easier. Once the cluster is up and running you may have no more need for it though

Routing Layer

This is the edge layer and what all external application traffic will run through. These nodes are on the edge of the Swarm cluster and are statically IP’d and have DNS entries that can be CNAME’d to for any services run on the cluster. These nodes listen on port 80/443 etc.. and have the following services running:

Consul-template: This daemon is polling Consul’s key/value store (under /services and when it detects a change, it writes a new HAProxy/Nginx config and gracefully reloads the service. The templates are written in Go templating and the output should be in standard HAProxy or Nginx form.
HAProxy or Nginx: Either of these servers are fully battle proven and ready for anything that is needed, even on the edge. The service is configured dynamically by Consul-template and reloaded when needed. The main change that happens frequently is the modification of a list of backends for a particular vhost. Since the list is maintained by what is actually alive and in Consul it changes as frequently as the containers do.

This is a high level overview of a Docker Swarm cluster that is based on an SOA. In the next post I will demonstrate a working infrastructure as described above in a Vagrant environment. This post will be coming after our Docker Denver Meetup⁴ so stay tuned (or better yet, come to the Meetup for the live demo)!

https://github.com/technolo-g/docker-swarm-demo

https://github.com/progrium/registrator ↩
https://github.com/progrium/registrator#single-service-with-metadata ↩
http://www.consul.io/docs/index.html ↩
http://www.meetup.com/Docker-Denver/events/218859311/ ↩

Intro to Docker Swarm: Part 3 - Example Swarm SOA was originally published by Matt Bajor at CI/CD Life on January 22, 2015.

Intro to Docker Swarm: Part 2 - Configuration Options and Requirements

2015-01-19T00:00:00-05:00

Minimum Requirements to run a Docker Swarm Cluster

The minimum requirements are minimal indeed to create a Docker Swarm cluster. In fact, it is definitely feasible (though perhaps not best practice) to run the Swarm daemon on an existing Docker Host making it possible to implement it without adding any more hardware or virtual resources. In addition, when running the file or nodes¹ based discovery mechanism there is no other infrastructure (besides of course Docker) that is required to run a basic Docker Swarm cluster.

I personally believe that spinning up another machine to run the Swarm master itself is a good idea. The machine does not have to be heavy in resources, but it does need to have a lot of file descriptors to handle all of the tcp connections coming and going. In the examples, I use dockerswarm01 as a dedicated Swarm master.

Configuration Options

There are a variety of configuration settings in Swarm that are sane by default, but give a lot of flexibility when it comes to running the daemon and its supporting infrastructure. Listed below are the different categories of config options and the options of how they can be configured.

Discovery

Discovery is the mechanism Swarm uses in order to maintain the status of the cluster. It can operate with a variety of backends, but it’s all pretty much the same concept:

The backend maintains a list of Docker nodes that should be part of the cluster.
Using the list of nodes, Swarm healtchecks each one and keeps track of the nodes that are in and out of the cluster

Node Discovery

Node discovery requires that everything be passed in on the command line. This is the most basic type of discovery mechanism as it requires no maintenance of config files or anything like that. An example startup command for the Swarm daemon using node discovery would look like:

swarm manage \
  --discovery dockerhost01:2375,dockerhost02:2375,dockerhost03:2375 \
  -H=0.0.0.0:2375

File Discovery

File discovery utilizes a configuration file placed on the filesystem (ie: /etc/swarm/cluster_config) with the format of <IP>:<Port> to list the Docker hosts in the cluster. Even though the list is static, healthchecking is used to determine the list of healthy and unhealthy nodes and filter requests going to the unhealthy nodes. An example of a file based discovery startup line and configuration file would be:

swarm manage \
  --discovery file:///etc/swarm/cluster_config \
  -H=0.0.0.0:2375

#/etc/swarm/cluster_config
dockerhost01:2375
dockerhost02:2375
dockerhost03:2375

Consul Discovery

Consul discovery is also supported out of the box by Docker Swarm. It works by utilizing Consul’s key value store to keep it’s list of <IP>:<Port>’s used to form the cluster. In this configuration mode, each Docker host runs a Swarm daemon in join mode that is pointed at the Consul cluster’s HTTP interface. This provides a little overhead to the configuration, runtime, and security of a Docker host, but not a significant amount. The Swarm client would be fired up as such:

swarm join \
  --discovery consul://consulhost01/swarm \
  # This can be an internal IP as long as the other
  # Docker hosts can reach it.
  --addr=10.100.199.200:2375

The Swarm master then reads it’s host list from Consul. It would be run with a startup line of:

swarm manage \
  --discovery consul://consulhost01/swarm \
  -H=0.0.0.0:2375

These key/value based configuration modes raise the question of how healthchecks within Swarm work in combination with the Swarm client in join mode. Since the list in key/value store is itself dynamic, is it required to run the internal Swarm healthchecks too? I’m not familiar with that area of functionality and so can’t speak to it but it’s worth noting.

EtcD Discovery

EtcD discovery works in much the same way as Consul discovery. Each Docker host in the cluster runs a Swarm daemon in join mode pointed at an EtcD endpoint. This provides a heartbeat to EtcD to maintain a list of active servers in the cluster. A Docker host running the standard Docker daemon would concurrently run a Swarm client with a configuration similar to:

swarm join \
  --discovery etcd://etcdhost01/swarm \
  --addr=10.100.199.200:2375

The Docker Swarm master would connect to EtcD, look at the path provided, and generate it’s list of nodes by starting with the following command:

swarm manage \
  --discovery etcd://etcdhost01/swarm \
  -H=0.0.0.0:2375

Zookeeper Discovery

Zookeeper discovery follows the same pattern as the other key/value store based configuration modes. A ZK ensemble is created to hold the host list information and a client runs alongside Docker in order to heartbeat in to the k/v store; maintaining the list in near real-time. The Swarm master is also connected to the ensemble and uses the information under /swarm to maintain its list of hosts (which it then healthchecks).

Swarm Client (alongside Docker):

swarm join \
  # All hosts in the ensemble should be listed
  --discovery zk://zkhost01,zkhost02,zkhost03/swarm \
  --addr=10.100.199.200

Swarm Master:

swarm manage \
  --discovery zk://zkhost01,zkhost02,zkhost03/swarm \
  -H 0.0.0.0:2375

Hosted Token Based Discovery (default)

I have not used this functionality and at this point have very little reason to.

Scheduling

Scheduling is the mechanism for choosing where a container should be created and started. It is made up of a combination of a packing algorithm and filters (or tags). Each Docker daemon is started with a set of tags like this:

docker -d \
  --label storage=ssd \
  --label zone=external \
  --label tier=data \
  -H tcp://0.0.0.0:2375

Then when a Docker container is started Swarm will choose a group of machines based on the filters, and then distributes each run command according to its scheduler. Filters tell Swarm where a container can and cannot run, while the scheduler places it amongst the available hosts. There are a few filtering mechanisms:

Constraint: This utilizes the tags that a Docker daemon was starting with. Currently it supports only ‘=’, but at some point in the future it may support ‘!=’. A node must match all of the constraints provided by a container in order to fit into scheduling. Starting a container with a few constraints would look like:

  docker run -d -P \
    -e constraint:storage=ssd \
    -e constraint:zone=external \
    -t nginx

Affinity: Affinity can work in two ways: affinity to containers or affinity to images. In order to start two containers on the same host the following command would be run:

  docker run -d -P \
    --name nginx \
    -t nginx

   docker run -d -P \
     --name mysql \
     -e affinity:container=nginx \
     -t mysql

Since Swarm does not handle image management, it is also possible to set affinity for an image. This means a container will only be started on a node that already contains the image. This negates the need to wait for an image to be pulled in the background before starting a container. An example:

  docker run -d -P \
    --name nginx \
    -e affinity:image=nginx \
    -t nginx

Port: The port filter will not allow any two containers with the same static port mapping to be started on the same host. This makes a lot of sense as you cannot duplicate a port mapping on a Dockerhost. For example, two nodes started with -p 80:80 will not be allowed to run on the same Dockerhost.
Healthy: This prevents the scheduling of containers on unhealthy nodes.

Once Swarm has narrowed the host list down to a set that matches the above filters, it then schedules the container on one of the nodes. Currently the following schedulers are built in:

Random: Randomly distribute containers across available backends.
Binpacking: Fill up a node with containers and then move to the next. This mode has the increased complexity of having to assign static resource amounts to each container at runtime. This means setting a limit on a container’s memory and cpu which may or may not seem OK. I personally like letting the containers fight amongst themselves to see who gets the resources.

In progress are the balanced strategy² and the ability to add Apache Mesos³.

TLS

I am happy to say that Swarm works with TLS enabled. This makes it more secure between both the client and Swarm daemon as well as between the Swarm daemon and the Docker daemons. This is good because my security guy says that there are no more borders in networks. Yey.

It does require a full PKI including CA, but I have this solved in another post already :) This is how to generate the required TLS certs for Docker and Swarm.

Once the certificates have been generated and installed as per my other blog post, the Docker and Swarm daemons can be fired up like this:

Docker:

docker -d \
  --tlsverify \
  --tlscacert=/etc/pki/tls/certs/ca.pem \
  --tlscert=/etc/pki/tls/certs/dockerhost01-cert.pem \
  --tlskey=/etc/pki/tls/private/dockerhost01-key.pem \
  -H tcp://0.0.0.0:2376

Swarm master:

swarm manage \
  --tlsverify \
  --tlscacert=/etc/pki/tls/certs/ca.pem \
  --tlscert=/etc/pki/tls/certs/swarm-cert.pem \
  --tlskey=/etc/pki/tls/private/swarm-key.pem  \
  --discovery file:///etc/swarm_config \
  -H tcp://0.0.0.0:2376

Then the client must know to connect via TLS. This is done with the following environment variables:

export DOCKER_HOST=tcp://dockerswarm01:2376
export DOCKER_CERT_PATH="`pwd`"
export DOCKER_TLS_VERIFY=1

You are now setup for TLS. WCGW?

More to come!

Well there is a lot to talk about when it comes to configuration of complex clustered software, but I feel this is a good enough overview to get you up and running and thinking about how to configure your Swarm cluster. In the next episode I’ll lay out some example architectures for your Swarm cluster. Stay tuned and please feel free to comment below!

https://github.com/technolo-g/docker-swarm-demo

https://github.com/docker/swarm/tree/master/discovery#using-a-static-list-of-ips ↩
https://github.com/docker/swarm/pull/227 ↩
https://github.com/docker/swarm/issues/214 ↩

Intro to Docker Swarm: Part 2 - Configuration Options and Requirements was originally published by Matt Bajor at CI/CD Life on January 19, 2015.

Intro to Docker Swarm: Part 1 - Overview

2015-01-18T00:00:00-05:00

What is Docker Swarm?

Docker Swarm¹ is a utility that is used to create a cluster of Docker hosts that can be interacted with as if it were a single host. I was introduced to it a few days before it was announced at DockerCon EU² at the Docker Global Hack Day³ that I participated in at work. During the introduction to hackday, a few really cool new technologies were announced⁴ including Docker Swarm, Docker Machine, and Docker Compose. Since Ansible fills the role of Machine and Compose, Swarm stuck out as particularly interesting to me.

Victor Vieux and Andrea Luzzardi announced the concept and demonstrated the basic workings of Swarm during the intros and made a statement that I found to be very interesting. They said that though the POC (proof of concept) was functional and able to demo, they were going to throw away all of that code and start from scratch. I thought that was great and try to keep that in mind when POC’ing a new technology.

The daemon is written in Go and at this point in time latest commit a0901ce8d6 is definitely Alpha software. Things are moving at a very rapid pace at this point in time and functionality + feature set vary almost daily. That being said, @vieux is extremely responsive with adding functionality and fixing bugs via GitHub Issues⁵. I would not recommend using it in production yet, but it is a very promising technology.

How does it work

Interacting with and operating Swarm is (by-design) very similar to dealing with a single Docker host. This allows interoperability with existing toolchains without having to make too many modifications (the major ones being splitting builds off of the Swarm cluster). Swarm is a daemon that is run on a Linux machine bound to a network interface on the same port that a standalone Docker instance (http/2375 or https/2376) would be. The Swarm daemon accepts connections from the standard Docker client >=1.4.0 and proxies them back to the Docker daemons configured behind Swarm which are also listening on the standard Docker ports. It can distribute the create commands based on a few different packing algorithms in combination with tags that the Docker daemons have been started with. This makes the creation of a partitioned cluster of heterogeneous Docker hosts that is exposed as a single Docker endpoint extremely simple.

Interacting with Swarm is ‘more or less’ the same as interacting with a single non-clustered Docker instance, but there are a few caveats. There is not 1-1 support for all Docker commands. This is due to both architectural and time based reasons. Some commands are just not implemented yet and I would imagine some might never be. Right now almost everything needed for running containers is available, including (amongst others):

docker run
docker create
docker inspect
docker kill
docker logs
docker start

This subset is the essential part of what is needed to begin playing with the tool in runtime. Here is an overview of how the technologies are used in the most basic configuration:

The Docker hosts are brought up with --label key=value listening on the network.
The Swarm daemon is brought up and pointed at a file containing a list of the Docker hosts that make up the cluster as well the ports they are listening on.
Swarm reaches out to each of the Docker hosts and determines their tags, health, and amount of resources in order to maintain a list of the backends and their metadata.
The client interacts with Swarm via it’s network port (2375). You interact with Swarm the same way you would with Docker: create, destroy, run, attach, and get logs of running containers amongst other things.
When a command is issued to Swarm, Swarm:
- decides where to route the command based off of the provided constraint tags, health of the backends, and the scheduling algorithm.
- executes the command against the proper Docker daemon
- returns the result in the same format as Docker does

The Swarm daemon itself is only a scheduler and a router. It does not actually run the containers itself meaning that if Swarm goes down, the containers it has provisioned are still up on the backend Docker hosts. In addition, since it doesn’t handle any of the network routing (network connections need to be routed directly to the backend Docker host) running containers will still be available even if the Swarm daemon dies. When Swarm recovers from such a crash, it is able to query the backends in order to rebuild its list of metadata.

Due to the design of Swarm, interaction with Swarm for all runtime activities is just about the same as it would be for other Docker daemon: the Docker client, docker-py, docker-api gem, etc.. Build commands have not yet been figured out, but you can get by for runtime today. Unfortunately at this exact time Ansible does not seem to work with Swarm in TLS mode⁶, but it appears to affect the Docker daemon itself not just Swarm.

This concludes the 1st post regarding Docker Swarm. I apologize for the lack of technical detail, but it will be coming in subsequent posts in the form of architectures, snippets, and some hands-on activities :) Look out for Part 2: Docker Swarm Configuration Options and Requirements coming soon!

https://github.com/technolo-g/docker-swarm-demo

https://github.com/docker/swarm ↩
http://blog.docker.com/2015/01/dockercon-eu-introducing-docker-swarm/ ↩
http://www.meetup.com/Docker-meetups/events/148163592/ ↩
http://blog.docker.com/2014/12/announcing-docker-machine-swarm-and-compose-for-orchestrating-distributed-apps/ ↩
https://github.com/docker/swarm/issues ↩
https://github.com/ansible/ansible/issues/10032 ↩

Intro to Docker Swarm: Part 1 - Overview was originally published by Matt Bajor at CI/CD Life on January 18, 2015.

Creating SSL/TLS Certificates for Docker and Docker Swarm

2015-01-18T00:00:00-05:00

Docker¹ has one of the most gentle learning slopes of a new technology to enter the mainstream in a long time. A developer can get up and running in a very short amount of time² and begin realizing value almost immediately with Docker, but the hard part comes when trying to secure the new technology for use in a production like environment. Production has a much higher standard when it comes to availability, security, and repeatability. This can lead to problems as the differences between the development and production environment are both:

Fairly complex to replicate in a secure way: It is not feasible to pass around the private key for your production certificate in the name of development environment automation. On the other hand, to generate a full CA and all of the certificates and keys required can be daunting.
Functionally quite a large delta: The difference between an insecure, non-tls environment and an SSL one can be significant. For instance, at this exact exact point in time it appears that Ansible does not yet support a TLS enabled Docker host³.

In order to attack this problem, we should attempt to replicate the prod environment when it is feasible and especially if it is easy and cheap. To this end, let’s create the full certificate chain needed to run a secure Docker Swarm⁴ cluster. I think you will find that it is both easy and cheap :)

The script

This is a bash script I used⁵ that will output everything within the directory it is run. It accomplishes the following things:

Creates a Certificate Authority
Creates a cert/key for Docker Swarm (supporting both client and server auth)
Generates 3 certificates for the individual Docker hosts with SAN IPs

It is required to set a config as we need to add a SAN IP Address entry to the certificate and CSR. This is required because without it, Swarm will spit out the following error:

ERRO[0282] Get https://10.100.199.201:2376/v1.15/info: x509: cannot validate certificate for 10.100.199.201 because it doesn't contain any IP SANs

Disclaimer: I am no SSL wizard and so some of the settings in the openssl.cnf may be insecure, not needed, or even both. In addition, you can see that this is totally insecure as

the password is in the script
the passwords are removed from the keys
many other reasons

Please don’t use these exact script or the generated certs for production use!

`gen_ssl.sh`

#!/bin/bash

export OPENSSL_CONF=openssl.cnf


echo 'Creating CA (ca-key.pem, ca.pem)'
echo 01 > ca.srl
openssl genrsa -des3 -passout pass:password -out ca-key.pem 2048
openssl req -new -passin pass:password \
        -subj '/CN=Non-Prod Test CA/C=US' \
        -x509 -days 365 -key ca-key.pem -out ca.pem


echo 'Creating Swarm certificates (swarm-key.pem, swarm-cert.pem)'
openssl genrsa -des3 -passout pass:password -out swarm-key.pem 2048
openssl req -passin pass:password -subj '/CN=dockerswarm01' -new -key swarm-key.pem -out swarm-client.csr
echo 'extendedKeyUsage = clientAuth,serverAuth' > extfile.cnf
openssl x509 -passin pass:password -req -days 365 -in swarm-client.csr -CA ca.pem -CAkey ca-key.pem -out swarm-cert.pem -extfile extfile.cnf
openssl rsa -passin pass:password -in swarm-key.pem -out swarm-key.pem

# Set the default keys to be Swarm
cp -rp swarm-key.pem key.pem
cp -rp swarm-cert.pem cert.pem

echo 'Creating host certificates (dockerhost01-3-key.pem, dockerhost01-3-cert.pem)'
openssl genrsa -passout pass:password -des3 -out dockerhost01-key.pem 2048
openssl req -passin pass:password -subj '/CN=dockerhost01' -new -key dockerhost01-key.pem -out dockerhost01.csr
openssl x509 -passin pass:password -req -days 365 -in dockerhost01.csr -CA ca.pem -CAkey ca-key.pem -out dockerhost01-cert.pem -extfile openssl.cnf
openssl rsa -passin pass:password -in dockerhost01-key.pem -out dockerhost01-key.pem

openssl genrsa -passout pass:password -des3 -out dockerhost02-key.pem 2048
openssl req -passin pass:password -subj '/CN=dockerhost02' -new -key dockerhost02-key.pem -out dockerhost02.csr
openssl x509 -passin pass:password -req -days 365 -in dockerhost02.csr -CA ca.pem -CAkey ca-key.pem -out dockerhost02-cert.pem -extfile openssl.cnf
openssl rsa -passin pass:password -in dockerhost02-key.pem -out dockerhost02-key.pem

openssl genrsa -passout pass:password -des3 -out dockerhost03-key.pem 2048
openssl req -passin pass:password -subj '/CN=dockerhost03' -new -key dockerhost03-key.pem -out dockerhost03.csr
openssl x509 -passin pass:password -req -days 365 -in dockerhost03.csr -CA ca.pem -CAkey ca-key.pem -out dockerhost03-cert.pem -extfile openssl.cnf
openssl rsa -passin pass:password -in dockerhost03-key.pem -out dockerhost03-key.pem

# We don't need the CSRs once the cert has been generated
rm -f *.csr

`openssl.cnf`

#
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
#

# This definition stops the following lines choking if HOME isn't
# defined.
HOME			= .
RANDFILE		= $ENV::HOME/.rnd
oid_section		= new_oids
extensions		= v3_req

[ new_oids ]
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

####################################################################
[ ca ]
default_ca	= CA_default		# The default ca section

####################################################################
[ CA_default ]
dir		= ./tls		# Where everything is kept
certs		= $dir/certs		# Where the issued certs are kept
crl_dir		= $dir/crl		# Where the issued crl are kept
database	= $dir/index.txt	# database index file.
new_certs_dir	= $dir/newcerts		# default place for new certs.
certificate	= $dir/cacert.pem 	# The CA certificate
serial		= $dir/serial 		# The current serial number
crlnumber	= $dir/crlnumber
crl		= $dir/crl.pem 		# The current CRL
private_key	= $dir/private/cakey.pem# The private key
RANDFILE	= $dir/private/.rand	# private random number file
x509_extensions	= usr_cert		# The extentions to add to the cert
name_opt 	= ca_default		# Subject Name options
cert_opt 	= ca_default		# Certificate field options
default_days	= 365			# how long to certify for
default_crl_days= 30			# how long before next CRL
default_md	= default		# use public key default MD
preserve	= no			# keep passed DN ordering
policy		= policy_match

[ policy_match ]
countryName		= match
stateOrProvinceName	= match
organizationName	= match
organizationalUnitName	= optional
commonName		= supplied
emailAddress		= optional

[ policy_anything ]
countryName		= optional
stateOrProvinceName	= optional
localityName		= optional
organizationName	= optional
organizationalUnitName	= optional
commonName		= supplied
emailAddress		= optional

####################################################################
[ req ]
default_bits		= 1024
default_keyfile 	= privkey.pem
distinguished_name	= req_distinguished_name
attributes		= req_attributes
x509_extensions	= v3_ca	# The extentions to add to the self signed cert
string_mask = utf8only
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName			= Country Name (2 letter code)
countryName_default		= AU
countryName_min			= 2
countryName_max			= 2
stateOrProvinceName		= State or Province Name (full name)
stateOrProvinceName_default	= Some-State
localityName			= Locality Name (eg, city)
0.organizationName		= Organization Name (eg, company)
0.organizationName_default	= Internet Widgits Pty Ltd
organizationalUnitName		= Organizational Unit Name (eg, section)
commonName			= Common Name (e.g. server FQDN or YOUR name)
commonName_max			= 64
emailAddress			= Email Address
emailAddress_max		= 64

[ req_attributes ]
challengePassword		= A challenge password
challengePassword_min		= 4
challengePassword_max		= 20
unstructuredName		= An optional company name

[ usr_cert ]
basicConstraints=CA:FALSE
nsComment			= "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ v3_ca ]
subjectAltName = @alt_names
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier=keyid:always

[ alt_names ]
# The IPs of the Docker and Swarm hosts
IP.1 = 10.100.199.200
IP.2 = 10.100.199.201
IP.3 = 10.100.199.202
IP.4 = 10.100.199.203

Installation

Once you have generated the TLS keys and certificates they must be installed on the target machine. I prefer to just copy the certificate and the key files into /etc/pki/tls/certs/ and /etc/pki/tls/private/ respectively. Once they are installed, you can then fire up your Docker and Swarm daemons like so:

Docker

/usr/bin/docker -d \
  --tlsverify \
  --tlscacert=/etc/pki/tls/certs/ca.pem \
  --tlscert=/etc/pki/tls/certs/dockerhost01-cert.pem \
  --tlskey=/etc/pki/tls/private/dockerhost01-key.pem \
  -H tcp://0.0.0.0:2376

Swarm

/usr/local/bin/swarm manage \
  --tlsverify \
  --tlscacert=/etc/pki/tls/certs/ca.pem \
  --tlscert=/etc/pki/tls/certs/swarm-cert.pem \
  --tlskey=/etc/pki/tls/private/swarm-key.pem  \
  --discovery file:///etc/swarm_config \
  -H tcp://0.0.0.0:2376

Using the TLS enabled Docker daemon

Now in order to use the Docker daemon, you will have to present a client cert that was generated from the same CA as the certificate Docker/Swarm is using. We have generated one here and more can be made if needed. Set the following environment variables in order to tell the Docker client what to use for the TLS config:

export DOCKER_HOST=tcp://dockerswarm01:2376
export DOCKER_CERT_PATH="`pwd`"
export DOCKER_TLS_VERIFY=1

This will now enable the Docker client to communicate ‘securely’ with Docker Swarm and Docker Swarm to communicate securely with the Docker nodes behind it.

https://www.docker.com ↩
http://goo.gl/QlZ5qv ↩
https://github.com/ansible/ansible/issues/10032 ↩
https://github.com/docker/swarm ↩
https://github.com/technolo-g/docker-swarm-demo/blob/master/bin/gen_ssl.sh ↩

Creating SSL/TLS Certificates for Docker and Docker Swarm was originally published by Matt Bajor at CI/CD Life on January 18, 2015.

Clean up your Git history

2015-01-17T00:00:00-05:00

We all like to keep our code looking as neat as possible, but sometimes you also need to keep track of those small changes you’ve been making. A good way to ABC (Always Be Committin’) is to work in a branch and hack your way through the problem and then to clean up before submitting a PR.

Cut a new branch

What we are about to do here can be destructive as well as confusing. With tasks like that, it’s always nice to have a backup and so we are going to cut a new branch to work with off of the branch that all of the nasty more changes live in:

git checkout -b squashed_feature

Rebase from master

This will give us a branch that we can then safely rebase from master. This process will allow you to pick the commits you would like to squash and the ones you would like to keep. You can do this by running:

git rebase -i master

Squash commits

If you have merged master into your branch during your development process you will be unable to use this method. I normally will rebase my branch on master instead of merging in, but workflows may vary. Once you begin the rebase process, your git editor will open a file that looks like:

pick 3e33836 Initial commit of consul-template role
pick 7a09b99 remove boilerplate
pick 935e5e4 Default openssl.cnf
pick 6277156 Add all ips as SAN
pick 8cc1e89 Add SAN IP to certificates
pick e5aa77c Have to repro with hosts
pick 9819551 Change from 5 to 3 dockerhosts to reduce time

# Rebase d33881f..9819551 onto d33881f
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

In this view the most recent commit will be at the top and the oldest at the bottom. To squash all of the commits into a single one, change all the ‘pick’s but one to ‘squash’. Note: ‘s’ will also work instead of ‘squash’

pick 3e33836 Initial commit of consul-template role
s 7a09b99 remove boilerplate
s 935e5e4 Default openssl.cnf
s 6277156 Add all ips as SAN
s 8cc1e89 Add SAN IP to certificates
s e5aa77c Have to repro with hosts
s 9819551 Change from 5 to 3 dockerhosts to reduce time

...

Rewrite

You want to squash these changes as if you were to remove the line itself, the actual commit will be removed from history:

# This is a combination of 7 commits.
# The first commit's message is:
Initial commit of consul-template role

# This is the 2nd commit message:

remove boilerplate

...

At this screen there is no need to change the commit messages. After saving and quitting the editor, a new file will open to allow you to edit the commit message. In that file, just remove all of the messages and replace them with the one that you want.

You can now push your branch up and open a PR!

git push origin squashed_feature

Source

Clean up your Git history was originally published by Matt Bajor at CI/CD Life on January 17, 2015.

CI/CD Life

Update from Matt

Brief update from Matt Bajor / Unauthorized

Containerizing local commands

Running yarn from a container

How can we run yarn in a container for a local project?

Step 1: Build a Docker image for yarn

Step 2: Volume mount the current directory into the Docker container

Step 3: Distribute the software

Step 4: ….

Step 5: Cleanup

Enhancements

build.sh

Versioning

Disclaimer!

Conclusion

Making HTTP GET and POST Requests to Docker Hub in a Jenkinsfile

API Calls

Explicit Versioning

Interacting with the Docker Hub API in a Jenkinsfile

Jenkinsfile

Script Security

Pre-authorizing Jenkins Signatures with Groovy

URL: http://localhost:8080/script

Modern Jenkins Unit 3 / Part 4: Configuring the GitHub Jenkins Plugin

Setting up the Jenkins <–> Git Interactions

Creating a machine user

PWD: ~/code/modern-jenkins/

Add the secrets to the credential store

Mount the secrets into the master container

deploy/master/docker-compose.yml

PWD: ~/code/modern-jenkins/deploy/master

Writing Groovy to configure GitHub

URL: http://localhost:8080/script

images/jenkins-plugins/files/init.groovy.d/02-configure-github-client.groovy

Add the groovy to the Docker image

PWD: ~/code/modern-jenkins/

Testing our changes

Commit, Push, PR

Modern Jenkins Unit 3 / Part 3: Managing Secrets in GitHub

Encrypting files in the repo with Transcrypt

PWD: ~/code/modern-jenkins/

Initialize the repo

PWD: ~/code/modern-jenkins/

PWD: ~/code/modern-jenkins/

PWD: ~/code/modern-jenkins/

./PULL_REQUEST_TEMPLATE.MD

Modern Jenkins Unit 3 / Part 2: Configure Jenkins URL

Configuring the Jenkins URL

Passing from Docker Compose

deploy/master/docker-compose.yml

PWD: ~/code/modern-jenkins

URL: http://localhost:8080/script

images/jenkins-plugins/files/init.groovy.d/01-cofigure-jenkins-url.groovy

Deploying our Groovy Jenkins Configs

PWD: ~/code/modern-jenkins/

images/jenkins-plugins/Dockerfile

PWD: ~/code/modern-jenkins

PWD: ~/code/modern-jenkins/deploy/master

PWD: ~/code/modern-jenkins/

Squirrel!

Let’s get that squirrel

deploy/master/start.sh

Modern Jenkins Unit 3 / Part 1: Intro to the Jenkins Groovy Init System

Configuring the Jenkins Master on Boot with Groovy

Intro to the Jenkins init system

The Groovy Script Console

Jenkins Groovy Hello World

URL: http://localhost:8080/script

Modern Jenkins Unit 2 / Part 5: Starting Jenkins with Docker Compose

“The Good Ole’ Days”

deploy/master/docker-compose.yml

Test it out

PWD: ~/code/modern-jenkins

deploy/README.md

WTF Matt, a 6 part blog series to replace “java -jar jenkins.war” and 6 clicks?

In conclusion

Modern Jenkins Unit 2 / Part 4: The Jenkins Plugin Image

The plugins image

images/jenkins-plugins/Dockerfile

Running `yarn` from a container

Step 1: Build a Docker image for `yarn`