moar articles
Tweet
Follow @nicferrier

docker blue/green deployment

see updates ...

Docker is the new hotness in Linux deployment. There's a lot of hype about it but it does have utility. In this article I want to show how docker and a simple bash script can achieve a quite advanced blue/green deployment pipeline.

what is blue/green deployment?

Here's what Martin Fowler says.

It's a description of deploying an app without disrupting the running of the existing app. It's a great method for webapps because it reduces downtime to a minimum.

The basic idea is that you run two instances of your whole app. One continues to run while you deploy to the other. When you have finished deploying (and that might include some tests) you can switch the live one over to the newly deployed version.

the app

Screenshot of GNU docapp

About 10 years ago someone asked me if we could make a better web based documentation viewer than what was spat out by GNU's makeinfo tool. I was sure that it could be done easily but at the time I didn't have the spare hours.

However, that stuff has got so much easier in the last 10 years. The history state API, reliable ajax and even multi column layouts all make this stuff much more possible.

In a few hours I was able to put together a simple demo of how this could be done using an Elnode proxy pulling a js based viewer and the online HTML of the GNU EmacsLisp manual.

So the app is a few functions of Elnode (nothing nodejs couldn't do) and some browerify'd JavaScript.

You can see the app here

deployment

I was able to really quickly make an app with Elnode and JavaScript. But how to deploy? I happen to have docker images for Elnode so I started to use docker to deploy.

Docker works well for this because it encapsulates the whole stack so well. Although I have to have something to do the blue/green switching, in this case nginx, that's so simple that I don't have to worry about errors.

Here's my nginx config:

server {
   server_name gnudoc.ferrier.me.uk;
   access_log /var/log/nginx/gnudoc-access.log;
   location / {
      proxy_pass http://localhost:8016;
      proxy_http_version 1.1;
   }
}

It really is trivial.

There is still a complicated deployment pipeline though:

This is still too much typing for me. So here's my effort at a BASH script to do this:

#!/bin/bash

# Simple bash deployment script for gnudoc
#
# Rebuilds the current docker, pushes it to the registry and then, on
# the remote: pulls and tries to start in a blue green way

# This is the script run on the remote machine
function remoteDeploy {
    # Pull the new dockers
    sudo docker.io pull nicferrier/elnode-gnudoc

    # What's the current deploy hosted on?
    local currentport=$(sudo sed -nre 's/.*(8[0-9]{3}).*/\1/p' /etc/nginx/sites-enabled/gnudoc.conf)

    # Pull out the IP/port of the dockers that are the correct image
    sudo docker.io ps -q |  while read dockid
    do 
        ( sudo docker.io inspect -f '{{ .Config.Image }}' $dockid \
            | grep nicferrier/elnode-gnudoc > /dev/null ) && echo $dockid
    done | while read dockid ;
    do
        echo "elnode-gnudoc $dockid docker found" > /dev/stderr
        # It's quite hard to access the keys of NetworkSettings.Ports
        printf "$dockid " ; sudo docker.io inspect $dockid \
            | jq -r '.[0] | .NetworkSettings.Ports | to_entries | map(select(.key == "8015/tcp")) | .[0] | .value | .[0] | .HostPort'
    done | while read dockid port 
    do
        echo "$dockid using $port" > /dev/stderr
        # Stop any elnode-gnudoc container that's running on a non-live port
        if [ "$currentport" != "$port" ] 
        then
            echo "killing $dockid because it's not on $CURRENTPORT" > /dev/stderr
            sudo docker.io kill $dockid
        fi
        echo $dockid $port
    done | while read dockid port
    do
        # When we get here we should only have the live docker running
        if [ "$currentport" == "$port" ]
        then
            local newport
            newport=$(expr $port + 1)
            echo "starting $dockid on $newport avoiding nginx on $CURRENTPORT" > /dev/stderr
            # Start the new docker
            sudo docker.io run -d -p $newport:8015 -t nicferrier/elnode-gnudoc # we could curl check here
            # Rewrite the nginx config
            sudo sed -ibk -re "s/$port/$newport/" /etc/nginx/sites-enabled/gnudoc.conf
            # Restart nginx
            sudo /etc/init.d/nginx reload
        fi
    done
}

cd $(dirname $0)
sudo docker build --no-cache -t nicferrier/elnode-gnudoc .
sudo docker push nicferrier/elnode-gnudoc

# Now the remote parts
( typeset -fp remoteDeploy ; echo remoteDeploy ) | ssh po5.ferrier.me.uk bash -

# deploy ends here

There are two interesting things here. First, controlling the remote machine is always a pain. Using a BASH function and then sending it as a script to the remote side for execution is something I'd not considered doing before:

function doStuffOnRemote {
    sudo /etc/init.d/nginx reload
}

( typeset -fp doStuffOnRemote ; echo doStuffOnRemote ) > /tmp/deployment-script

Anything we put in the function can be sent to the remote side and if we ust add a line to call the function at the end we have a usable script.

It seems a useful way to avoid more complicated ssh tools like Capistrano.

pipemills and fp

The second interesting thing about the script is how we can use functional programming style in that remote deploy step. The whiles are pipemills, they chain a series of transformations together. The sequence of mills above could be expressed in an imaginary fp notation like this:

->> list-dockers
   filter (docker) { return docker.image == "elnode-gnudoc" }
   map (docker) { 
     docker["port"] = docker.port
     return docker
   }
   map (docker) { 
     if docker.port == currentNginxPort {
        dockerStop currentNginxPort
     }
     return docker
   }
   map (docker) {
     if docker.port != currentNginxPort {
        dockerRun docker.port
        sed nginx.conf
        nginxReload
     }
   }

It makes me wonder again if pipemills should be supported in shells with some better abstraction then just the while read ... form.

So.

Using a pretty simple BASH script, coded in a functional way that we can reason about, we can do blue/green in a stable and controllable way.

update - 2014 August 5th

I was sending the shell function to a file and then scping the file but after a little consideration I realized it was very possible to just send the function directly to bash on the other side:

( typeset -fp functionName ; echo functionName ) | ssh host bash -

works a charm.

update - 2014 August 20th

I needed this again and again so I turned the above into a generic tool in a hideous week (shell is hard to genericize and debug).

The result is docker shell deploy which you can use to add a deploy script to your repo. The deploy script is very minimal and just pulls in more code from GitHub, thus achieving a level of upgradability.

It's still the sort of thing I may turn into a tool and package up.