Fault Tolerant Jenkins with Docker Swarm

Installing Docker Swarm

I have chosen to use CentOS 7 for my cluster of machines. So these instructions are for CentOS 7.

First I need to install docker on all of the machines in the cluster.

Set up yum so it can see the latest docker packages.

# sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Next install docker onto each machine in your cluster.

# sudo yum install docker-ce

Once you have installed docker on every node in your cluster you can now set up your swarm. First you have to choose which machines will be your manager(s).

On one of the masters you need to initialize the swarm

# docker swarm init

If your machine has more than one network then you will need to specify the ip address to use for the master.

# docker swarm init --advertise-addr 172.16.0.100
Swarm initialized: current node (a2anz4z0mpb0vmcly5ksotfo1) is now a manager.
To add a worker to this swarm, run the following command:
    docker swarm join \    --token SWMT....wns 172.16.0.100:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

In this example the IP address is 172.16.0.100 for the swarm master. Now on each worker node I just run the join command as specified in the output of the init command.

# docker swarm join --token SWMT...wns 172.16.0.100:2377

If you want to add another master then you run the command

# docker swarm join-token manager

It will tell you exactly what you need to do.

Setting up Jenkins in your Swarm

Now this is the easy part. Sort of. With docker swarm and services this has just gotten much easier. There are several docker images that are available with jenkins already installed in them. So it is best if we just use one of them. The most popular is "jenkins". Go figure. Now with the image name all we need to do is start a service in the swarm. We can simply write a small compose file and we will be set.

# docker-jenkins.yaml
version: '3'
services:
  jenkins:
    image: jenkins
    ports:
      - "8082:8080"
      - "50000:50000"
    environment:
      JENKINS_OPTS: --prefix=/jenkins
    deploy:
      placement:
        constraints: [node.role == manager]
    volumes:
      - $PWD/docker/jenkins:/var/jenkins_home

There are a couple of things to note. 

• Ports -  First the ports are mapped to 50000 and 8082. This are external ports and will be accessibe outside of the container.
• Environment - You can set any jenkins options on this line any following environment lines
• Volumes - This will give us the ability to "mount" a directory from the host machine into the container. So if the container goes down we still have our jenkins installation.  You will need to create the directory using

# mkdir ~/docker/jenkins && chmod 777 ~/docker/jenkins

if you don't do this you will have problems with jenkins coming up.

Now it is time to actually start the service.

# docker stack deploy -c docker-jenkins.yaml buildCreating network build_default
Creating service build_jenkins

Two things where created when the deploy was run. A default network "build_default" and the service "build_jenkins" notice all of the artifacts created will begin with "build_". The default network is created when a network is not specified.

Now you should be able to access the jenkins web site at

http://172.16.0.100:8082/jenkins

Jenkins now requires a password when you install. You can find the password in the secrets directory in the docker/jenkins base directory.

# cat ~/docker/jenkins/secrets/initialAdminPassord
asldfkasdlfkjlasdfj23iwrh

Cut and paste this into your browser and you will be set and ready to go.

Debugging Tools

Here are a couple of things I found useful when I was setting up the environment.

# docker ps

CONTAINER ID        IMAGE                                                                             COMMAND                  CREATED             STATUS              PORTS                 NAMES
91aa53f4642a        jenkins@sha256:c0cac51cbd3af8947e105ec15aa4bcdbf0bd267984d8e7be5663b5551bbc5f4b   "/bin/tini -- /usr..."   5 hours ago         Up 5 hours          8080/tcp, 50000/tcp   build_jenkins.1.abu55c8tybjwrsd35ouaor1d2

Shows the containers that are currently running. This will include the containers that are running the services. I found that some of the containers never started up. So I was trying to find out what happen. So I ran the following command:

# docker service ps build_jenkins

 ID            NAME                 IMAGE           NODE               DESIRED ST    ATE  CURRENT STATE        ERROR                      PORTS
abu55c8tybjw  build_jenkins.1      jenkins:latest  node0.intel.local  Running            Running 5 hours ago
nac73zp1gc68   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
xyrmzvx1pnnp   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
phycp5ypp61o   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
o3ewixv3hvcy   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"

This will show the tasks for the services before the containers get launched and their status.

Moving Docker Compose to Docker Stack

Ok. I am finally moving from Docker Compose to Docker Stack. It has been a while since I updated my Docker environment and I am very happy with the direction that docker has moved. They have moved in the direction that I am personally have been promoting. Check out my blog on Services in Multiple environments. Multiple Environment Development.

Swarm Concepts

The first thing I did was read up on the changes in concepts between compose and stack. Docker introduced new concepts of Stack, Service and Task. The easiest way to think of it is a Stack consists of several services, networks and volumes. A Stack can represent a complex application that has multiple services.

A Service can have a set or replicas that consists of a image running in a container and tasks that are run on the container. A Service has State. This is where things are different between compose and stack. Compose launches all of the containers and runs tasks and then forgets about it. Stack can keep track of the state of the containers even after they have been launched. This means if a container that correlates with a service goes down it will launch another one in its place based on policies. Basically your application will be kept up by Swarm. Built in HA, load balancing and Business continuity.

When you specify a service you specify:

• the port where the swarm will make the service available outside the swarm
• an overlay network for the service to connect to other services in the swarm
• CPU and memory limits and reservations
• a rolling update policy
• the number of replicas of the image to run in the swarm

Notice the word Swarm here. You must have a docker swarm before you use services and stacks. 

Practical differences

Compose files can be used for Stack deployments. But there are a couple of things to watch out for.

• "buid" is not supported in stack you have to build with docker build
• "external_links" is not supported in stack this is covered by links and external hosts.
• "env_file" is not supported in stack you have to specify each environment variable with "environment"

Wow! That was a problem for me because my compose file had build directives for my project and had env_file to pass in environment variables. Now I had to make changes to make things work the way before.

"build" Alternative

Simply put stack services only take images. So that means that you must build your image before you deploy or update your stack. So instead of just specifying the build in the service definition you must call docker build before calling docker stack deploy.

File: docker-compose.yaml

etsy-web:
  build: .
  expose:
    - 80
    - 8080
    - 1337
  links:
    - etsy-mongo
    - etsy-redis
  ports:
    - "1337:1337"
    - "80:80"
  command: npm start 

And to launch my containers then I just call

# docker-compose up

To make this work properly we need to remove the build line above. And replace it with an image key.

etsy-web:
  image: etsy-web
  expose:
    - 80
    - 8080
    - 1337
  links:
    - etsy-mongo
    - etsy-redis
  ports:
    - "1337:1337"
    - "80:80"
  command: npm start

Then you have to build the etsy-web image first and then deploy the stack

# docker build .
# docker stack deploy --compose-file docker-compose.yaml

So it is that easy. Change one key in your yaml file and you can be up and running.

env_file alternative

With stack you specify the environments using the environment tag.

This can be using dictionary or array formats.

Dictionary

environment: 

  PASSWORD: qwerty

  USERNAME: admin

or Array

environment:

  - PASSWORD=qwerty

  - USERNAME=admin

Also take note that the environment variables in the docker-compose.yaml file override any environment variables defined in the dockerfile for the container. Additionally environment variables can be passed into the command line when calling "docker stack". These environment variable override the environment variables in the both the docker-compose.yaml and dockerfile.yaml files.

external_links alternative

"stack" uses link and external hosts to establish service names that are looked up when the containers are launched. This is a change from before when a changes to /etc/hosts was changed for each container to establish container connectivity. See Docker Service Deiscovery.

Benefits of Stack

Even though there are changes to some of the yaml file format and some additional command line options the benefits of having services over containers, stacks over containers is huge. I can now have a managed stack that keeps my services up and running based on policies that I have established for the each service. 

DWP

Tool box of Container Technology

This last weekend I started cleaning out my garage. I said I started, because my garage will never completely be cleaned out. But I did manage to clean up my tool box. I found lots of interesting things in my tool box. Some broken tools, some tools that I have no idea how they got there. I am sure my neighbors and friends probably are missing them.  I also found half started projects that just got thrown in their that don't matter anymore. Probably in a hurry to clean up the mess in the garage. Anyway. I started thinking about all of these tools in my tool box, and of course comparing them to the tools I use at work. They are not things I can touch like the hardware in my tool box, but they are invaluable tools that I can use to get jobs done. I quickly broke down the tools into categories and found a group of tools that I have downloaded, played with and am using everyday that fit in the "Container Technology" category. So I gathered those tools together and decided to write this blog to help me keep track of what container tool to use for what job. This is what I have so far. Please note this is not a complete list, but a good start I think.

Container Tools

• Docker - Container Definition and nice CLI to control Containers
• Docker Compose - Define applications that contain more than one container.
• Mesos - Meta-scheduler, Distribute Kernel
• Docker Swarm - Container Scheduler that can talk to MesosS
• Kubernetes - Container Scheduler that can schedule and control Containers.
• Marathon - Service based scheduler for containers. Plugin to Mesos
• Chronos - Temporal Based Scheduler for containers. Plugin to Mesos

This is how I see everything fitting together.

Docker (Production - 1.7.0)

This helps developers define how to configure a container. Starts with a base image and then allows you to run commands, expose ports, and copy data into the container through a simple configuration file. Docker also gives CLI and REST API to allow users to control the individual containers. It requires a Docker controller running on a physical or virtual machine.

Docker Compose (Beta - 1.7.0)

Gives developers the ability to define an application as a set of micro-services. It shows the dependencies between the micro-services (containers). Which ports they expose to each other, their startup order, and any shared resources(data) between the containers that make up the application.

Mesos (Production - 0.23.0)

"A distributed systems kernel" from the mesos web site.

Mesos describes itself as a distributed system kernel. It is responsible for picking physical compute nodes to run containers, jobs, micro-services, etc... I gets telemetry from the physical hardware and can determine which machines can best handle incoming jobs. It then provisions machines and executes job on those machines. Schedulers like Kubernetes, Marathon, Chronos, and Docker Swarm sit ontop of Mesos and act as the orchestrator for the containers running in the cloud.

Schedulers

Docker Swarm (Still in Alpha - 0.3.0 )

If you want to use a familiar Docker API you can use Docker Swarm with Mesos to control multiple containers on mutilple hosts. There is rumor it may also allow you to talk to Kubernetes in the future. Check out http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/ for more information.

Kubernetes ( pre-production Beta - 0.19.0 )

• If you want to launch groups of containers (K8 Pods) co-scheduled and co-located together, sharing resources on the same machine.
• If you want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
• if you want to use the K8s label-based service-discovery, load-balancing, and replication control. Very cool stuff for managing large number of containers.
• If you want to manage groups of containers(K8 Pods).
• If you want to load balance in isolation (K8 Pods).

Marathon (Production - 0.8.2)

• If you want to launch applications that contain long running heterogeneous apps/services (Docker and Non-Docker).
• If you want to use Mesos attributes for constraint-based scheduling.
• If you want to use application groups and dependencies to launch, scale, or upgrade related services.
• If you want to use event driven health checks to automatically restart unhealthy services.
• If you want to integrate HAProxy or Consul for service discovery.
• If you want a nice web UI or REST API to launch and monitor apps.

Chronos (Production - 2.3.4)

• If you want to launch applications that contain short running heterogeneous apps/services (Docker and Non-Docker).
• If you want to schedule a remporal task to run at a specific time/schedule, just like cron.
• If you want to schedule a DAG workflow of dependent tasks.
• If you want a nice web UI or REST API to launch and monitor apps.
• If  you want to use a scheduler that was built from the start with Mesos in mind.

Ok. This is the start. I am sure many of you have more to add to the list. Please let me know what I am missing.

DWP.