Fault Tolerant Jenkins with Docker Swarm

Installing Docker Swarm

I have chosen to use CentOS 7 for my cluster of machines. So these instructions are for CentOS 7.

First I need to install docker on all of the machines in the cluster.

Set up yum so it can see the latest docker packages.

# sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Next install docker onto each machine in your cluster.

# sudo yum install docker-ce

Once you have installed docker on every node in your cluster you can now set up your swarm. First you have to choose which machines will be your manager(s).

On one of the masters you need to initialize the swarm

# docker swarm init

If your machine has more than one network then you will need to specify the ip address to use for the master.

# docker swarm init --advertise-addr 172.16.0.100
Swarm initialized: current node (a2anz4z0mpb0vmcly5ksotfo1) is now a manager.
To add a worker to this swarm, run the following command:
    docker swarm join \    --token SWMT....wns 172.16.0.100:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

In this example the IP address is 172.16.0.100 for the swarm master. Now on each worker node I just run the join command as specified in the output of the init command.

# docker swarm join --token SWMT...wns 172.16.0.100:2377

If you want to add another master then you run the command

# docker swarm join-token manager

It will tell you exactly what you need to do.

Setting up Jenkins in your Swarm

Now this is the easy part. Sort of. With docker swarm and services this has just gotten much easier. There are several docker images that are available with jenkins already installed in them. So it is best if we just use one of them. The most popular is "jenkins". Go figure. Now with the image name all we need to do is start a service in the swarm. We can simply write a small compose file and we will be set.

# docker-jenkins.yaml
version: '3'
services:
  jenkins:
    image: jenkins
    ports:
      - "8082:8080"
      - "50000:50000"
    environment:
      JENKINS_OPTS: --prefix=/jenkins
    deploy:
      placement:
        constraints: [node.role == manager]
    volumes:
      - $PWD/docker/jenkins:/var/jenkins_home

There are a couple of things to note. 

• Ports -  First the ports are mapped to 50000 and 8082. This are external ports and will be accessibe outside of the container.
• Environment - You can set any jenkins options on this line any following environment lines
• Volumes - This will give us the ability to "mount" a directory from the host machine into the container. So if the container goes down we still have our jenkins installation.  You will need to create the directory using

# mkdir ~/docker/jenkins && chmod 777 ~/docker/jenkins

if you don't do this you will have problems with jenkins coming up.

Now it is time to actually start the service.

# docker stack deploy -c docker-jenkins.yaml buildCreating network build_default
Creating service build_jenkins

Two things where created when the deploy was run. A default network "build_default" and the service "build_jenkins" notice all of the artifacts created will begin with "build_". The default network is created when a network is not specified.

Now you should be able to access the jenkins web site at

http://172.16.0.100:8082/jenkins

Jenkins now requires a password when you install. You can find the password in the secrets directory in the docker/jenkins base directory.

# cat ~/docker/jenkins/secrets/initialAdminPassord
asldfkasdlfkjlasdfj23iwrh

Cut and paste this into your browser and you will be set and ready to go.

Debugging Tools

Here are a couple of things I found useful when I was setting up the environment.

# docker ps

CONTAINER ID        IMAGE                                                                             COMMAND                  CREATED             STATUS              PORTS                 NAMES
91aa53f4642a        jenkins@sha256:c0cac51cbd3af8947e105ec15aa4bcdbf0bd267984d8e7be5663b5551bbc5f4b   "/bin/tini -- /usr..."   5 hours ago         Up 5 hours          8080/tcp, 50000/tcp   build_jenkins.1.abu55c8tybjwrsd35ouaor1d2

Shows the containers that are currently running. This will include the containers that are running the services. I found that some of the containers never started up. So I was trying to find out what happen. So I ran the following command:

# docker service ps build_jenkins

 ID            NAME                 IMAGE           NODE               DESIRED ST    ATE  CURRENT STATE        ERROR                      PORTS
abu55c8tybjw  build_jenkins.1      jenkins:latest  node0.intel.local  Running            Running 5 hours ago
nac73zp1gc68   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
xyrmzvx1pnnp   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
phycp5ypp61o   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"
o3ewixv3hvcy   \_ build_jenkins.1  jenkins:latest  node0.intel.local  Shutdown           Failed 5 hours ago   "task: non-zero exit (1)"

This will show the tasks for the services before the containers get launched and their status.