OSCON 2015 Day 1 - Docker Training

I had the opportunity to attend a couple of days of OSCON this year. Basically this was full of Open Source Developers learning about the latest and greatest of Open Source projects. As my wife said. I got to "get my geek on" during the two days I spent at OSCON.

The first day I spent the day in a Kubernetes (K8s) and Docker hands on training from the guys at Mesos and Google. The room was packed and they actually increased the size of the room about an hour and a half into the training to allow more people to come in.

There where four sessions in the training: Container overview, Docker overview and setup, K8s overview, K8s hands on.

Training Environment - Google Cloud

To decease the amount of set up the tutorial started with a running cloud that would be easy to install and deploy Docker and Kubernetes. The obvious choice was Google Cloud. So that is what we did. Everyone downloaded the Google cloud command line tools to their laptop. I was at a disadvantage because I had a windows laptop and most of these guys had Linux or MacOS boxes. But with a couple of tweaks I got everything working fine.  The setup of the tutorial was easy to get going because they had a git hub repo that we just had to clone to our boxes. Everything was in there including the slides for the lecture. Very simple and made the start of the training fast.

Container Session

This was a great overview on how containers work, why docker came about, and why using docker makes life so much easier. Containers use old technology that have been in Unix OSs for about 20 years. They are based primarily on chroot, namespaces and cgroups. There were a couple of BKMs and warnings from the trainers:

• Containers are not as secure as VMs or bare-metal.
• Having multiple tenants on the same VM with multiple containers is not advised. It can be done, but not advised.
• Use Docker images to spin up Containers faster.
• There are public repositories with images that groups can share.
• Create your own private repositories for secure images.
• Do not use Docker images from the public domain unless you know they are secure. Because Docker images can access the root on your os. You have to be careful what you are down loading.

Docker Session

Because they had a git repo with all of the scripts and command line setup for us. The tutorial was easy and it worked right out of the box. I created a docker container that had a little nodejs web app and made sure that I could access it remotely. I used the command line to accomplish all of the google cloud commands. There were a couple of problems with copying files from my laptop to the instance because of the windows laptop and ssh keys, but I worked around that by downloading the files from the github repository. Here are the steps we took in the tutorial:

1. Create a VM Instance to running my docker container on.
   
2. Create a Docker image
3. Inspect a running Docker container
4. Find Docker images
5. Launch pre-defined Docker image

I used the command line for most of the work and it was very easy to set up a VM and get my own docker container working on it.

There where a couple of gotcha's I took note:

• Much of the docker work needed to be done on the VM and could not be done remote on my Laptop.
• Windows is not a "first class" citizen with Docker Containers You really need to be running command line tools on Linux.
• You need to install the google cloud command line tools on the VM as well as your laptop.
• Store your Docker images in your own repository or make sure you have a global unique name in the google repo. We stepped on each other several times.
• Make sure you expose the ports on the VM that you want to access any web app from the Docker Container Instance.

Running through the tutorial gave me a better understanding of terminology that is thrown around and confusing at times. The following is the list of terms that we covered and finally made some sense:

• Image - This is a description of the container to run. Similar to a VM image. Images are built and stored in a registry.
• Container - This is an instance of a image running on a VM in its own namespace and cgroup.
• VM Instance - This is a virtual machine running in a cloud, in this example it is a Compute Engine.
• Docker Daemon - This manages the containers on a specific VM or Bare metal machine. The docker command communicates with the Docker Daemon.

Overall the tutorial took about 30 minutes. I would say about 90% of the people made it through the tutorial without any problems. I got some terminology cleared up and found how easy it was to get containers up and running in Google Compute Engine. The one thing that was very clear was that docker managed containers on one machine and not multiple machines.

Darren

Tool box of Container Technology

This last weekend I started cleaning out my garage. I said I started, because my garage will never completely be cleaned out. But I did manage to clean up my tool box. I found lots of interesting things in my tool box. Some broken tools, some tools that I have no idea how they got there. I am sure my neighbors and friends probably are missing them.  I also found half started projects that just got thrown in their that don't matter anymore. Probably in a hurry to clean up the mess in the garage. Anyway. I started thinking about all of these tools in my tool box, and of course comparing them to the tools I use at work. They are not things I can touch like the hardware in my tool box, but they are invaluable tools that I can use to get jobs done. I quickly broke down the tools into categories and found a group of tools that I have downloaded, played with and am using everyday that fit in the "Container Technology" category. So I gathered those tools together and decided to write this blog to help me keep track of what container tool to use for what job. This is what I have so far. Please note this is not a complete list, but a good start I think.

Container Tools

• Docker - Container Definition and nice CLI to control Containers
• Docker Compose - Define applications that contain more than one container.
• Mesos - Meta-scheduler, Distribute Kernel
• Docker Swarm - Container Scheduler that can talk to MesosS
• Kubernetes - Container Scheduler that can schedule and control Containers.
• Marathon - Service based scheduler for containers. Plugin to Mesos
• Chronos - Temporal Based Scheduler for containers. Plugin to Mesos

This is how I see everything fitting together.

Docker (Production - 1.7.0)

This helps developers define how to configure a container. Starts with a base image and then allows you to run commands, expose ports, and copy data into the container through a simple configuration file. Docker also gives CLI and REST API to allow users to control the individual containers. It requires a Docker controller running on a physical or virtual machine.

Docker Compose (Beta - 1.7.0)

Gives developers the ability to define an application as a set of micro-services. It shows the dependencies between the micro-services (containers). Which ports they expose to each other, their startup order, and any shared resources(data) between the containers that make up the application.

Mesos (Production - 0.23.0)

"A distributed systems kernel" from the mesos web site.

Mesos describes itself as a distributed system kernel. It is responsible for picking physical compute nodes to run containers, jobs, micro-services, etc... I gets telemetry from the physical hardware and can determine which machines can best handle incoming jobs. It then provisions machines and executes job on those machines. Schedulers like Kubernetes, Marathon, Chronos, and Docker Swarm sit ontop of Mesos and act as the orchestrator for the containers running in the cloud.

Schedulers

Docker Swarm (Still in Alpha - 0.3.0 )

If you want to use a familiar Docker API you can use Docker Swarm with Mesos to control multiple containers on mutilple hosts. There is rumor it may also allow you to talk to Kubernetes in the future. Check out http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/ for more information.

Kubernetes ( pre-production Beta - 0.19.0 )

• If you want to launch groups of containers (K8 Pods) co-scheduled and co-located together, sharing resources on the same machine.
• If you want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
• if you want to use the K8s label-based service-discovery, load-balancing, and replication control. Very cool stuff for managing large number of containers.
• If you want to manage groups of containers(K8 Pods).
• If you want to load balance in isolation (K8 Pods).

Marathon (Production - 0.8.2)

• If you want to launch applications that contain long running heterogeneous apps/services (Docker and Non-Docker).
• If you want to use Mesos attributes for constraint-based scheduling.
• If you want to use application groups and dependencies to launch, scale, or upgrade related services.
• If you want to use event driven health checks to automatically restart unhealthy services.
• If you want to integrate HAProxy or Consul for service discovery.
• If you want a nice web UI or REST API to launch and monitor apps.

Chronos (Production - 2.3.4)

• If you want to launch applications that contain short running heterogeneous apps/services (Docker and Non-Docker).
• If you want to schedule a remporal task to run at a specific time/schedule, just like cron.
• If you want to schedule a DAG workflow of dependent tasks.
• If you want a nice web UI or REST API to launch and monitor apps.
• If  you want to use a scheduler that was built from the start with Mesos in mind.

Ok. This is the start. I am sure many of you have more to add to the list. Please let me know what I am missing.

DWP.

Running tests in a CI system for SailsJS

I recently started working with SailsJS. I am enjoying some of the libraries and how easy and fast it is to get things up and running. But I do miss some of the built in tools that come with Grails. A good example of this is the "grails test" tool. This made it easy to run unit level tests, integration or functional tests from the command line with the same command line for all of my projects.

grails test-app 

This made it very easy to integrate into a CI tool like TeamCity. I didn't need any special scripts, relative paths or some dark art of getting environment variables right. Things just worked out of the box.

Since SailsJS lets you plug in any testing you want. AKA it does not come built into the framework. I had to figure out how to run the tests in my IDE (Intellij in this case), and in my CI system (TeamCity). As you can see I am trying to keep the tool set in the same family.

I decided to use Mocha as the test framework. It seemed to have everything that I wanted and was the closest to what I was familiar with in the grails framework. But the commands to get it running were a little more than just typing "sailjs test" in my application directory. But I found a cool little trick to make it easy. Using npm and my package.json file. I can use the same command for both CI and IDE.

By simply adding a line to the package.json file in the root directory of the application source I can then call

npm test

Here is an example of my package.json file.

...

  "sails-mongo": "^0.11.1"},
"scripts": {
  "start": "node app.js",
  "test": "node ./node_modules/mocha/bin/_mocha --recursive --ui bdd ./test",
  "debug": "node debug app.js"},
"main": "app.js",

...

The test script calls the node command with the proper mocha script to run my tests. Notice that the path to the mocha script is relative to the application root directory. It is important in the IDE and CI that you set the working directory to the application root. Which should be the default for both. Now I can simply call:

npm test

from the command line, in the IDE, or in the CI system.

DWP

Father's Day Poem.

My genius daughter wrote me a poem for Father's day. Here it is in all of it glory. Of course I could not read it all the way through without crying which my kids think is very funny to watch me cry.

To my Father

From infancy you held me,
From infancy you clothed me,
You are the one who helped me see
Who I truly was going to be.
In confusion you help me understand,
In sickness your hand were on my head.
You are my teacher, leader, provider, and friend.
I know you will help me to the end.
No matter how much I must learn
No matter how much my life will turn
You will be my teach,leader, provider and friend
And your fatherhood will have no end.

Julianne Pulsipher

Characteristics of Hate

This last week I began teaching the history of Zeniff and his people in the book of Mosiah in the Book of Mormon. One of the interesting comparisons between the Nephites and the Lamanites is the profound hate that the Lamanites have toward the Nephites. The hate was so strong that it was passed down from generation to generation. So I asked my seminary kids how could so much hate be passed down for so many years. We came up with some interesting characteristics. 

First we saw that the Lamanites were "Quick to take offence". It was almost like they were looking for reasons to be take offense. We see the same sorts of things today in our society. Some of the kids mentioned many racial tension can point to this one characteristic. Easy to take offense. The kid that pointed this out happen to be multiracial. He was frustrated with people from both of his races that took offense to everything. Then one of the girls in class mentioned that they have fallen into this satanic trap as well. When they took offense from a friend when no offense was meant. Destroying their friendship.

Second we saw that the Lamanites "held on to anger". They wanted to be angry at the Nephites and they held on to that anger for over 500 years. Sound familiar to the world today. Just think about the middle east. Thousands of years of anger. To the point that they want thrive on anger and hate.

Third, the Lamanites blamed the Nephites for everything. Check out Mosiah 10:12-18. The Lamanites were "wroth" (intensely angry) with the Nephites about everything. Even good things that happen to the Lamanites, they still found a reason to hate them. Reading through the reasons it is almost comical. In today's venacular we call it being a victim. When we become victims and blame everyone else for what is going wrong in our lives we fall into this characteristic. It fuels anger and hatred, and slowly destroys our spirit.

Lastly, the Lamanites refused to forgive. You can see something in someone's countenance when they carry a grudge with people. They have a demeanor of anger. We all recognize it in other people all of the time. We sometimes miss it in ourselves.

So how do we avoid these characteristics Elder Halstrom said in an article in the Ensign 

“If you feel you have been wronged—by anyone (a family member, a friend, another member of the Church, a Church leader, a business associate) or by anything (the death of a loved one, health problems, a financial reversal, abuse, addictions)—deal with the matter directly and with all the strength you have. … And, without delay, turn to the Lord. Exercise all of the faith you have in Him. Let Him share your burden. Allow His grace to lighten your load. … Never let an earthly circumstance disable you spiritually” (“Turn to the Lord,” Ensign or Liahona, May 2010, 80).

Something I think we all need to watch out for in our individual lives.

DWP

Christmas Gift ideas

Shameless plug for my wife's etsy store. Christmas is coming get your orders in fast.

Nephi's last words

I get the opportunity to teach seminary in the morings for my church. This week we are finishing up the 2nd book of Nephi in the Book of Mormon. Theses are the prophet Nephi's last words he writes to his children, decendents and us. Every time I read Nephi's last words to us in 2 Nephi 33, I begin tear up and wonder what my last words would be to my wife, my children, and my friends. He mentions four things

1. I pray continually for ...
2. I know ...
3. I glory ...
4. I have ...

After reading this again. I figure I need to answer these blanks for my self. I guess that is one of the things we need to figure out in life. 

Darren