Friday, December 4, 2015
Friday, August 21, 2015
Using AWS CloudFormation for DevOps
I have a simple DevOps setup I want to put into the cloud. I was looking at the fastest way I could repeatability set up an infrastructure. I found AWS CloudFormation to be pretty straight forward. And I found that I can use my same CloudFormation Templates in my OpenStack cloud as well as AWS.
In this configuration I am setting up a BuildServer(TeamCity), BuildAgents, and a Defect/Work Management Server (YouTrack). Since these tools come from the same vendor they have nice integrations that I want to take advantage.
CloudFormation Templates
CloudFormation templates are XML or JSON files that are used to describe resources, their configuration and applications running on those resources. When I first started looking at CloudFormation I put all three nodes into the same Template file. This did not work since I can only define one EC2 instance per Template file. You can describe multiple resources like Databases, Storage, Load Balanacers, etc...
The Template has two major parts: the Description and the Resources.
- Description
- Resources
- Auto Scaling Group – Ensure that at least one is running
- Launch Group – Defines the EC2 Image, Init Scripts and where to get binary.
- EC2 image – ami file, Instance Type, and Security key to use for SSH
- Init Scripts- Installation scripts, start server scripts, stop server scripts.
- Sources for installation – Stored in an S3 location. Automatically unzips in specified directory.
- Security Group
- Defines the ports to open in the firewall.
Installations of Applications
Inside the CloudFormation Template you can specify the installation zip or tar file that can be used for the installation of the application. It will automatically unzip/untar the file and put it in the directory you specify. Make sure the installation zip file is in a location that your instances can use. I use S3 containers to do this. It helps decrease cost from loading over the internet everytime it starts up. It also gives me a way to do "poor-mans" configuration management. I couple of things to remember when you are doing this.
- Make sure you set permission so your instances can access them.
- Get the install zip file URL and put it in your Template file.
- If you are using public installations. You can make use the Public URL and set the Permission to public (readonly).
Init Scripts
These are used to setup and tear-down your applications on the Instances. I use the cfn-init helper script that gives my instance access to the template file.
The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to:
- Fetch and parse metadata from CloudFormation
- Install packages
- Write files to disk
- Enable/disable and start/stop services
Store the stop, start, install scripts in the metadata in the template file. It allows for multiple configsets, which gives me the ability to use the same template file for different configurations of the same application or different platform configurations.
Kicking off your instance
There are two ways to kick off your instances.
- AWS Console UI
- https://console.aws.amazon.com/cloudformation
- AWS Command Line
- aws cloudformation …
- aws cloudformation create-stack –-stack-name “MyStack” -–template_body file://home/mystack.template ...
CloudFormation Learnings
Here are some key learnings
- One Node - One Template File
- Only one type of EC2 Image can be used in a CloudFormation File. (No-heterogeneous environments). You can have more than one instance in your autoscaling group but they must all be the same instance type.
- Most AWS services can be used. ElasticBeanStalk, OpsWare, EC2, S3, SecurityGroups, etc..
- Use a Common SSH key for everything in the same infrastructure deployment. It makes it easier to check on everything.
- Create an Custom Image for instance that you want quick spin up times. No additional installation of software or packages. (BuildAgents are images).
TeamCity Template File
- For the TeamCity Template file I set up the following for AWS:
- EC2 Image – ami-e7527ed7 – AWS Linx image
- Instance Type – t2.micro
- Ports: 3389, 80, 8111, and 22
- Init Scripts
- 01-stop - teamcity-server.sh stop
- 02-start - teamcity-server.sh start
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "TeamCity Server for Linux",
"Resources": {
"TeamCityAS": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": ""
},
"LaunchConfigurationName": {
"Ref": "TeamCityLC"
},
"MaxSize": "1",
"MinSize": "0",
"DesiredCapacity": "1"
}
},
"TeamCityLC": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-e7527ed7",
"InstanceType": "t2.micro",
"KeyName": "teamcity",
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"yum update -y aws-cfn-bootstrap\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource TeamCityLC",
" --configsets all ",
" --region ",
{ "Ref": "AWS::Region" },
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"all": [
"server"
]
},
"server": {
"commands": {
"01-stop": {
"command": "/opt/TeamCity/bin/teamcity-server.sh stop",
"waitAfterCompletion": 5,
"ignoreErrors": "true"
},
"02-start": {
"command": "/opt/TeamCity/bin/teamcity-server.sh start",
"waitAfterCompletion": 120
}
},
"sources": {
"/opt/": "https://s3-us-west-1.amazonaws.com/com.yoly.teamcity.installation/installations/TeamCity-9.1.tar.gz"
}
}
}
}
},
"SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "TeamCity Server Security Group",
"SecurityGroupIngress": [
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8111", "ToPort": "8111" },
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
]
}
}
}
}
YouTrack Template File
- EC2 Image – ami-e7527ed7 – AWS Linx image
- Instance Type – t2.micro
- Ports: 3389, 8080, 80, and 22
- Init Scripts
- 01-permissions – chmod –R 755 /opt/YouTrack
- 02-stop – youtrack.sh stop
- 03-start – youtrack.sh start
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "YouTrack Server for Linux",
"Resources": {
"YouTrackAS": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": ""
},
"LaunchConfigurationName": {
"Ref": "YouTrackLC"
},
"MaxSize": "1",
"MinSize": "0",
"DesiredCapacity": "1"
}
},
"YouTrackLC": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-e7527ed7",
"InstanceType": "t2.micro",
"KeyName": "teamcity",
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"yum update -y aws-cfn-bootstrap\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource YouTrackLC ",
" --configsets youtrack ",
" --region ",
{ "Ref": "AWS::Region" },
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"youtrack": [
"youtrack"
]
},
"youtrack": {
"commands": {
"01-permissions": {
"command": "chmod -R 777 /opt/YouTrack",
"waitAfterCompletion": 1,
"ignoreErrors": "true"
},
"02-stop": {
"command": "/opt/YouTrack/bin/youtrack.sh stop",
"waitAfterCompletion": 5,
"ignoreErrors": "true"
},
"03-start": {
"command": "/opt/YouTrack/bin/youtrack.sh start",
"waitAfterCompletion": 120
}
},
"sources": {
"/opt/YouTrack": "https://s3.....installation/installations/youtrack-6.0.12634.zip"
}
}
}
}
},
"SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "TeamCity Server Security Group",
"SecurityGroupIngress": [
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8080", "ToPort": "8080" },
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
]
}
}
}
}
Thursday, August 20, 2015
Julianne Pulsipher: Thank You
Julianne Pulsipher: Thank You: One thing I have noticed over the past few years is that teenagers are spoiled. I do not say such a thing in a rotten or a cruel way, but r...
Friday, July 31, 2015
OSCON 2015 Day 1 - Kubernetes Training
The Kubernetes Training (k8s) was built on top of the understanding of Docker and running a simple webapp (NodeJS) in a container and accessing it through security.
This is the second blog in a series of blogs about Docker and Kubernetes training. The first blog covers Docker Containers and a tutorial on how to set things up in Google Compute Engine. Check it out here (Docker Training)
Key Concepts
One of the things I quickly learned was that managing a single Container in Docker is very easy, but I can see how managing several instances of the container can be difficult and managing hundreds of heterogeneous containers would be impossible.
This is where k8s comes in. It basically lets me manage clusters of containers running on different machines in my cloud. To start k8s has a whole new nomenclature
- Master - maintains the state of k8s server runtime, control entry point for nodes, pods, services
- Node - represents a resource (machine) that pods are provisioned on. It runs docker, etcd, and kubelet daemons.
- Pod - a collections of containers that run on a machine in a node. Way to group containers together. Good way to share storage volumes across multiple containers in the same pod.
- Service and Labels - Defines a logical set of pods that make up a service or application. Policies are set up to the containers in the pods can communicate with each other. Sets up IP/Port configurations for inter-service communication. Creates a kube-proxy to front end the Service for external access.
- Volume - Storage element served up from etcd. Can connect to AWS EBS, GCE Persistent disk, iSCSI volumem etc...
- Container - Docker Container.
Hands On Tutorial
After the session went over the concepts we dove right into getting everything up and running. This is when things began to get hard. Mostly because I was on a windows box and "kubectl" (Command to control the k8s master) did not run on my windows box. So I created another VM in the Google Cloud to run through my tutorial. That ran into problems with ssh keys and security so it took much longer than it should have. I was not the only one having problems at this point. So we threw together a quick diagram to see how the k8s installation worked with the docker installation I had set up from the previous session (Docker Training) This diagram on the right shows how this all fits together.
Kubernetes introduces another command line tool "kubectl" This is accessible after loading components via the glcoud components command. The problem mentioned before is it is not supported on Windows without jumping through some hoops. So I had to create a linux VM instance to run the commands. But I had a hard time getting the right ssh keys on the VM to talk to my Google Cloud Engine Instance. The presenter of and his helpers were having a hard time getting everyone up and running on this step as well. After over an hour of trying to get this set up. about 70% of the people were able to create a k8s pod. "kubeclt create"
One thing that k8s gives developers is the ability to define higher level services with a simple script. They have chosen YAML for the description language to do this. Since YAML is quickly becoming a de facto standard for service description this was the logical choice. My first task was to create a multi-tierd web application. The tutorial walk me through a wordpress installation that had a mysql database, and the wordpress web application.
There are 4 YAML files that need to be created.
- mysql pod - Describes the mysql node. Which volume to mount for the containers in the node.
- mysql-service file - Describes the mysql-service. It contains the docker container description, ports to expose, volumes to access, user names and passwords.
- wordpress pod file - Describes the wordpress node.
- wordpress-service file - Describes the wordpress service. It contains the docker container description, ports, volumes and configuration information.
Some of the things to watch out for using k8s to define services:
- There is no way to reference secure passwords/usernames for authenticated services. These are stored as clear text in the YAML file descriptions.
- Don't even try this with Windows boxes or images. It just is not there yet.
- There is not a federated k8s Master. So if your master goes down, you have lost controller of your containers and volumes. They will still be running and consuming resources, with no way to control and monitor them.
- k8s really requires a higher level scheduler to set up kubelet nodes. Mesos is a good option for this.
- SSH key management is not as easy as it was with just a simple docker set up.
Darren
OSCON 2015 Day 1 - Docker Training
I had the opportunity to attend a couple of days of OSCON this year. Basically this was full of Open Source Developers learning about the latest and greatest of Open Source projects. As my wife said. I got to "get my geek on" during the two days I spent at OSCON.
The first day I spent the day in a Kubernetes (K8s) and Docker hands on training from the guys at Mesos and Google. The room was packed and they actually increased the size of the room about an hour and a half into the training to allow more people to come in.
There where four sessions in the training: Container overview, Docker overview and setup, K8s overview, K8s hands on.
Training Environment - Google Cloud
To decease the amount of set up the tutorial started with a running cloud that would be easy to install and deploy Docker and Kubernetes. The obvious choice was Google Cloud. So that is what we did. Everyone downloaded the Google cloud command line tools to their laptop. I was at a disadvantage because I had a windows laptop and most of these guys had Linux or MacOS boxes. But with a couple of tweaks I got everything working fine. The setup of the tutorial was easy to get going because they had a git hub repo that we just had to clone to our boxes. Everything was in there including the slides for the lecture. Very simple and made the start of the training fast.
Container Session
This was a great overview on how containers work, why docker came about, and why using docker makes life so much easier. Containers use old technology that have been in Unix OSs for about 20 years. They are based primarily on chroot, namespaces and cgroups. There were a couple of BKMs and warnings from the trainers:
- Containers are not as secure as VMs or bare-metal.
- Having multiple tenants on the same VM with multiple containers is not advised. It can be done, but not advised.
- Use Docker images to spin up Containers faster.
- There are public repositories with images that groups can share.
- Create your own private repositories for secure images.
- Do not use Docker images from the public domain unless you know they are secure. Because Docker images can access the root on your os. You have to be careful what you are down loading.
Docker Session
Because they had a git repo with all of the scripts and command line setup for us. The tutorial was easy and it worked right out of the box. I created a docker container that had a little nodejs web app and made sure that I could access it remotely. I used the command line to accomplish all of the google cloud commands. There were a couple of problems with copying files from my laptop to the instance because of the windows laptop and ssh keys, but I worked around that by downloading the files from the github repository. Here are the steps we took in the tutorial:
- Create a VM Instance to running my docker container on.
- Create a Docker image
- Inspect a running Docker container
- Find Docker images
- Launch pre-defined Docker image
I used the command line for most of the work and it was very easy to set up a VM and get my own docker container working on it.
There where a couple of gotcha's I took note:
- Much of the docker work needed to be done on the VM and could not be done remote on my Laptop.
- Windows is not a "first class" citizen with Docker Containers You really need to be running command line tools on Linux.
- You need to install the google cloud command line tools on the VM as well as your laptop.
- Store your Docker images in your own repository or make sure you have a global unique name in the google repo. We stepped on each other several times.
- Make sure you expose the ports on the VM that you want to access any web app from the Docker Container Instance.
Running through the tutorial gave me a better understanding of terminology that is thrown around and confusing at times. The following is the list of terms that we covered and finally made some sense:
- Image - This is a description of the container to run. Similar to a VM image. Images are built and stored in a registry.
- Container - This is an instance of a image running on a VM in its own namespace and cgroup.
- VM Instance - This is a virtual machine running in a cloud, in this example it is a Compute Engine.
- Docker Daemon - This manages the containers on a specific VM or Bare metal machine. The docker command communicates with the Docker Daemon.
Overall the tutorial took about 30 minutes. I would say about 90% of the people made it through the tutorial without any problems. I got some terminology cleared up and found how easy it was to get containers up and running in Google Compute Engine. The one thing that was very clear was that docker managed containers on one machine and not multiple machines.
Darren
Monday, June 22, 2015
Tool box of Container Technology
This last weekend I started cleaning out my garage. I said I started, because my garage will never completely be cleaned out. But I did manage to clean up my tool box. I found lots of interesting things in my tool box. Some broken tools, some tools that I have no idea how they got there. I am sure my neighbors and friends probably are missing them. 
I also found half started projects that just got thrown in their that don't matter anymore. Probably in a hurry to clean up the mess in the garage. Anyway. I started thinking about all of these tools in my tool box, and of course comparing them to the tools I use at work. They are not things I can touch like the hardware in my tool box, but they are invaluable tools that I can use to get jobs done. I quickly broke down the tools into categories and found a group of tools that I have downloaded, played with and am using everyday that fit in the "Container Technology" category. So I gathered those tools together and decided to write this blog to help me keep track of what container tool to use for what job. This is what I have so far. Please note this is not a complete list, but a good start I think.Container Tools
- Docker - Container Definition and nice CLI to control Containers
- Docker Compose - Define applications that contain more than one container.
- Mesos - Meta-scheduler, Distribute Kernel
- Docker Swarm - Container Scheduler that can talk to MesosS
- Kubernetes - Container Scheduler that can schedule and control Containers.
- Marathon - Service based scheduler for containers. Plugin to Mesos
- Chronos - Temporal Based Scheduler for containers. Plugin to Mesos
This is how I see everything fitting together.
Docker (Production - 1.7.0)
This helps developers define how to configure a container. Starts with a base image and then allows you to run commands, expose ports, and copy data into the container through a simple configuration file. Docker also gives CLI and REST API to allow users to control the individual containers. It requires a Docker controller running on a physical or virtual machine.
Docker Compose (Beta - 1.7.0)
Gives developers the ability to define an application as a set of micro-services. It shows the dependencies between the micro-services (containers). Which ports they expose to each other, their startup order, and any shared resources(data) between the containers that make up the application.
Mesos
(Production - 0.23.0)
"A distributed systems kernel" from the mesos web site.
Mesos describes itself as a distributed system kernel. It is responsible for picking physical compute nodes to run containers, jobs, micro-services, etc... I gets telemetry from the physical hardware and can determine which machines can best handle incoming jobs. It then provisions machines and executes job on those machines. Schedulers like Kubernetes, Marathon, Chronos, and Docker Swarm sit ontop of Mesos and act as the orchestrator for the containers running in the cloud.
Schedulers
Docker Swarm (Still in Alpha - 0.3.0 )
If you want to use a familiar Docker API you can use Docker Swarm with Mesos to control multiple containers on mutilple hosts. There is rumor it may also allow you to talk to Kubernetes in the future. Check out http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/ for more information.
Kubernetes ( pre-production Beta - 0.19.0 )
- If you want to launch groups of containers (K8 Pods) co-scheduled and co-located together, sharing resources on the same machine.
- If you want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
- if you want to use the K8s label-based service-discovery, load-balancing, and replication control. Very cool stuff for managing large number of containers.
- If you want to manage groups of containers(K8 Pods).
- If you want to load balance in isolation (K8 Pods).
Marathon (Production - 0.8.2)
- If you want to launch applications that contain long running heterogeneous apps/services (Docker and Non-Docker).
- If you want to use Mesos attributes for constraint-based scheduling.
- If you want to use application groups and dependencies to launch, scale, or upgrade related services.
- If you want to use event driven health checks to automatically restart unhealthy services.
- If you want to integrate HAProxy or Consul for service discovery.
- If you want a nice web UI or REST API to launch and monitor apps.
Chronos (Production - 2.3.4)
- If you want to launch applications that contain short running heterogeneous apps/services (Docker and Non-Docker).
- If you want to schedule a remporal task to run at a specific time/schedule, just like cron.
- If you want to schedule a DAG workflow of dependent tasks.
- If you want a nice web UI or REST API to launch and monitor apps.
- If you want to use a scheduler that was built from the start with Mesos in mind.
Ok. This is the start. I am sure many of you have more to add to the list. Please let me know what I am missing.
DWP.
Subscribe to:
Posts (Atom)