I have a simple DevOps setup I want to put into the cloud. I was looking at the fastest way I could repeatability set up an infrastructure. I found AWS CloudFormation to be pretty straight forward. And I found that I can use my same CloudFormation Templates in my OpenStack cloud as well as AWS.
In this configuration I am setting up a BuildServer(TeamCity), BuildAgents, and a Defect/Work Management Server (YouTrack). Since these tools come from the same vendor they have nice integrations that I want to take advantage.
CloudFormation Templates
CloudFormation templates are XML or JSON files that are used to describe resources, their configuration and applications running on those resources. When I first started looking at CloudFormation I put all three nodes into the same Template file. This did not work since I can only define one EC2 instance per Template file. You can describe multiple resources like Databases, Storage, Load Balanacers, etc...
The Template has two major parts: the Description and the Resources.
- Description
- Resources
- Auto Scaling Group – Ensure that at least one is running
- Launch Group – Defines the EC2 Image, Init Scripts and where to get binary.
- EC2 image – ami file, Instance Type, and Security key to use for SSH
- Init Scripts- Installation scripts, start server scripts, stop server scripts.
- Sources for installation – Stored in an S3 location. Automatically unzips in specified directory.
- Security Group
- Defines the ports to open in the firewall.
Installations of Applications
Inside the CloudFormation Template you can specify the installation zip or tar file that can be used for the installation of the application. It will automatically unzip/untar the file and put it in the directory you specify. Make sure the installation zip file is in a location that your instances can use. I use S3 containers to do this. It helps decrease cost from loading over the internet everytime it starts up. It also gives me a way to do "poor-mans" configuration management. I couple of things to remember when you are doing this.
- Make sure you set permission so your instances can access them.
- Get the install zip file URL and put it in your Template file.
- If you are using public installations. You can make use the Public URL and set the Permission to public (readonly).
Init Scripts
These are used to setup and tear-down your applications on the Instances. I use the cfn-init helper script that gives my instance access to the template file.
The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to:
- Fetch and parse metadata from CloudFormation
- Install packages
- Write files to disk
- Enable/disable and start/stop services
Store the stop, start, install scripts in the metadata in the template file. It allows for multiple configsets, which gives me the ability to use the same template file for different configurations of the same application or different platform configurations.
Kicking off your instance
There are two ways to kick off your instances.
- AWS Console UI
- https://console.aws.amazon.com/cloudformation
- AWS Command Line
- aws cloudformation …
- aws cloudformation create-stack –-stack-name “MyStack” -–template_body file://home/mystack.template ...
CloudFormation Learnings
Here are some key learnings
- One Node - One Template File
- Only one type of EC2 Image can be used in a CloudFormation File. (No-heterogeneous environments). You can have more than one instance in your autoscaling group but they must all be the same instance type.
- Most AWS services can be used. ElasticBeanStalk, OpsWare, EC2, S3, SecurityGroups, etc..
- Use a Common SSH key for everything in the same infrastructure deployment. It makes it easier to check on everything.
- Create an Custom Image for instance that you want quick spin up times. No additional installation of software or packages. (BuildAgents are images).
TeamCity Template File
- For the TeamCity Template file I set up the following for AWS:
- EC2 Image – ami-e7527ed7 – AWS Linx image
- Instance Type – t2.micro
- Ports: 3389, 80, 8111, and 22
- Init Scripts
- 01-stop - teamcity-server.sh stop
- 02-start - teamcity-server.sh start
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "TeamCity Server for Linux",
"Resources": {
"TeamCityAS": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": ""
},
"LaunchConfigurationName": {
"Ref": "TeamCityLC"
},
"MaxSize": "1",
"MinSize": "0",
"DesiredCapacity": "1"
}
},
"TeamCityLC": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-e7527ed7",
"InstanceType": "t2.micro",
"KeyName": "teamcity",
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"yum update -y aws-cfn-bootstrap\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource TeamCityLC",
" --configsets all ",
" --region ",
{ "Ref": "AWS::Region" },
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"all": [
"server"
]
},
"server": {
"commands": {
"01-stop": {
"command": "/opt/TeamCity/bin/teamcity-server.sh stop",
"waitAfterCompletion": 5,
"ignoreErrors": "true"
},
"02-start": {
"command": "/opt/TeamCity/bin/teamcity-server.sh start",
"waitAfterCompletion": 120
}
},
"sources": {
"/opt/": "https://s3-us-west-1.amazonaws.com/com.yoly.teamcity.installation/installations/TeamCity-9.1.tar.gz"
}
}
}
}
},
"SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "TeamCity Server Security Group",
"SecurityGroupIngress": [
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8111", "ToPort": "8111" },
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
]
}
}
}
}
YouTrack Template File
- EC2 Image – ami-e7527ed7 – AWS Linx image
- Instance Type – t2.micro
- Ports: 3389, 8080, 80, and 22
- Init Scripts
- 01-permissions – chmod –R 755 /opt/YouTrack
- 02-stop – youtrack.sh stop
- 03-start – youtrack.sh start
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "YouTrack Server for Linux",
"Resources": {
"YouTrackAS": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": ""
},
"LaunchConfigurationName": {
"Ref": "YouTrackLC"
},
"MaxSize": "1",
"MinSize": "0",
"DesiredCapacity": "1"
}
},
"YouTrackLC": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-e7527ed7",
"InstanceType": "t2.micro",
"KeyName": "teamcity",
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"yum update -y aws-cfn-bootstrap\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource YouTrackLC ",
" --configsets youtrack ",
" --region ",
{ "Ref": "AWS::Region" },
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"youtrack": [
"youtrack"
]
},
"youtrack": {
"commands": {
"01-permissions": {
"command": "chmod -R 777 /opt/YouTrack",
"waitAfterCompletion": 1,
"ignoreErrors": "true"
},
"02-stop": {
"command": "/opt/YouTrack/bin/youtrack.sh stop",
"waitAfterCompletion": 5,
"ignoreErrors": "true"
},
"03-start": {
"command": "/opt/YouTrack/bin/youtrack.sh start",
"waitAfterCompletion": 120
}
},
"sources": {
"/opt/YouTrack": "https://s3.....installation/installations/youtrack-6.0.12634.zip"
}
}
}
}
},
"SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "TeamCity Server Security Group",
"SecurityGroupIngress": [
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8080", "ToPort": "8080" },
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
]
}
}
}
}
