Monday, May 14, 2012

Speeding up EC2 work by using AWS tools and scripts to bypass the AWS management console

Believe me managing EC2 instances is not as simple or magical as marketers would like for you to believe. The cloud gets complicated when it gets bigger. EC2 really only enables a person to ignore power, network layout (which is bad), and getting more servers that end up costing more then actual servers fast.

Things that EC2 is missing to make life easier for the developer:

Ability to update all servers with packages and code. Natively they do not support the ability to push files or install new software packages to server groups. Thus install cluster-it and puppet and write your own deploy program.

Server names and the EC2 AWS management console do not match. Everything is referenced by instance ids. The interface does not allow one to launch many instances in a named pattern so you have to go back and sync up the instance with the internal named used in the app. Syncing up the names is very tedious and time consuming process through the console.

Assigning EBS volumes is a pain in the ass as well. Essentially you need to assign them one by one, where each add takes more then 2 mins todo this is not good time spent. For instance it took me roughly 20 mins to attach 8 63GB EBS volumes to a single server.

Amazon is perfectly aware of these limitations-and do not hide from it. They are going after something bigger. They are providing a platform. There is an API for everything you need to make your work easier. Now the pain is to learn the API and use it in your favor. There are companies that built a business on making a better interface for the AWS console but getting it done yourself is cheaper.

My personal mantra is to automate things that I have to do more then once. Anytime that I deploy new instances, I take the private IP add it to DNS, make an API call through ec2-describe-instances, find the instance id and update the name through ec2-create-tags. This solves the problem that I have with mapping instance ids to my internal name which the app uses. For instance:
#!/usr/bin/perl -w 
use strict;
use Data::Dumper;
open(HOSTS, "</etc/hosts") or die($!);

my $hosts = {};
    my ($ip, $hostname, undef) = split(/\s+/, $_);
    $hosts->{$ip} = $hostname;

open(FH, "/opt/aws/bin/ec2-describe-instances -C cert.pem -K x509.pem --region us-west-1|") or die($!);
    if ($_ =~ /^INSTANCE\t(.*)/){
        my (@fields) = split(/\s+/, $1);
        # 0 - instance
        # 1 - ami
        # 2 - public dns
        # 3 - private dns
        # 4 - state
        # 5 - ??
        # 6 - ??
        # 7 - instance type
        # 8 - date created
        # 9 - DC
        # 10 - ??
        # 11 - monitoring state
        # 12 - public ip
        # 13 - private ip
        # 14 - ebs
        # 15 - ??
     if ($fields[4] eq 'running'){
            my $role;
            my $hostname = $hosts->{$fields[13]};
            if (!$hostname) {
                print "$fields[13] is not in the hosts file skipping..\n";
            if ($hostname =~ /^job/){
                $role = 'gearman-worker';
            if ($hostname =~ /^gearman/){
                $role = 'gearman-queue';
            if ($hostname =~ /^www/){
                $role = 'webserver';
            if ($hostname =~ /^memc/){
                $role = 'memcache';
            if ($hostname =~ /^db/){
                $role = 'database';
            if ($hostname =~ /^dbshard/) {
                $role = 'database-shard';
            if (!$role){
                print "$hostname does not have a role\n";
                $role = 'other';
            system("./aws/bin/ec2-create-tags -C cert.pem -K x509.pem --region us-west-1 ".$fields[0] ." --tag Na
me=$hostname --tag Role=$role");

Now to attach disks to an instance, that I am upgrading or re-purposing I wrote a quick script that describes the input instance after translating from my internal name to instance id. Calculates the size of each disk and attaches said disks. For instance:
#!/usr/bin/perl -w
use strict;
use Data::Dumper;
use POSIX qw(ceil);
print "Enter Hostname: ";
my $hostname = <>;
my $cmd = './aws/bin/ec2-describe-instances -C cert.pem -K x509.pem --region us-west-1 --filter="tag-key=Name" --
filter="tag-value=' . $hostname . '"';
open(FH, "$cmd|") or die ("Awesome death: $!\n");
my $instance = "";
my $lastDisk = "";
my $diskCount = "";
if($_ =~ /^INSTANCE\t(.*)/){
        my (@fields) = split(/\s+/, $1);
        $instance = $fields[0];
        print "Instance=$instance\n";
    if($_ =~ /^BLOCKDEVICE\t(.*)/){
        my (@fields) = split(/\s+/, $1);
        $lastDisk = $fields[0];
        print "$lastDisk\n";
print "How many disks you would like to add: ";
my $totalAddDisks = <>;
print "You picked $totalAddDisks\n";
print "What is the total size of the Raid0 Array in GB: ";
my $totalSize = <>;
print "You picked $totalSize GB\n";
my $sizeperdisk = ceil($totalSize/$totalAddDisks);
print "The size per disk: $sizeperdisk\n";
$lastDisk =~ /sd(\S)/;
my $lastDeviceLetter = $1;
my @devicesavail = ($lastDeviceLetter .. 'z');
for(my $i = 1; $i <= $totalAddDisks; $i++){
    $cmd = "./aws/bin/ec2-create-volume --size $sizeperdisk --region us-west-1 --availability-zone us-west-1c -C cert.pem  -K
    my $ret = `$cmd`;
    my (@output) = split(/\s+/, $ret);
    $cmd = "./aws/bin/ec2-attach-volume --region us-west-1 -C cert.pem -K x509.pem $output[1] --instance $instanc
e --device /dev/sd$devicesavail[$i]";
    $ret = `$cmd`;
    if ($ret =~ /attaching/){
        print "All good do the next one\n";
    } else {
        die("Did not work\n");

These are rough and dirty scripts that get the job done for my environment. The end goal when given time is to turn these scripts into a package talking over httpd that makes life easier when working in EC2. Using these two script have reduced the management time from 1 hour per server upgrade to a few minutes.


Ronald Bradford said...

You cannot manage 500 EC2 MySQL servers with the console. I have open sourced a number of my tools at

Launching, Switching from spot to on demand, EC2 cost per hour graphs etc.

Dathan Pattishall said...

@Ronald Agree the management console is just not forgiving. API calls are very slow. Additionally I looked at your scripts they're awesome. I need to start putting stuff in github.

Streeter said...

The java ec2-* api commands are quite slow compared to client libraries. Look to see if there is a client library for your language. boto is awesome for Python.