Ways to automatically deploy Multi-Machine Blueprint in AWS

This week I was working on a small engagement on automating AWS Deployment. Instead of covering the details of this specific project – I’d like to discuss the ways the deployment can be done in AWS and hope some of the readers can come up with more suggestions.

First of all it’s really unbelievable on how everything is documented and examples are provided with every statement.

Additionally, there are things that in most platforms you have to program and tweak yourself – in AWS the functionality is provided out-of-the-box. In example – when your code needs to wait for deployment – usually you would create a cycle that will be checking (kind of pinging) service until it responds. In AWS it is a built-in function – you can “pause” your script until the component is deployed (or destroyed etc) using a single command – no cycles, no “pinging”.

So let’s look what tools can be used to deploy Multi-Machine Blueprint (MMB can be VMware term). There is AWS Command-line interface – it comes in Windows and Linux flavors. The commands and parameters are identical that makes it easy to migrate script from batch file (.bat) on windows to Linux shell script. As you probably know both of those are simple set of commands that executed sequentially.

Numerous parameters can make command pretty heavy to read – in AWS there is always way to define those parameters (such as Security Groups, VPCs, ELBs etc) in form of JSON file.

--generate-cli-skeleton creates empty json template that has all required parameters fields in it

What I was doing to streamline the process is using AWS UI to create an object than export the object details into json using describe-<object> command and then moving necessary parameters to my own template to create a new object with this template.

The only caveat I found is getting specific parameter extracted from JSON output to be used in different command. I’m using Windows machine (even my familiarity with Linux command line is similar to my Windows skill - modest me). So I had to experiment with set “for /f” parameters (old ms-dos).

Below is a simple example of how you can get Load Balancer DNS name extracted from JSON generated with aws elb describe-load-balancers --load-balancer-names <load balancer name>. Make sure you specify only one load-balancer – if your file has data for multiple Load Balancers – the variable will be set to the last found in the file.

To run it you have to use .bat file – it will not run as a simple command line command.

@echo off
 for /f usebackq^ tokens^=2^,4^ delims^=^" %%a in ("blog.json") do ( if /i "%%a"=="DNSName" set "myelbhost=%%b"    )
 echo %myelbhost%

While writing this article I just discovered (thank you to this article) that –-query parameter can be used for a-la-unix “piping” aka “|” – however you still have to create a batch file – no direct command will be accepted. More details on --query can be found here, Below is the code that needs to be put into batch file. Please note that quotes, double quotes and some other formatting of query parameters are slightly different. I believe it’s Windows specifics vs Linux ones.

@echo off

for /f "delims=" %%A in ('aws elb describe-load-balancers --load-balancer-names <load balancer name> --query LoadBalancerDescriptions[*].{URL:DNSName} --output text') do set "myelbhost1=%%A"
echo %myelbhost1%

Hope it will help to some of you searching for elegant solution of JSON to Variable problem

Lets conclude – Deployment can be done through:

•           AWS UI - easy but not automated not human error-prone
•           Windows command-line with AWS CLI tools installed (or batch file)
•           Linux command-line with AWS CLI tools installed (or through Linux-shell script)
•           The same can be achieved through AWS SDK that uses very well documented API calls
•           Additionally, CloudFormation template can be used to deploy MMB or if a specific application is used than Beanstalk can be leveraged to automate more – here is a good high-level explanation of how these two work together. 

There is probably more such tools and I’d like to hear your ideas on what else can be used to automate deployment in AWS.

I’ll do my best to cover long-promised Puppet in my next post.

PS And thank you to the code formatter for saving me some time writing this