EPAM Cloud: Automatic Configuration of Virtual Machines in the Cloud Using Metadata

In God we trust, the rest we automate
— unknown DevOps Engineer

The use of virtualization and cloud platforms allows launching and maintaining IT infrastructure ten time faster. A single person can manage tens, hundreds and even thousands of virtual servers: easily launch them, stop, clone, configure hardware, and use to create pre-configured system images.

In case all of your servers have a similar configuration, it is all quite simple – you can configure a server once, create its image and launch as many machines, as you like using the image. However, if you use multiple operating systems with different software packages, or need to quickly launch and stop complex cluster configurations, maintaining even a few dozens of such servers becomes very time-consuming. Naturally, you can have a set of regularly updated scripts and images for all occasions. Still, it might be more rational to use a single script with several images and provide all required parameters during system launch. Many cloud computing platforms offer a so-called metadata or user-data mechanism. It enables you to pass all necessary data on configuration of a certain virtual machine or even send the whole script to be launched on startup.

This article addresses the following cloud platforms, to some extent:

· Amazon EC2

· Eucalyptus

· Nimbula Director

· VMWare vCloud Director

1. User-Data Concept Review. Use Cases for Different Platforms via CLI or Common Scripts

1.1 Amazon EC2

Amazon allows providing user-data in free format on instance startup. You can later receive them by following a certain link:

Link example:

curl 169.254.169.254/latest/user-data

The IP address above is a virtual one and all requests it receives are forwarded to internal API of the EC2 service, in accordance with the source’s IP address.

All standard system images, provided by Amazon have an embedded ability to perform Bash and Power Shell scripts, transmitted in user-data. If user-data begin with shebang (#!), the system will try and launch the script using the specified interpreter. This ability had initially been implemented in a separate ‘cloud init’ package for Ubuntu, however by now it has been included in all standard system images, including Windows.

Windows allows specifying execution of both common console commands,

netsh advfirewall set allprofiles state off

</script>

and Power Shell code:

$source = "http://www.example.com/myserverconfig.xml"

$destination = "c:\myapp\myserverconfig.xml"

$wc = New-Object System.Net.WebClient

$wc.DownloadFile($source, $destination)

</powershell>

This functionality can be used jointly with Cloud Formation templates to run whole stacks of servers, having provided correct user-data:

{

"AWSTemplateFormatVersion" : "2010-09-09",

"Parameters" : {

"AvailabilityZone" : {

"Description" : "Name of an availability zone to create instance",

"Default" : "us-east-1c",

"Type" : "String"

"KeyName" : {

"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",

"Default" : "test",

"Type" : "String"

"InstanceSecurityGroup" : {

"Description" : "Name of an existing security group",

"Default" : "default",

"Type" : "String"

}

"Resources" : {

"autoconftest" : {

"Type" : "AWS::EC2::Instance",

"Properties" : {

"AvailabilityZone" : { "Ref" : "AvailabilityZone" },

"KeyName" : { "Ref" : "KeyName" },

"SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }],

"ImageId" : "ami-31308xxx",

"InstanceType" : "t1.micro",

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[

"#!/bin/bash","\n",

"instanceTag=WebServer","\n",

"confDir=/etc/myconfig","\n",

"mkdir $confDir","\n",

"touch $confDir/$instanceTag","\n",

"IPADDR=$(ifconfig eth0 | grep inet | awk '{print $2}' | cut -d ':' -f 2)","\n",

"echo $IPADDR myhostname","\n",

"hostname myhostname","\n" ]]

}

Sometimes launching a script on instance launch can be unfit for you purpose. Suppose you want other people to use your images without the need to sort out your code. In this case, you can install the script into the system, add it to startup list and create a system image.

Then you should provide your users a description of all parameters that can be provided using user-data. The following example is a list of key=value parameters, separated by semicolons:

graylogserver=«192.168.1.1»;chefnodename=«chef_node_name1»;chefattributes=«recipe1.attribute1=value1,recipe1.attribute2=value2,customparameter1=value1»;chefserver=«192.168.1.38:4000»;chefrole=«apache,mysql,php»;

The whole line can be obtained using Bash as follows:

function get_userdata {

user_data=$(curl -w "%{http_code}" -s http://169.254.169.254/latest/user-data)

result_code=${user_data:(-3)}

if [ -z "$user_data" ] || [ $result_code != "200" ]

then

echo "$CurrentDate: Couldn't receive user-data. Result code: $result_code"

return 1

else

export user_data=${user_data%%$result_code}

return 0

}

You will then be able to get the required value from the received list:

function get_userdata_value {

IFS=';'

for user_data_list in $user_data

user_data_name=${user_data_list%%=*}

if [ $user_data_name = $1 ]

then

user_data_value=${user_data_list#*=}

user_data_value=$(echo $user_data_value | tr -d '\"')

return 0

done

return 1

}

This done, you can proceed with system configuration in accordance with the received data. You do not have to store all scripts inside an image. It is enough that you have a startup script that reads user-data and then downloads and launches all necessary parameters or delegates control to Chef or Puppet.

Similar functionality can be implemented using Power Shell.

1.2 Eucaliptus
This product is compatible with Amazon AWS and shares its user-data implementation.

1.3 Nimbula
The system is relatively young and rapidly developed. Nimbula’s purpose is creation of private cloud systems using KVM virtualization. Its creators, once Amazon employees themselves, announced full AWS compatibility. However, it not so. They support user-data mechanism over virtual IPs, yet it should be provided in the ‘key=value’ fashion.
The list of all keys is available at:
192.0.0.192/latest/attributes or 169.254.169.254/latest/attributes

Example:
curl 169.254.169.254/latest/attributes
nimbula_compressed_size
nimbula_decompressed_size
chefserver
hostname

Getting the exact key value:
curl 169.254.169.254/latest/attributes/chefserver
192.168.1.45:4000

You cannot, however, pass whole scripts this way. To do this you should create your own system images with an embedded startup script.

Bash code example:

function get_value {

user_data_value=$(curl curl -w "%{http_code}" -s http://169.254.169.254/latest/attributes/"$1")

result_code=${user_data_value:(-3)}

if [ -z "$user_data_value" ] || [ $result_code != "200" ]

then

echo "$CurrentDate: $1 variable is not set, skip it, return code: $result_code" >> $LogFile

return 1

else

user_data_value=${user_data_value%%$result_code}

return 0

}

1.4 VMWare vCloud Director
Starting from version 1.5 vCloud Director supports use of metadata within vApp (VM container). Data must be specified as ‘key=value’. In order to provide metadata you should create an XML-file with their description:

<Key>app-owner</Key>

</MetadataEntry>

<Key>app-owner-contact</Key>

</MetadataEntry>

<Key>system-owner</Key>

</MetadataEntry>

</Metadata>

Next step is executing a POST request via URL of the respective vApp:

$ curl -i -k -H «Accept:application/*+xml;version=1.5» -H «x-vcloud-authorization: jmw43CwPAKdQS7t/EWd0HsP0+9/QFyd/2k/USs8uZtY=» -H «Content-Type:application/vnd.vmware.vcloud.metadata+xml» -X POST 10.20.181.101/api/vApp/vapp-1468a37d-4ede-4cac-9385-627678b0b59f/metadata -d @metadata-request.

You can read all metadata with the following GET request:

$ curl -i -k -H «application/*+xml;version=1.5» -H «x-vcloud-authorization: jmw43CwPAKdQS7t/EWd0HsP0+9/QFyd/2k/USs8uZtY=» -X GET 10.20.181.101/api/vApp/vapp-1468a37d-4ede-4cac-9385-627678b0b59f/metadata.

In order to read value of a certain key your request must be arranged like this:
$ curl -i -k -H «application/*+xml;version=1.5» -H «x-vcloud-authorization: jmw43CwPAKdQS7t/EWd0HsP0+9/QFyd/2k/USs8uZtY=» -X GET 10.20.181.101/api/vApp/vapp-1468a37d-4ede-4cac-9385-627678b0b59f/metadata/asset-tag

the answer will be provided as XML.

Details on metadata in vCloud are described here: blogs vmware

2. User-Data Management Using Chef, Puppet and Similar Systems

2.1 Chef
Chef-client installation is up to you: you can install it manually and create system images afterwards, or you can install it automatically on system startup. Both ways have their pros and cons: the first one reduces time on instance startup configuration, while the second one enables you to install the latest client version, or the one that suits you best, whatever works for you. In any case, we should send a list of roles and recipes to be executed on the machine (this list can be obtained via user-data) and configure our Chef client on system startup. Also, in case we do install and configure the client on system startup, we should download the validation.pem key for the respective Chef server (the data on which can be passed using user-data)

Below is an example of Bash script retrieving a list of roles:

rolefile="/etc/chef/role.json"

function get_role {

get_value "chefrole"

if [ $? = 0 ]

then

chefrole=$user_data_value

else

echo "$CurrentDate: Couldn't get any Chef role, use base role only."

chefrole="base"

commas_string=${chefrole//[!,]/}

commas_count=${#commas_string}

echo '{ "run_list": [ ' > $rolefile

IFS=","

for line in $ep_chefrole

if [ $commas_count = 0 ]

then

echo "\"role[$line]\" " >> $rolefile

else

echo "\"role[$line]\", " >> $rolefile

commas_count=$(($commas_count-1))

done

echo ' ] }' >> $rolefile

}

It also created a client configuration file:

function set_chef {

if [ -d $chef_dir ] && [ -e $chef_bin ]

then

service $chef_service stop

sleep 10

echo -e "chef_server_url \"http://$1\"" > $chef_dir/client.rb

echo -e "log_location \"$chef_log\"" >> $chef_dir/client.rb

echo -e "json_attribs \"$rolefile\"" >> $chef_dir/client.rb

echo -e "interval $chef_interval" >> $chef_dir/client.rb

echo "$CurrentDate: Writing $chef_dir/client.rb"

service $chef_service start

else

echo "$CurrentDate: Chef directory $chef_dir or chef binary $chef_bin does not exist. Exit."

exit 1

}

The ‘json_attributes’ parameter set a path to the JSON file with a list of roles and recipes.

Once control has been delegated to the Chef client, it will register on a server, download the list of recipes and proceed with their execution. However, there are some nuances:

· It takes a lot of time to execute certain recipes, we should know once system configuration has been finished and whether it has been successful.

· Sometimes we don’t want to execute recipes with default recipes, changing some attributes instead, e.g. install LAMP with Apache working over port 8080, not 80.

The first issue can be solved using a cookbook by Opscode, called ‘chef_handler’. It provides a mechanism, called Exception and Report Handlers. It is called after Chef client finishes executing recipes. This cookbook enables us to check the result of the last client’s execution and perform certain actions. We can send messages containing execution results (example from an Opscode manual) or upload the results to Chef server to check this value using own applications and display the execution status.

Recipe example:

Setting default attribute values

default['lastrun']['state'] = "unknown"

default['lastrun']['backtrace'] = "none"

Specifying what needs to be executed

include_recipe "chef_handler"

chef_handler "NodeReportHandler::LastRun" do

source "#{node.chef_handler.handler_path}/nodereport.rb"

action :nothing

end.run_action(:enable)

Configuring execution

module NodeReportHandler

class LastRun < Chef::Handler

def report

if success? then

node.override[:lastrun][:state] = "successful"

node.override[:lastrun][:backtrace] = "none"

else

node.override[:lastrun][:state] = "failed"

node.override[:lastrun][:backtrace] = "#{run_status.formatted_exception}"

end

node.save

end

As a result we get values of ‘lastrun.state’ and ‘lastrun.backtrace’ parameters initially set as 'unknown' and 'none' respectively, and once the client finishes execution, receive either ‘successful’ or ‘failed’ record, with the error description in lastrun.backtrace.
This recipe should be placed on top of the execution list to cover all errors received during execution of any recipes.

Change default attributes requires us to get them somehow, then store and start executing recipes. Once again, we can receive those using user-data.

Example of a recipe, retrieving user-data from Amazon:
Getting the whole string

# Get whole user-data string

def GetUserData(url)

uri = URI.parse(url)

http = Net::HTTP.new(uri.host, uri.port)

http.open_timeout = 5

http.read_timeout = 5

proto = url.split(":", 2)

if proto[0] == "https"

http.use_ssl = true

http.verify_mode = OpenSSL::SSL::VERIFY_NONE

end

request = Net::HTTP::Get.new(uri.request_uri)

begin

http.open_timeout = 5

http.read_timeout = 5

request = Net::HTTP::Get.new(uri.request_uri)

response = http.request(request)

result = response.body.to_s

if response.is_a?(Net::HTTPSuccess)

Chef::Log.info("Successfuly get user-data.")

return result

else

return false

end

rescue Exception => e

Chef::Log.info("HTTP request failed.")

return false

end

getting the value of a certain parameter

# Get specified user-data value

def GetValue(user_data,attribute)

user_data.split(";").each do |i|

attribute_name=i.split("=", 2)

if attribute_name[0] == attribute

return attribute_name[1].strip

end

return false

end

Now that we can receive values of certain parameters from passed data, we can set them:

сhefnodename=«chef_node_name1»;chefattributes=«recipe1.attribute1=value1,recipe1.attribute2=value2,customparameter1=value1»;chefserver=«192.168.1.38:4000»;chefrole=«apache,mysql,php»

We used the ‘chefattributes’ parameter to send a list of properties we would like changed. They are set in the ‘cookbookname.attributename=value’ fashion. In order to change the default Apache port, we should set ‘chefattributes=apache.port=8080’.

Below is the recipe that reads this value and saves it:

chefattributes = GetValue("#{node[:user_data]}","chefattributes")

if chefattributes != false

сhefattributes.split(",").each do |i|

attribute_name=i.split("=")

recipe_name=attribute_name[0].split(".", 2)

node.override[:"#{recipe_name[0]}"][:"#{recipe_name[1].strip}"]="#{attribute_name[1].strip}"

Chef::Log.info("Save node attributes.")

node.save

else

Chef::Log.info("Couldn't get Chef attributes. Skip.")

end

This recipe should be executed in the first place.

Drawbacks of the aforementioned recipes

The ‘node.save’ operation sends the whole JSON array for a certain node to a server for storing, including information, gathered by Ohai. Thousands of machines that constantly try to rewrite their attributes on a server will have a negative impact of the server’s performance. This also concerns using a flexible and powerful search, like the one provided by Chef. The search operation is very time-consuming and placing heavy load on a server. In this case we should use different means, not described here.

2.2 Puppet

Receiving user-data with Puppet is similar to using Chef. Addresses of a Puppet server and the rest of necessary data for agent configuration are received by means of a startup script. Facter add-on enables handy uploading of your facts to a server.

Below is an example of a ruby script that receives necessary data from user-data and sends them to a server as additional facts for each machine:

require 'facter'

user_data = `curl http://169.254.169.254/latest/user-data`

user_data = user_data.split(";")

user_data.each do |line|

user_data_key_value = line.split('=', 2)

user_data_key = user_data_key_value[0]

user_data_value = user_data_key_value[1]

Facter.add(user_data_key) do

setcode { user_data_value }

end

instance_id = `curl http://169.254.169.254/latest/meta-data/instance-id`

Facter.add('instance-id') do

setcode { instance_id }

end

Naturally, it can all appear simple and not requiring such complex layouts. You can just create the required set of images with pre-installed software and make all changes using scripts that go by SSH and edit configuration files.

This article, however, covers elementary steps only. In case you require a Hadoop or MySQL cluster, not to mention a cluster of Front-End, Back-End, App, and DB servers, supporting automatic configuration of all machines automatic scalability, you will not be able to go without the aforementioned techniques.

Should you be aware of the ways to transmit metadata for other cloud platforms, or other means of controlling VM configuration on startup, you are welcome to discuss it in the comments section.

Thursday, February 7, 2013

Automatic Configuration of Virtual Machines in the Cloud Using Metadata

No comments:

Post a Comment