Getting Familiar with Amazon EC2

Web posted: April 15, 2015 Due: April 24 2015.

The main goal of this hands-on homework is to become faimilar with the Amazon elastic compute cloud (EC2) platform.

The exercise involves a series of steps. Please follow the instructions carefully to execute these steps. Each step provides familiarity with a certain functionality provided by EC2.

You will need security credentials (certificates and keys) to carry out the assignment. The TA will provide these credentials to you soon.

Please turn in a short (1-2 page) report with your observations and output from some of the key steps in this assignment. You will also required to provide an estimate (approximate estimates are OK) on how much your EC2 operations will cost the instructor. Use Amazon's pricing policy and monitoring data to compute your estimates. Please remember that we are paying actual money to Amazon based on your usage -- so be very careful with the resources you use, and if you are unclear on any aspect, ask us for clarifications.

WARNING: All users in this class will be placed in the same security group - which means that you have privileges to terminate a server started by another fellow student. Please do not abuse these privileges.

We may add further explanations on some of these steps to clarify - we will notify the class by email if we do so.

Step 1: Install EC2 tools

We will download and install the tools in the EDlab machines, but for those who want to create an environment in their local machine, here is what you need to do (Linux/Unix only; might work on Macs with suitable modifications)

if you are using your edlab account edit your ~/.cshrc and add the mentioned lines at the end of the file and you are all set

setenv EC2_BASE0 /courses/cs600/cs677/cs677.s2014/EC2
setenv EC2_HOME ${EC2_BASE0}/ec2api
setenv AWS_CLOUDWATCH_HOME ${EC2_BASE0}/cloudWatch
set path=( $path $EC2_HOME/bin $AWS_CLOUDWATCH_HOME/bin)

create a directory in your home and copy all your keys over there; say the name of the directory is EC2 and then run the following commands to set your EC2 credentials (or you can append it to your ~/.cshrc ) - remember to fix the actual filenames to fit the files given to your group (see below)
```
setenv EC2_BASE ~/EC2
setenv EC2_PRIVATE_KEY $EC2_BASE/upen_pk.pem
setenv EC2_CERT $EC2_BASE/upen_cert.pem
setenv AWS_CREDENTIAL_FILE $EC2_BASE/upen_aws_credential_file
setenv JAVA_HOME /usr/lib/jvm/default-java
  
```
If you are installing on your local machine:
1. Install Java
2. Download ec2-tools EC2 API Tools
3. Install ec2-tools and setup the environment variables:
```
  $ cd ~/; mkdir EC2
  $ export EC2_BASE=~/EC2
  $ cd $EC2_BASE
  $ unzip ec2-api-tools.zip
  $ ln -s ec2-api-tools-1.4.0.2 ec2-api
  $ export EC2_HOME=$EC2_BASE/ec2-api
```
4. please also add the export statements in your ~/.bash_profile (or ~/.bashrc).
5. On Linux and UNIX, you can update your PATH as follows.
```
    $ export PATH=$PATH:$EC2_HOME/bin 
    
```

Each of you will be given a set of four files. Let us assume that you are group1.

group1_pk.pem (which will look like this)

group1_cert.pem (which will look like this)

group1_pair.pem (which will look like this)

KEYPAIR group1 01:f8:73:c5:63:e7:02:50:a3:a9:xx:xx:xx:xx:xx:xx
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAknnSdVJUuO1QBHV3/3eHw4OKUPtJ+fIgjuhxE0b/aMaSYADJh6zhnGoiSWFv
rzEqxFjPXW1RlKao6/tj+rrBBYQcYH56YX/seQCPUzbfa3U0y/g/id9VN+thnwomWAHooKlWgJLa
foDRpy5TaoS+ouhuV+Zme8qVCOrVd7iCb9ll+s+RrnakusKUYYz8kA6mUf+vGl/lbJZmHQgRMnaV
+FyePvd88wbpJwXtweJCnDIuQe5gWHV9G/4EWCgtF0eoSvlbx2abhcwvTkSUeNStn3bEeTES3wAn
5CIwPLrWyZg+c0w/eM6yyM8E18Rygb2I3WuOuk9ZhwP0fyE9Emd4twIDAQABAoIBAFQ8PLz9qP1k
dGTsDSNcYTYd2phxuvxDs1eb8xNL/CwMLjeJnFwrHt/t5WPf5fORSe/6xt4KtSTQp0p2muuhJMoA
TLVtZwHVKG/503LelqPclo1KaXfneMGc2to0YxKvnlFusPFmAsNHX8pOCOi0CFH1ayeft4YvXX/w

[only part of the key is listed for security reasons]
-----END RSA PRIVATE KEY-----

group1_ids (which will look like this)
```
AKIKI6YAKCM3UQLHOXJO
DvUNt2aXZgGwoPw/2ZJK3TuyE8ZUdmduqZGarOB4
arn:aws:iam::101240882738:user/group1
FIAAI3QY3XPDSCJ2CPXEI
```
You must create your AWS_CREDENTIAL_FILE mentioned above using the info in this file.
The AccessKeyId is on the first line and the Secret Key on the second line. Create your credentials file with the following information (using the ids in the ids file):
```
AWSAccessKeyId=AKIKI6YAKCM3UQLHOXJO
AWSSecretKey=DvUNt2aXZgGwoPw/2ZJK3TuyE8ZUdmduqZGarOB4
```
Then put the file where referenced by your environment variables (e.g., in .bashrc/.cshrc):
```
export AWS_CREDENTIAL_FILE=/home/.../the_file_you_created_above
```

Setting up the security credentials;

    # group1_pk.pem and group1_cert.pem will be provided to each group by the TA
    # mkdir $EC2_BASE/keys
    # cp group1_pk.pem group1_cert.pem $EC2_BASE/keys
    export EC2_PRIVATE_KEY=$EC2_BASE/keys/group1_pk.pem
    export EC2_CERT=$EC2_BASE/keys/group1_cert.pem
    export AWS_CREDENTIAL_FILE=$EC2_BASE/users/group1_aws_credential_file

Test your environment by running ec2-describe-regions command

    $ ec2-describe-regions
    REGION  eu-west-1       ec2.eu-west-1.amazonaws.com
    REGION  us-east-1       ec2.us-east-1.amazonaws.com
    REGION  ap-northeast-1  ec2.ap-northeast-1.amazonaws.com
    REGION  us-west-1       ec2.us-west-1.amazonaws.com
    REGION  ap-southeast-1  ec2.ap-southeast-1.amazonaws.com

Step 2: Create an Instance and record its approximate starting time

In this step, you will start up a new Linux server on the EC2 cloud. A server is refered to as an EC2 instance. To start a server, you need to specify a machine image (think of it as a boot disk). Machine images are called AMI (amazon machine image) in EC2 terminology.

One can create an EC2 instance by specifying an ami-id. But before you create an instance you need to know what kind of ami to use. This is how you would do it

Find the right AMI (Amazon provides several default machine images that you can use to start your server; you can also create your own images as we will see in a later step).

The following command lists your AMIs and Amazon's public AMIs. The following example shows only part of the resulting output.

$ ec2-describe-images -o self -o amazon

 
IMAGE   ari-96c527ff    ec2-public-images/initrd-2.6.21.7-2.ec2.v1.2.fc8xen.i686.ari.manifest.xml   amazon      available       public          i386
ramdisk                         instance-store
IMAGE   ari-56ce2c3f    ec2-public-images/initrd-2.6.21.7-2.ec2.v1.2.fc8xen.x86_64.ari.manifest.xml amazon      available       public          x86_64
ramdisk                         instance-store
IMAGE   aki-9b00e5f2    ec2-public-images/vmlinuz-2.6.18-xenU-ec2-v1.0.i386.aki.manifest.xml    amazon  available       public          i386    kernel
instance-store

We will use the AMI with ID ami-f8b35e91 to create your instance; when creating a server, EC2 allows you to specify a security policy to indicate which ports are open, which ones are blocked, etc. For now we will use the default security group that allows all ports. Please do not create bigger instances as every time an instance is created the account is charged (even if it is immediately terminated)

You can create an instance of this AMI as follows: In this example the key of user upen (-k upen) but for each group it shall be grou$id (for group-1 it will be group1)

ec2-run-instances ami-f8b35e91 -k upen -t m1.small  -n 1
RESERVATION     r-9ad0a0f7      168263115459    default
INSTANCE        i-e06b948f      ami-f8b35e91                    pending upen    0               m1.small        2011-03-19T12:37:18+0000  us-east-1c      aki-a71cf9ce    ari-a51cf9cc            monitoring-disabled                              instance-store

Check whether the instance is up using ec2-describe-instances with your instance-id

  $ ec2-describe-instances i-e06b948f
RESERVATION     r-9ad0a0f7      168263115459    default
INSTANCE        i-e06b948f      ami-f8b35e91    ec2-67-202-xx-xx.compute-1.amazonaws.com        ip-10-202-xx-x.ec2.internal      running upen    0               m1.small        2011-03-19T12:37:18+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc              monitoring-disabled     67.202.xx.xx    10.202.xx.xx                   instance-store           paravirtual      xen

Notice that my key-name (upen) is mentioned in the seventh column.

Upon booting and running, it will report the status of running and also its external name (e.g. ec2-50-16-2-138.compute-1.amazonaws.com). Record the approximate time it took for the instance to get created.
Record the approximate time it took to create the instance.
Step 3: ssh into the Newly created instance
Next you will log into your newly created server.
- ssh into the instance using your private key:
```
 $ ssh -i upen_pair.pem root@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
  
```
  Note that upen_pair.pem is used to login into the machine; replace the xx with the hostname of your instance.
Step 4: Write a small program which uses 100% CPU on your instance
If you do not wish to write your own ... you can use this small script consumeCPU. This will run as a daemon and will compute factorials of random numbers from 1 -100.
Step 5: Monitor your EC2 instance
monitor various metrics of your EC2 instance and also monitor the CPU utilization after starting your program.

install monitoring tools

download command line tools from here

unzip them and create the right environmet as shown below (edlab account users do not have to do this)

$ cd $EC2_BASE
$ unzip CloudWatch-2010-08-01.zip
$ ln -s CloudWatch-1.0.9.5/ cloudWatch
## you will have to put things in .bashrc for edlab machines
$ vi ~/.bash_profile

export export AWS_CLOUDWATCH_HOME=$EC2_BASE/cloudWatch
export export PATH=$PATH:$EC2_HOME/bin:$AWS_CLOUDWATCH_HOME/bin

checking what metrics can be monitored:

$ mon-list-metrics
CPUUtilization        AWS/EC2  {InstanceId=i-e028d78f}
CPUUtilization        AWS/EC2  
CPUUtilization        AWS/EC2  {InstanceType=t1.micro}
CPUUtilization        AWS/EC2  {InstanceId=i-e06b948f}
DiskReadBytes         AWS/EC2  {ImageId=ami-8c1fece5}
DiskReadBytes         AWS/EC2  
DiskReadBytes         AWS/EC2  {InstanceId=i-e028d78f}
DiskReadBytes         AWS/EC2  {InstanceType=t1.micro}
DiskReadBytes         AWS/EC2  {InstanceId=i-e06b948f}
DiskReadOps           AWS/EC2  {InstanceId=i-e06b948f}
DiskReadOps           AWS/EC2  {InstanceType=t1.micro}
DiskReadOps           AWS/EC2  {InstanceId=i-e028d78f}
DiskReadOps           AWS/EC2  
DiskReadOps           AWS/EC2  {ImageId=ami-8c1fece5}
DiskWriteBytes        AWS/EC2  {InstanceId=i-e028d78f}
DiskWriteBytes        AWS/EC2  {InstanceType=t1.micro}
DiskWriteBytes        AWS/EC2  {ImageId=ami-8c1fece5}
DiskWriteBytes        AWS/EC2  {InstanceId=i-e06b948f}
DiskWriteBytes        AWS/EC2  
DiskWriteOps          AWS/EC2  {InstanceId=i-e06b948f}
DiskWriteOps          AWS/EC2  {ImageId=ami-8c1fece5}
DiskWriteOps          AWS/EC2  
DiskWriteOps          AWS/EC2  {InstanceType=t1.micro}
DiskWriteOps          AWS/EC2  {InstanceId=i-e028d78f}
NetworkIn             AWS/EC2  
NetworkIn             AWS/EC2  {InstanceType=t1.micro}
NetworkIn             AWS/EC2  {InstanceId=i-e06b948f}
NetworkIn             AWS/EC2  {ImageId=ami-8c1fece5}
NetworkIn             AWS/EC2  {InstanceId=i-e028d78f}
NetworkOut            AWS/EC2  
NetworkOut            AWS/EC2  {ImageId=ami-8c1fece5}
NetworkOut            AWS/EC2  {InstanceId=i-e06b948f}
NetworkOut            AWS/EC2  {InstanceType=t1.micro}
NetworkOut            AWS/EC2  {InstanceId=i-e028d78f}
VolumeIdleTime        AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeQueueLength     AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeReadBytes       AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeReadOps         AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeTotalReadTime   AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeTotalWriteTime  AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeWriteBytes      AWS/EBS  {VolumeId=vol-c97b59a1}
VolumeWriteOps        AWS/EBS  {VolumeId=vol-c97b59a1}

Before actually monitoring your instance, you will required to enable monitoring
```
$ ec2-monitor-instances i-e06b948f
i-e06b948f      monitoring-pending
```

once monitoring has started, check some monitoring metric, say CPUUtilization (metrics can be found using the command mentioned above)

$ mon-get-stats CPUUtilization  --period 60 --statistics "Average" --namespace "AWS/EC2" --dimensions "InstanceId=i-e06b948f"

2011-03-19 12:39:00  0.43200000000000005  Percent
2011-03-19 12:44:00  0.0                  Percent
2011-03-19 12:49:00  0.088                Percent
2011-03-19 12:54:00  0.0                  Percent
2011-03-19 12:59:00  0.176                Percent
2011-03-19 13:04:00  0.0                  Percent
2011-03-19 13:09:00  0.088                Percent
2011-03-19 13:15:00  0.0                  Percent
2011-03-19 13:16:00  0.0                  Percent
2011-03-19 13:17:00  0.0                  Percent
2011-03-19 13:18:00  0.0                  Percent
2011-03-19 13:19:00  0.0                  Percent

more details on using this api are available here

Step 6: Install an application

Since we used a default machine image, we can customize our barebones linux sever by installing an application. Doing so will also incur network I/O cost (for downloading the application) and you will need to figure out how much cost this will add to the total bill.

The idea is to make some network usage and monitor the network usage results and report the final cost increase. Lets install firefox and java on the server (around 56M of data)

yum install -y perl java-1.6.0-openjdk.i386 firefox

Report the network usage using mon-get-stats and also the total amount it will cost; you can monitor the network utilization (by giving start-time and end-time as specified in teh guide).

$ mon-get-stats NetworkIn  --period 60 --statistics "Average" --namespace "AWS/EC2" --dimensions "InstanceId=i-e06b948f"

2011-03-19 12:39:00  165.2        Bytes
2011-03-19 12:44:00  0.0          Bytes
2011-03-19 12:49:00  0.0          Bytes
2011-03-19 12:54:00  0.0          Bytes
2011-03-19 12:59:00  4672.0       Bytes
2011-03-19 13:04:00  0.0          Bytes
2011-03-19 13:09:00  0.0          Bytes
2011-03-19 13:15:00  0.0          Bytes
2011-03-19 13:16:00  0.0          Bytes
2011-03-19 13:17:00  0.0          Bytes
2011-03-19 13:18:00  0.0          Bytes
2011-03-19 13:19:00  0.0          Bytes
2011-03-19 13:20:00  0.0          Bytes
2011-03-19 13:21:00  0.0          Bytes
2011-03-19 13:22:00  0.0          Bytes
2011-03-19 13:23:00  0.0          Bytes
2011-03-19 13:24:00  0.0          Bytes
2011-03-19 13:25:00  25634.0      Bytes
2011-03-19 13:26:00  9.3764883E7  Bytes

Step 7: Create a new Image from running instance -- record the time of each operation

Now we can take our customized server and create a new machine image using its disk state. This will allow us to create a new future instance with these customized applications preinstalled.

We shall bundle the volume using ec2-bundle-vol command; record the time of operation and size of the bundled image in /mnt.

Note that user_id specified as -u 168263115459 remains unchanged for all users and if your are using the same AMI mentioned above then the kernel and ramdisk options also remain the same.

First, ssh into the machine

$ ssh -i $EC2_BASE/keys/group1_pair.pem  root@ec2-67-202-56-22.compute-1.amazonaws.com

Install the ec2-ami command line tools ... easiest is to download the rpm from here and install it as this

$  yum localinstall ec2-ami-tools.noarch.rpm  --nogpgcheck

now put your keys on the machine ... basically scp your keys to your instance and create the EC2 environment as mentioned in the steps earlier and check by running ec2-describe-instances to see if everything is working.

$ time ec2-bundle-vol -k $EC2_BASE/keys/upen_pk.pem -c $EC2_BASE/keys/upen_cert.pem -u168263115459  --arch i386 --kernel   aki-a71cf9ce  --ramdisk ari-a51cf9cc -d /mnt

Copying / into the image file /mnt/image...
Excluding: 
         /sys
         /proc
         /dev/pts
         /proc/sys/fs/binfmt_misc
         /dev
         /media
         /mnt
         /proc
         /sys
         /mnt/image
         /mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.008404 seconds, 125 MB/s
mke2fs 1.39 (29-May-2006)
Bundling image file...
Splitting /mnt/image.tar.gz.enc...
Created image.part.00
Created image.part.01
Created image.part.02
Created image.part.03
[...]
Created image.part.68
Generating digests for each part...
Digests generated.
Unable to read instance meta-data for ancestor-ami-ids
Unable to read instance meta-data for product-codes
Creating bundle manifest...
ec2-bundle-vol complete.

real    13m41.641s
user    3m1.015s
sys     1m6.916s

Then upload to S3 (-b contains the name of the S3 bucket where you want to upload the image):

$ time ec2-upload-bundle --batch -b CS677/upen -m /mnt/image.manifest.xml -a $AWSAccessKeyId -s $AWSSecretKey

The specified bucket is not S3 v2 safe (see S3 documentation for details):
CS677
Creating bucket...
Uploading bundled image parts to the S3 bucket CS677 ...
Uploaded image.part.00
Uploaded image.part.01
Uploaded image.part.02
Uploaded image.part.03
[...]
Uploaded image.part.68
Uploading manifest ...
Uploaded manifest.
Bundle upload completed.

real    2m33.281s
user    0m4.140s
sys     0m2.192s

Finally, register the image to start new instances from it:

$ time ec2-register -K $EC2_BASE/keys/upen_pk.pem -C $EC2_BASE/keys/upen_cert.pem -n Centos5.4-upen CS677/upen/image.manifest.xml

IMAGE   ami-2a748643

real    0m9.295s
user    0m2.240s
sys     0m0.088s

check if the AMI has gotten registered ...

$ ec2-describe-images

IMAGE   ami-2a748643    168263115459/Centos5.4-upen     168263115459    available       private         i386    machine aki-a71cf9ce      ari-a51cf9cc            instance-store

Step 8: Create an instance of this new image and terminate the current instance -- record time of each operation

$ ec2-run-instances ami-2a748643 -k upen -t m1.small  -n 1

RESERVATION     r-a40a79c9      168263115459    default
INSTANCE        i-36fe0059      ami-2a748643                    pending upen    0               m1.small        2011-03-19T15:21:02+0000  us-east-1c      aki-a71cf9ce    ari-a51cf9cc            monitoring-disabled                              instance-store                                           xen

Step 9: Compute the cost of this entire operation

Use Amazon's pricing policy to compute the cost of all operations you performed in this assignment---take into account the time for which your EC2 instances ran, cost of network I/O, cost of S3 operations etc.

Step10: Terminate all your instances ...

Be sure to terminate only your instances; since all projects groups are in the same AWS security group, you can indvertanly terminate someone else's instanes as well, so be careful with this command.

$ec2-describe-instances

RESERVATION     r-9ad0a0f7      168263115459    default
INSTANCE        i-e06b948f      ami-f8b35e91    ec2-67-202-56-22.compute-1.amazonaws.com        ip-10-202-162-3.ec2.internal      running upen    0               m1.small        2011-03-19T12:37:18+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc              monitoring-enabled      67.202.56.22    10.202.162.3                    instance-store           paravirtual      xen
RESERVATION     r-a40a79c9      168263115459    default
INSTANCE        i-36fe0059      ami-2a748643    ec2-50-17-104-128.compute-1.amazonaws.com       ip-10-114-45-185.ec2.internal     running upen    0               m1.small        2011-03-19T15:21:02+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc              monitoring-disabled     50.17.104.128   10.114.45.185                   instance-store           paravirtual      xen

$ ec2-terminate-instances i-e06b948f i-36fe0059

INSTANCE        i-36fe0059      running shutting-down
INSTANCE        i-e06b948f      running shutting-down

Last modified: Mon Mar 21 07:14:21 EDT 2011