The main goal of this hands-on homework is to become faimilar with the Amazon elastic compute cloud (EC2) platform.

The exercise involves a series of steps. Please follow the instructions carefully to execute these steps. Each step provides familiarity with a certain functionality provided by EC2.

You will need security credentials (certificates and keys) to carry out the assignment. The TA will provide these credentials to you soon.

Please turn in a short (1-2 page) report with your observations and output from some of the key steps in this assignment. You will also required to provide an estimate (approximate estimates are OK) on how much your EC2 operations will cost the instructor. Use Amazon's pricing policy and monitoring data to compute your estimates.

In your report, you should clearly mention which EC2 api commands you tried, and All the AMI and instance IDs that you created.

Please remember that we are paying actual money to Amazon based on your usage -- so be very careful with the resources you use, and if you are unclear on any aspect, ask us for clarifications.

WARNING: All users in this class will be placed in the same security group - which means that you have privileges to terminate a server started by another fellow student. Please do not abuse these privileges.

We may add further explanations on some of these steps to clarify - we will notify the class by email if we do so.

Step 1: Install EC2 tools

• Download and extract the ec2-tools EC2 API Tools
• It is convenient to define some environment variables. EC2_HOME represents where your ec2-api-tools are located. We will also use this location for storing your keys (provided by the course instructors). Remember to source ~/.bashrc for these exports to take effect.

  export JAVA_HOME=/usr/lib/jvm/default-java
  export EC2_HOME=Location of ec2-api-tools
  export EC2_CERT=$EC2_HOME/groupID_cert.pem
  export EC2_PRIVATE_KEY=$EC2_HOME/groupID_pk.pem
  export AWS_ACCESS_KEY=your-aws-access-key-id (see below)
  export AWS_SECRET_KEY=your-aws-secret-key (see below)
  export PATH=$PATH:$EC2_HOME
    

• Each of you will be given a set of four files. Let us assume that you are group1. Copy these files to your EC2_HOME
  1. group1_pk.pem (which will look like this)

     -----BEGIN RSA PRIVATE KEY-----
     MIICXgIBAAKBgQDbH8TXzjL6jDTMC/mYfAb6kyUEbzqeQ/Ww94d3Gi/qwUq4nsL8
     rjYBdAJb4H6hyJlHkNf339lr+JKBEzEN5fTXEjRGbSIs8pD3aVEg7DzD8I+vLZZ2
     dj/yMmeM2DMaSdp86/W62fq5sXAFMmF7hpd2Yam1/RYJS6HrgRYppsEw7QIDAQAB
     AoGBAMX5FvRPVRmu0QjFjPUqoXeEt/4grFpPVkxT521BGKfFiUfyEIMQtrRrZrIx
     ytdKkMnq4GUQNrTpORwrBbfj2mdGDqJmXyVVrCYi+mENmsBWma9cozmdoq/HEe49
     
     [only part of the key is listed for security reasons]
     -----END RSA PRIVATE KEY-----
         

  2. group1_cert.pem (which will look like this)

     -----BEGIN CERTIFICATE-----
     MIIDYDCCAsmgAwIBAgIJAIztM80g1y3gMA0GCSqGSIb3DQEBBQUAMH4xCzAJBgNV
     BAYTAlVTMQswCQYDVQQIEwJNQTEQMA4GA1UEBxMHQW1oZXJzdDEOMAwGA1UEChMF
     VU1BU1MxCzAJBgNVBAsTAkNTMQ4wDAYDVQQDEwVDUzY3NzEjMCEGCSqGSIb3DQEJ
     ARYUcGxlYXNldGVzdEBnbWFpbC5jb20wHhcNMTEwMzA3MjEzOTMxWhcNMTEwNDA2
     MjEzOTMxWjB+MQswCQYDVQQGEwJVUzELMAkGA1UECBMCTUExEDAOBgNVBAcTB0Ft
     aGVyc3QxDjAMBgNVBAoTBVVNQVNTMQswCQYDVQQLEwJDUzEOMAwGA1UEAxMFQ1M2
     NzcxIzAhBgkqhkiG9w0BCQEWFHBsZWFzZXRlc3RAZ21haWwuY29tMIGfMA0GCSqG
     SIb3DQEBAQUAA4GNADCBiQKBgQDbH8TXzjL6jDTMC/mYfAb6kyUEbzqeQ/Ww94d3
     
     [only part of the certificate is listed for security reasons]
     -----END CERTIFICATE-----
         

  3. group1_pair.pem (which will look like this)

     KEYPAIR group1 01:f8:73:c5:63:e7:02:50:a3:a9:xx:xx:xx:xx:xx:xx
     -----BEGIN RSA PRIVATE KEY-----
     MIIEowIBAAKCAQEAknnSdVJUuO1QBHV3/3eHw4OKUPtJ+fIgjuhxE0b/aMaSYADJh6zhnGoiSWFv
     rzEqxFjPXW1RlKao6/tj+rrBBYQcYH56YX/seQCPUzbfa3U0y/g/id9VN+thnwomWAHooKlWgJLa
     foDRpy5TaoS+ouhuV+Zme8qVCOrVd7iCb9ll+s+RrnakusKUYYz8kA6mUf+vGl/lbJZmHQgRMnaV
     +FyePvd88wbpJwXtweJCnDIuQe5gWHV9G/4EWCgtF0eoSvlbx2abhcwvTkSUeNStn3bEeTES3wAn
     5CIwPLrWyZg+c0w/eM6yyM8E18Rygb2I3WuOuk9ZhwP0fyE9Emd4twIDAQABAoIBAFQ8PLz9qP1k
     dGTsDSNcYTYd2phxuvxDs1eb8xNL/CwMLjeJnFwrHt/t5WPf5fORSe/6xt4KtSTQp0p2muuhJMoA
     TLVtZwHVKG/503LelqPclo1KaXfneMGc2to0YxKvnlFusPFmAsNHX8pOCOi0CFH1ayeft4YvXX/w
     
     [only part of the key is listed for security reasons]
     -----END RSA PRIVATE KEY-----
     

  4. group1_ids (which will look like this)

     AKIKI6YAKCM3UQLHOXJO
     DvUNt2aXZgGwoPw/2ZJK3TuyE8ZUdmduqZGarOB4
     arn:aws:iam::101240882738:user/group1
     FIAAI3QY3XPDSCJ2CPXEI
     

     The first line is the access key, and the next one is the secret key. Insert the correct values from the files provided to you of the AWS access key and secret key in your .bashrc,

           export AWS_ACCESS_KEY=AKI....
           export AWS_SECRET_KEY=DvU....
     

• Test your environment by running ec2-describe-regions command

      $ ec2-describe-regions
      REGION  eu-west-1       ec2.eu-west-1.amazonaws.com
      REGION  us-east-1       ec2.us-east-1.amazonaws.com
      REGION  ap-northeast-1  ec2.ap-northeast-1.amazonaws.com
      REGION  us-west-1       ec2.us-west-1.amazonaws.com
      REGION  ap-southeast-1  ec2.ap-southeast-1.amazonaws.com
  

Step 2: Create an Instance and record its approximate starting time

One can create an EC2 instance by specifying an ami-id. But before you create an instance you need to know what kind of ami to use. This is how you would do it

• Amazon provides default Linux machine images, and we will be using ami-22111148. The next step is to choose an instance type, for which we pick the m1.small type.
Please do not create bigger instances as every time an instance is created the account is charged (even if it is immediately terminated)
• You can create an instance of this AMI as follows: In this example the key of user prateeks (-k prateeks) but for each group it shall be grou$id (for group-1 it will be group1)

  ec2-run-instances ami-22111148 -k KeyNameHere -t m1.small 
  

• This will return the instance id, something like i-487587d5. The instance id is the identifier for your instance, and will be required for most of the EC2 commands. After a few seconds, the instance should have started, so check the status of this instance.

    ec2-describe-instances i-487587d5.

  This gives various details about the instance type, who is running it, and the address where you can access the instance, something like ec2-54-197-107-158.compute-1.amazonaws.com .
• Record the approximate time it took to create the instance.

Step 3: ssh into the Newly created instance

• ssh into the instance using your private key pair and the domain name of your instance:

   $ ssh -i groupX_pair.pem ec2-user@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
    

Step 4: Write a small program which uses 100% CPU on your instance

If you do not wish to write your own ... you can use this small script consumeCPU. This will run as a daemon and will compute factorials of random numbers from 1 -100.

Step 5: Monitor your EC2 instance

Monitor various metrics of your EC2 instance and also monitor the CPU utilization after starting your program.
• install monitoring tools
  1. download command line tools from here
  2. unzip them and create the right environmet as shown below (edlab account users do not have to do this)

     $ cd $EC2_BASE
     $ unzip CloudWatch-2010-08-01.zip
     $ ln -s CloudWatch-1.0.9.5/ cloudWatch
     ## you will have to put things in .bashrc for edlab machines
     $ vi ~/.bash_profile
     
     export export AWS_CLOUDWATCH_HOME=$EC2_BASE/cloudWatch
     export export PATH=$PATH:$EC2_HOME/bin:$AWS_CLOUDWATCH_HOME/bin
     
       

• checking what metrics can be monitored:

  $ mon-list-metrics
  CPUUtilization        AWS/EC2  {InstanceId=i-e028d78f}
  CPUUtilization        AWS/EC2  
  

• Before actually monitoring your instance, you will required to enable monitoring

  $ ec2-monitor-instances i-e06b948f
  i-e06b948f      monitoring-pending
  

• once monitoring has started, check some monitoring metric, say CPUUtilization (metrics can be found using the command mentioned above)

  $ mon-get-stats CPUUtilization  --period 60 --statistics "Average" --namespace "AWS/EC2" --dimensions "InstanceId=i-e06b948f"
  
  2011-03-19 12:39:00  0.43200000000000005  Percent
  2011-03-19 12:44:00  0.0                  Percent
  

  more details on using this api are available here

  Step 6: Install an application

  Since we used a default machine image, we can customize our barebones linux sever by installing an application. Doing so will also incur network I/O cost (for downloading the application) and you will need to figure out how much cost this will add to the total bill.
  The idea is to make some network usage and monitor the network usage results and report the final cost increase. Lets install firefox and java on the server (around 56M of data)

  yum install -y perl java-1.6.0-openjdk.i386 firefox 
  

  Report the network usage using mon-get-stats and also the total amount it will cost; you can monitor the network utilization (by giving start-time and end-time as specified in teh guide).

  $ mon-get-stats NetworkIn  --period 60 --statistics "Average" --namespace "AWS/EC2" --dimensions "InstanceId=i-e06b948f"
  
  2011-03-19 12:39:00  165.2        Bytes
  2011-03-19 12:44:00  0.0          Bytes
  

  Step 7: Create a new Image from running instance -- record the time of each operation

  Now we can take our customized server and create a new machine image using its disk state. This will allow us to create a new future instance with these customized applications preinstalled.
• Use the ec2-create-image command, and give it the instance-id and a name for your new image. The name should be something like 677img_groupId.

  ec2-create-image i-487587d5 -n 677_test_image

  If successful, this returns a new AMI, note down this ami-id.
• check if the AMI has gotten registered ...

  $ ec2-describe-images | grep your-new-ami-id 

  Step 8: Create an instance of this new image and terminate all your instances -- record time of each operation

• You can now terminate your old instance, since we already have created a machine image with your new changes.

   ec2-terminate-instances instance-id 

  You can approximate the time it takes to terminate by looking at the ec2-describe-instances output and see the status of your instance (running/ shutting-down/ stopped/ terminated, etc)
• Now you can start another instance from your customized AMI!

  $ ec2-run-instances new-ami-id -k groupKeyHere -t m1.small
  

• Note that the instance id of this newly created instance is different. Note the new instance id, because the next step is to terminate this new instance as well. Terminate the instance using the ec2-terminate-instances new-instance-id command as before.

Step 9: Compute the cost of this entire operation

Use Amazon's pricing policy to compute the cost of all operations you performed in this assignment---take into account the time for which your EC2 instances ran, cost of network I/O, cost of S3 operations etc.

Step10: Cleaning Up..

Be sure to terminate only your instances; since all projects groups are in the same AWS security group, you can indvertanly terminate someone else's instanes as well, so be careful with this command.
• The last step is to deregister your newly created AMI by

   ec2-deregister ami-id-here 

• Deregistering an AMI does not actually delete it. You need to delete the snapshot associated with the AMI. First find the snapshot by

   ec2-describe-snapshots | grep ami-id-here
  
      SNAPSHOT        snap-beba4fd9   vol-9ccd304d    completed       2016-04-10T19:03:51+0000        100%    101840822718    8       Created by CreateImage(i-487587d5) for ami-9eddcdf4 from vol-9ccd304d 
  
    

• We are finally ready to delete this snapshot. Note the snapshot ID from the previous command and then:

      ec2-delete-snapshot snap-beba4fd
    

---