Load Testing With EC2 and JMeter

Submitted by ian on Thu, 07/30/2009 - 16:46

I need to load test a website. I want to really stress it, i.e. testing it from my desktop just won't cut it. So I've decided to load test it using servers in the EC2 cloud. This is a great option because

  • it can scale very well (I can requisition as many hosts as I like)
  • I can see how the site is performing under load from my desktop easily (my network is not clogged with traffic)

I'm going to use JMeter for the load testing.

My plan is to do the following:

  1. Create an AMI (Amazon Machine Instance) with the necessary software to run the tests
  2. Create a JMeter test plan
  3. Write a shell script to do it all for me. The shell script will take the following parameters: number of hosts to requisition, the JMeter test plan file, probably some more stuff too

Here we go!

Create AMI
I'm going to start with a base AMI from http://alestic.com. The Ubuntu 9.04 Jaunty 32 bit server will do just fine (ami-0d729464). I've already installed the EC2 command line tools (see http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351).

ec2-run-instances -g ssh -k rcc_test_001 -t m1.small -n 1 ami-0d729464

I've previously created a group that only allows ssh (that's the "-g ssh" option) and a key named rcc_test_001 (that's the "-k rcc_test_001" option).

Give it a minute or two and you can see your instance if you do a

$ ec2-describe-instances

Here's what mine said

RESERVATION     r-8b3c61e2      936911469460    ssh
INSTANCE        i-c59099ac      ami-0d729464    ec2-67-202-3-9.compute-1.amazonaws.com  domU-12-31-38-00-51-A1.compute-1.internal       running rcc_test_001   0m1.small        2009-07-30T23:58:51+0000        us-east-1c      aki-a71cf9ce   ari-a51cf9cc             monitoring-disabled

Log into the new host and get it up to date

$ ssh -i .ec2/rcc_test_001.pem root@ec2-67-202-3-9.compute-1.amazonaws.com
$ apt-get update
$ apt-get -y upgrade
$ apt-get -y vim

I install vim because I get annoyed when it's not on a system I'm working on. This is not necessary.

We need to add Java and JMeter. I've got a Java binary installer and a JMeter archive laying around on my desktop that I'm going to scp to the EC2 host:

$ scp -i .ec2/rcc_test_001.pem /opt/jdk-6u14-linux-i586.bin  root@ec2-67-202-3-9.compute-1.amazonaws.com:/opt
$ scp -i .ec2/rcc_test_001.pem /opt/jakarta-jmeter-2.3.4.tgz  root@ec2-67-202-3-9.compute-1.amazonaws.com:/opt

There are other ways to get these onto your host, but this was the easiest for me.

On your EC2 host, install Java and JMeter:

$ cd /opt
$ chmod 700 jdk-6u14-linux-i586.bin
$ ./jdk-6u14-linux-i586.bin
$ tar -xzf jakarta-jmeter-2.3.4.tgz
$ cd
$ echo 'export JAVA_HOME=/opt/jdk1.6.0_14' >>.bashrc
$ echo 'export JMETER_HOME=/opt/jakarta-jmeter-2.3.4' >>.bashrc
$ echo 'export PATH=$JAVA_HOME/bin:$JMETER_HOME/bin:$PATH' >>.bashrc

Now let's build the AMI. First, you'll need your EC2 private key and cert files on the server. You can put them in the /tmp directory as that dir will not be included in the AMI.

$ scp -i .ec2/rcc_test_001.pem  .ec2/* root@ec2-67-202-3-9.compute-1.amazonaws.com:/tmp

Next, let's clean up the server

$ rm -f /root/.*hist*
$ rm -rf /root/.ssh
$ sudo rm -f /var/log/*.gz
$ sudo find /var/log -name mysql -prune -o -type f -print |  while read i; do sudo cp /dev/null $i; done

You will need a bucket in S3 to put your AIM (I use S3Fox (https://addons.mozilla.org/en-US/firefox/addon/3247) to do this).

Now build and upload the AMI

$ ec2-bundle-vol -r i386 -d /mnt -p http-load-test-ubuntu-9.04-i386-server-2009-07-30 -u 1111-1111-1111 -k /tmp/pk-*.pem -c /tmp/cert-*.pem -s 10240 -e /mnt,/tmp,/root/.ssh
$ ec2-upload-bundle -b bucket-name -m /mnt/http-load-test-ubuntu-9.04-i386-server-2009-07-30.manifest.xml -a S3_ACCESS_KEY_ID -s S3_SECRET_ACCESS_KEY

Register the AMI on your desktop

$ ec2-register bucket-name/http-load-test-ubuntu-9.04-i386-server-2009-07-30.manifest.xml

And you'll get an AMI image ID. You will need to make a note of this. My image ID is ami-71719018 (yours won't be).

Now we can terminate the instance that we were running to create the AMI. From your desktop

$ ec2-terminate-instances i-c59099ac

Your instance ID will be different.

$ ec2-run-instances -g ssh -k rcc_test_001 -t m1.small -n 1 ami-71719018

Here is a shell script I wrote to automate the load tests. It's not very pretty, but it got the job done:

!/bin/bash
 
server_count=8
jmeter_test_plan=/path/to/your/jmeter/test/plan.jmx
jmeter_procs_per_server=1
key_pair_file=/path/to/your/key/pair/file.pem
 
iids=()
 
i=0
while test $i -lt $server_count; do
    echo "Running instance" 1>&2
    ec2-run-instances -g ssh -k rcc_test_001 -t m1.small -n 1 ami-71719018 >/tmp/$$.$i.out
    iids[$i]=$(awk -F'\t' 'NR == 2 {print $2}' /tmp/$$.$i.out)
    rm /tmp/$$.$i.out
    i=$(($i + 1))
done
 
# Wait for all instances to start up
waiting=1
while test $waiting = 1; do
    ec2-describe-instances >/tmp/$$.instances
    waiting=0
    for i in ${!iids[*]}; do
        iid=${iids[$i]}
        status=$(awk -F'\t' -viid=$iid '$1 == "INSTANCE" && $2 == iid {print $6}' /tmp/$$.instances)
        echo $status
        if test $status != 'running'; then
            waiting=1
            echo "Instance $iid has not started yet" 1>&2
            echo "Waiting for instances to start up..." 1>&2
            sleep 5s
            break
        fi
    done
    rm /tmp/$$.instances
done
 
# Get all IPs
ec2-describe-instances >/tmp/$$.instances
ips=()
for i in ${!iids[*]}; do
    iid=${iids[$i]}
    ips[$i]=$(awk -F'\t' -viid=$iid '$1 == "INSTANCE" && $2 == iid {print $4}' /tmp/$$.instances)
done
rm /tmp/$$.instances
 
for ip in ${ips[*]}; do
    scp -o StrictHostKeyChecking=no -i $key_pair_file $jmeter_test_plan root@$ip:/tmp
done
 
for ip in ${ips[*]}; do
    i=0
    while test $i -lt $jmeter_procs_per_server; do
        echo "Starting jmeter process on $ip" 1>&2
        ssh -o StrictHostKeyChecking=no -i $key_pair_file root@$ip "export JVM_ARGS=\"-Djmeterengine.nongui.port=445$i\"; export PATH=/opt/jdk1.6.0_14/bin:\$PATH; /opt/jakarta-jmeter-2.3.4/bin/jmeter -n -t /tmp/$(basename $jmeter_test_plan) -l /tmp/jmeter.log.$i.jtl" &
        i=$(($i + 1))
    done
done
 
echo "IIDs"
echo ${iids[*]}
 
echo "IPs"
echo ${ips[*]}

I should note that I had problems with the environment when I would ssh into the server and run a command e.g. ssh user@host "run this command". It would not read my .bashrc file (or any other files I tried). This is why I had to manually set the PATH in the end of the above script.

Powered by Drupal, an open source content management system

Navigation