Part 3–Load Testing In the Cloud Automating Test Rig Deployment

Is this blog post worth your time?

It’s not challenging setting up the Load Test Rig in the Cloud, it’s more challenging to convince enterprises to embrace the cloud. Not all enterprises will accept their business critical applications being profiled from a VM hosted outside their network. Though through Windows Azure Connect you can join the Cloud hosted Role to the on premise domain effectively forcing the domain policies on the external VM but how many enterprises will authorize you to do this? I have worked with various Energy Trading companies who admire the flexibility that Azure brings but often shy away from the implementation because they still consider it risky. There is no immediate solution to this, but I sure think that the thought process could be transformed by implementing the change gradually. The client may not be willing to experiment with the business critical applications, but you stand a good chance if you can identify the not so important applications and highlight the benefits of Azure-ing the Test Rig such as,

• Zero Administration overhead.
• Since Microsoft manages the hardware (patching, upgrades and so forth), as well as application server uptime, the involvement of IT pros is minimized.
• Near to 100% guaranteed uptime.
• Utilizing the power of Azure compute to run a heavy virtual user load.
• Benefiting from the Azure flexibility, destroy Test Agents when not in use, takes < 25 minutes to spin up a new Test Agent.
• Most important test Network Latency, (network latency and speed of connection are two different things – usually network latency is very hard to test), by placing the Test Agents in Microsoft Data centres around the globe, one can actually test the lag in transferring the bytes not because of a slow connection but because the page has been requested from the other side of the globe.

Welcome to Part 3, In Part 1 we discussed the advantages of taking the Test Rig to the cloud, In Part 2 we manually set up the Test Rig in the cloud, In Part 3 I intent to completely automate the end to end deployment of the Test Agent to the cloud.

Another Azure Gem – Startup Tasks in Worker Role

Let’s get started by what we need to automate the Test Agent deployment to the cloud. We can use startup tasks to perform operations before the role starts. Operations that one might want to perform include installing a component, registering COM components, setting registry keys, or starting a long running process. Read more about start up tasks here. I will be leveraging startup tasks to install Visual Studio Test Agent software and further registering the Test Agent to the controller.

I. Packaging everything in to the Solution

In Part 2 we created a Solution using the Windows Azure Project Template, I’ll be extending the same solution,

DOWNLOAD A WORKING SOLUTION OR follow the instructions below, 

1. StartupTask - Create a new Folder under the WorkerRole1 Project and call it StartupTask.

2. VS2010Agents.zip - If you haven’t already, download the Visual Studio Agents Software from MSDN, I used the ‘en_visual_studio_agents_2010_x86_x64_dvd_509679.iso’ extracted the iso and zipped the TestAgents folder and called it VS2010Agents.zip. Add the VS2010Agents.zip to the folder StartupTask, the approximate size of the VS2010Agents.zip would be around 75 MB.

3. Setup.cmd - Create a new file Setup.cmd and add it to the StartupTask folder. Setup.cmd will just be the workflow manager. The cmd file will call the Setup.ps1 script that we will generate in the next step and log the output to a log file. Your cmd file should look like below.

<span style="color: black"><font color="#00ff00">REM </font></span><span style="color: black"><font color="#00ff00">Allow Power Shell scripts to run unrestricted</font>
powershell -command </span><span style="color: maroon">"set-executionpolicy Unrestricted"

</span><span style="color: black"><font color="#00ff00">REM </font></span><span style="color: black"><font color="#00ff00">Unzip VS2010 Agents
</font>powershell -command </span><span style="color: maroon">".\setup.ps1" </span><span style="color: black">-NonInteractive > out.txt</span>

4. Setup.ps1 - Unzipping VS2010Agents.zip – I found a very helpful post that shows you how to unzip a file using powershell. Create a function Unzip-File that accepts two parameters, basically, the source and destination, verifies that the source location is valid creates the destination location and unzips the file to the destination location. So, your Setup.ps1 file should look like below.

<span style="color: blue">function </span><span style="color: #5f9ea0"><font style="background-color: #ffff00">Unzip-File</font> </span><span style="color: black">{
    </span><span style="color: blue">PARAM </span><span style="color: black">(

        </span><span style="color: purple">$zipfilename</span><span style="color: red">=</span><span style="color: maroon">"E:\AppRoot\StartupTask\VS2010Agents.zip"</span><span style="color: black">,
        </span><span style="color: purple">$destination</span><span style="color: red">=</span><span style="color: maroon">"C:\resources\temp\vsagentsetup" 
    </span><span style="color: black">)

    </span><span style="color: blue">if</span><span style="color: black">(test-path(</span><span style="color: purple">$zipfilename</span><span style="color: black">))
    {    
        </span><span style="color: purple">$shellApplication </span><span style="color: red">= </span><span style="color: #5f9ea0">new-object -com </span><span style="color: maroon">shell.application 
        </span><span style="color: purple">$zipPackage </span><span style="color: red">= </span><span style="color: purple">$shellApplication</span><span style="color: black">.NameSpace(</span><span style="color: purple">$zipfilename</span><span style="color: black">) 
        </span><span style="color: purple">$destinationFolder </span><span style="color: red">= </span><span style="color: purple">$shellApplication</span><span style="color: black">.NameSpace(</span><span style="color: purple">$destination</span><span style="color: black">) 
        </span><span style="color: purple">$destinationFolder</span><span style="color: black">.CopyHere(</span><span style="color: purple">$zipPackage</span><span style="color: black">.Items(), 1040) 
    } 
    </span><span style="color: blue">else 
   </span><span style="color: black">{ 
       </span><span style="color: #5f9ea0">echo </span><span style="color: maroon">"The source ZIP file does not exist"</span><span style="color: black">; 
   }
}

</span><span style="color: #5f9ea0">md </span><span style="color: maroon">"C:\resources\temp\vsagentsetup"

</span><span style="color: #5f9ea0"><font style="background-color: #ffff00">Unzip-File</font> </span><span style="color: maroon">"E:\AppRoot\StartupTask\VS2010Agents.zip" "C:\resources\temp\vsagentsetup"</span>

5. SetupAgent.ini – Unattended Installation - When the setup.ps1 has been run, it would unzip the VisualStudioAgents.zip to C:\resources\temp\vsagentsetup. Now you need to install the Agents software, but you would want to run an unattended Installation. I found a great walkthrough on MSDN that shows you how to create an unattended installation of Visual Studio, I have used this walkthrough to create the setupAgent.ini file, don’t worry if you are having trouble creating the ini file, the entire solution is available for download at the bottom of the post.

Once you have created or downloaded the setupagent.ini file, you need to append instructions to the setup.cmd to start the unattended install using the setupagent.ini that you just created. You can add the below statements to the setup.cmd.

<span style="color: black"><font color="#00ff00">REM Unatteneded Install of Test Agent Software
</font>C:\resources\temp\vsagentsetup\VS2010TestAgents\testagent\setup.exe /q /f /norestart /unattendfile e:\approot\StartupTask\setupagent.ini</span>

6. ConfigAgent.cmd – Now you need to configure the agent to work with the test controller. This is easy to do, you can call the TestAgentConfig.exe from the command line by passing the user details and controller details. The MSDN post here was helpful. Create a new file call it ConfigAgent.cmd, it should look like below,

<span style="color: maroon">"D:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TestAgentConfig.exe" </span><span style="color: black">configureAsService ^</span>

<span style="color: black">/userName:vstestagent /password:P2ssw0rd /testController:TFSSPDEMO:6901</span>

you will need to append instructions to setup.cmd to call the ConfigAgent.cmd and run it as a scheduled task, with elevated permissions. You can add the below statements to the setup.cmd

<span style="color: black"><font color="#00ff00">REM Create a task that will run with full network privileges.</font>
net </span><span style="color: #5f9ea0">start </span><span style="color: maroon">Schedule

</span><span style="color: black">schtasks /CREATE /TN </span><span style="color: maroon">"Configure Test Agent Service" </span><span style="color: black">/SC ONCE /SD 01</span><span style="color: red">/</span><span style="color: black">01</span><span style="color: red">/</span><span style="color: black">2020 ^</span>

<span style="color: black">/ST 00:00:00 /RL HIGHEST /RU admin /RP P2ssw0rd /TR e:\approot\configagent.cmd /F</span>

<span style="color: black"></span>

<span style="color: black">schtasks /RUN /TN </span><span style="color: maroon">"Configure Test Agent Service"</span>

7. Setupfw.cmd – You will need to open up ports for communication between the test controller and the test agent. In normal developer environment at times you can get away by turning off the firewall, however, this will not be a possibility in a corporate network. Found a very useful post that shows you how to configure the Windows 2008 Server Advanced firewall from the CLI. Once you are through the Setupfw.cmd should look like below,

<span style="color: black">netsh advfirewall firewall add rule name</span><span style="color: red">=</span><span style="color: maroon">"RPC" ^</span>

<span style="color: maroon"></span><span style="color: black">            dir=</span><span style="color: blue">in </span><span style="color: black">action=allow service=any enable=yes profile=any localport=rpc protocol=tcp
netsh advfirewall firewall add rule name</span><span style="color: red">=</span><span style="color: maroon">"RPC-EP" ^</span>

<span style="color: maroon"></span><span style="color: black">            dir=</span><span style="color: blue">in </span><span style="color: black">action=allow service=any enable=yes profile=any localport=rpc-epmap protocol=tcp

</span><span style="color: black">.</span>

<span style="color: black">.</span>

<span style="color: black">
netsh advfirewall firewall add rule name</span><span style="color: red">=</span><span style="color: maroon">"Port 6910 TCP" ^</span>

<span style="color: maroon">            </span><span style="color: black">dir=</span><span style="color: blue">in </span><span style="color: black">action=allow service=any enable=yes profile=any localport</span><span style="color: red">=</span><span style="color: black">6910 protocol=tcp
netsh advfirewall firewall add rule name</span><span style="color: red">=</span><span style="color: maroon">"Port 6910 UDP" ^</span>

<span style="color: maroon">            </span><span style="color: black">dir=</span><span style="color: blue">in </span><span style="color: black">action=allow service=any enable=yes profile=any localport</span><span style="color: red">=</span><span style="color: black">6910 protocol=udp</span>

II. How will the Startup Tasks be executed?

• Set the value of Copy to Output Directory to Copy Always. Failing to do so will result in a deployment error. This step will ensure that all the startupTask files are added to the package and uploaded to the Test Agent VM. By default the files are uploaded to E:\AppRoot\

• Open ServiceDefinition.csdef and include a trigger to call the workflow cmd to start the execution of the command file. The csdef file should look like below once you have added the Startup Task tags.

<span style="color: blue"><?</span><span style="color: #a31515">xml </span><span style="color: red">version</span><span style="color: blue">=</span>"<span style="color: blue">1.0</span>" <span style="color: red">encoding</span><span style="color: blue">=</span>"<span style="color: blue">utf-8</span>"<span style="color: blue">?>
<</span><span style="color: #a31515">ServiceDefinition </span><span style="color: red">name</span><span style="color: blue">=</span>"<span style="color: blue">WindowsAzureProject2</span>" 
                   <span style="color: red">xmlns</span><span style="color: blue">=</span>"<span style="color: blue">http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition</span>"<span style="color: blue">>
  <</span><span style="color: #a31515">WorkerRole </span><span style="color: red">name</span><span style="color: blue">=</span>"<span style="color: blue">WorkerRole1</span>" <span style="color: red">vmsize</span><span style="color: blue">=</span>"<span style="color: blue">Small</span>"<span style="color: blue">>
    <</span><span style="color: #a31515">Imports</span><span style="color: blue">>
      <</span><span style="color: #a31515">Import </span><span style="color: red">moduleName</span><span style="color: blue">=</span>"<span style="color: blue">Diagnostics</span>" <span style="color: blue">/>
      <</span><span style="color: #a31515">Import </span><span style="color: red">moduleName</span><span style="color: blue">=</span>"<span style="color: blue">RemoteAccess</span>" <span style="color: blue">/>
      <</span><span style="color: #a31515">Import </span><span style="color: red">moduleName</span><span style="color: blue">=</span>"<span style="color: blue">RemoteForwarder</span>" <span style="color: blue">/>
      <</span><span style="color: #a31515">Import </span><span style="color: red">moduleName</span><span style="color: blue">=</span>"<span style="color: blue">Connect</span>" <span style="color: blue">/>
    </</span><span style="color: #a31515">Imports</span><span style="color: blue">>
    <font style="background-color: #ffff00"><</font></span><span style="color: #a31515"><font style="background-color: #ffff00">Startup</font></span><span style="color: blue"><font style="background-color: #ffff00">></font>
      <</span><span style="color: #a31515">Task </span><span style="color: red">commandLine</span><span style="color: blue">=</span>"<span style="color: blue"><font style="background-color: #ffff00">StartupTask\setup.cmd</font></span>" <span style="color: red">executionContext</span><span style="color: blue">=</span>"<span style="color: blue"><font style="background-color: #ffff00">elevated</font></span>" <span style="color: red">taskType</span><span style="color: blue">=</span>"<span style="color: blue"><font style="background-color: #ffff00">background</font></span>" <span style="color: blue">/>
      <</span><span style="color: #a31515">Task </span><span style="color: red">commandLine</span><span style="color: blue">=</span>"<span style="color: blue"><font style="background-color: #ffff00">StartupTask\setupfw.cmd</font></span>" <span style="color: red">executionContext</span><span style="color: blue">=</span>"<span style="color: blue">elevated</span>" <span style="color: red">taskType</span><span style="color: blue">=</span>"<span style="color: blue">background</span>" <span style="color: blue">/>
    <font style="background-color: #ffff00"></</font></span><span style="color: #a31515"><font style="background-color: #ffff00">Startup</font></span><span style="color: blue"><font style="background-color: #ffff00">></font>
    <</span><span style="color: #a31515">LocalResources</span><span style="color: blue">>
    </</span><span style="color: #a31515">LocalResources</span><span style="color: blue">>
  </</span><span style="color: #a31515">WorkerRole</span><span style="color: blue">>
</</span><span style="color: #a31515">ServiceDefinition</span><span style="color: blue">>
</span>

• ExecutionContext – Make sure you set the execution context to run as elevated. The execution of the cmd file will fail if the file is executed in non elevated mode.
• TaskType – You have the option to set the task type as simple, backgroup, foreground. If you set the taskType to simple, then until the startup task is complete the deployment of the VM will not complete. I won’t recommend running the startup Task as simple, you should run it as background, in case something goes wrong you will have the ability to remotely login to the machine and investigate the reason for the failure. a

FAQ: Something has failed but i don’t know what?

Well to be honest there isn’t too much that can go wrong during the deployment, but in case there is a failure, your best bet is to look at the VM creation logs, the output.log created at the time of running the cmd file. I found some great troubleshooting tips and tricks and gotchas at the blog posts (this, this) of the channel 9 @cloudcover team.

III. Publish, Scaling up or down

Now the most important step, once you have finished writing all the startup tasks and changing the setting of the startup tasks to copy always, you are ready to test the deployment.

You have two options to carry out the deployment,

• Publish From Visual Studio – Right click the Azure project and choose Publish

• Package From Visual Studio – Once the package is ready, you can start deployment from the Windows Azure Portal.

Once the deployment is complete you still have a handle on changing the configurations. For example, in the screen shot below, I can change the Instance count configuration from 1 to 10. This will trigger an additional deployment of 9 more instances using the same package that was used to create the original machine. Benefit, you can scale up your environment in a very short interval. One also has the option to stop or delete the machines once they are not required.

Review and What’s next?

In this blog post, we have automated the process of spinning up the Test Agents in windows Azure. You can download the demo solution from here.

You can read the earlier posts in the series here,

Earlier Azure-ing your Test Rig posts

- Part 1 – Load Testing in the Cloud

- Part 2 – Load Testing in the Cloud

I have various blog posts on Performance Testing with Visual Studio Ultimate, you can follow the links and videos below,

Blog Posts:

- Part 1 – Performance Testing using Visual Studio 2010 Ultimate

- Part 2 – Performance Testing using Visual Studio 2010 Ultimate

- Part 3 – Performance Testing using Visual Studio 2010 Ultimate

Videos:

- Test Tools Configuration & Settings in Visual Studio

- Why & How to Record Web Performance Tests in Visual Studio Ultimate

- Goal Driven Load Testing using Visual Studio Ultimate

I have some ideas around Azure and TFS that I am continuing to work on, not giving away any hints here, but remember to subscribe to http://feeds.feedburner.com/TarunArora I’ll be blogging the results of these experiments soon! Hope you enjoyed this post, I would love to hear your feedback! If you have any recommendations on things that I should consider or any questions or feedback, feel free to leave a comment.

Share this post :

CodeProject