Automated Testing in Virtual Labs

Introduction

Using virtual machines in a virtual testing lab is becoming more popular as a QA testing practice. This approach allows QA teams to save money by running a number of virtual machines on fewer computers and also allows testers to test different application builds on a permanent system configuration. However, the efficiency of testing in virtual labs increases when you automate the testing process with an automated testing tool.

TestComplete is a comprehensive automated testing tool with extended features for testing applications running on virtual machines and it supports any kind of QA testing methodology: unit testing, regression testing, GUI testing , Web testing, distributed testing and many others.

Automated Testing with Virtual Machines

Virtual machines or Virtual PC’s are software, like VMWare that emulates real computers. By using virtual machines in your QA testing, you can emulate different computers with different OS’s on a single physical computer or create a whole virtual lab, with several differently configured virtual machines. These virtual computers work independently from each other and you can launch two or more virtual platforms simultaneously on one computer, saving the cost of having to purchase more hardware just to run your QA tests. Applications running on a virtual machine behave as if they were running on their own physical system. This is very useful for testing applications and allows testers to test their application under various platforms using a single computer. It can also be helpful for testing web applications since you can simultaneously test your web application in multiple browsers running independently from each other under different virtual machines, again, without the cost of purchasing more hardware for your testing needs.

Testing applications on virtual machines can have different purposes. You can use it for distributed testing of client-server applications, functional testing or in regression testing. But, no matter which kind of QA testing you use, it will be more effective if automated and any kind of testing in virtual labs can be easily automated with TestComplete.

An Affordable, Stable, System Configuration for QA Testing

A computer system is a dynamic system and operating system configurations continuously change. Installing or updating software and hardware drivers happens frequently and installing different versions of an application affects the system’s internal structure, the registry and other crucial components that persuade testing results. While buying multiple computers to support various platforms is an option for some, it is often too expensive for most testing labs.

Since computer systems are ever changing, it causes a problem for QA testing. We cannot fully understand testing results when testing is performed on different configurations. In other words, if our application does not work properly, it is more difficult to determine the reason in a constantly changing system. Another problem is that it is difficult to test different application builds installed on a single operating system. It can cause conflicts and may even provoke system crashes.

Fortunately, virtual machines are a workaround for these problems and are a lot more cost effective. Once you have created and configured a virtual machine or a full lab of virtual machines, you will have a stable system configuration, which is very important when testing applications.

Virtual Testing Labs on a Single Computer

Virtual machines can work independently from each other on a single physical computer, creating a virtual testing lab on one computer. This is a huge benefit which is often used in QA testing.

For example, you can use one computer to test your application under Windows XP, Windows Vista and Windows 7. In virtual labs, there is no need for three different computers to execute tests simultaneously; you can just sequentially execute tests on a single computer with three different virtual configurations. However, you may need a more powerful computer to run several virtual machines on one computer at once, but this is often cheaper than buying three physical computers.

Secure QA Testing in a Virtual Lab with its Own Network

Sometimes testing applications is an unpredictable process – a tested application may hang the operating system, provoke a system crash and endanger the corporate network’s security. To isolate your testing lab, you can join your virtual labs into a subnet within your network and then isolate it from the rest of the computers in your company. You get a virtual lab network fully isolated from your company’s computers that will increase the security level of your testing process.

Isolating your virtual lab on its own network does not determine the type of software testing you perform; you can still perform distributed testing, GUI testing, web testing and any other kind of testing. You can also remotely launch automated tests on virtual machines and use any automated testing technique provided by TestComplete.

Locking Computers in Your Virtual Lab

QA testing teams may want to execute their automated software tests on locked computers when running tests overnight. Software tests that don’t interact with the GUI can be run on locked computers. GUI tests or functional tests that simulate user actions cannot be run on locked computers, since the user session is frozen (no windows can be shown) and the automated testing tool cannot simulate user actions.

Virtual machine software allows you to run GUI and functional tests on locked computers with TestComplete. To do this:

• Install TestComplete and your application on a virtual machine. Also, copy all files that your application and TestComplete project may require for testing.
• Lock the virtual machine’s host computer.
• Log onto the virtual machine from any other computer.
• Start executing tests.

TestComplete will run tests on the virtual machine despite the host being locked. However, if you disconnect from the virtual machine while running GUI tests, the automated tests will fail.

Automating Tests on a Virtual Machine

Usually virtual machine software provides a special COM API that you can use to manage virtual computers. It is easy to work with COM objects from TestComplete, so you can use this COM API to automate testing over virtual machines directly from your virtual testing lab.

The following code sample (VBScript) demonstrates how to automate the running and stopping of a VMware virtual machine directly from an automated test script. We use a .NET wrapper for VIX COM API to power on and then to power off a virtual PC. This script routine works with the VMware Workstation installed on the computer where the routine is launched.

The dotNET object is a special TestComplete object that you can use to call routines and methods that reside in any .NET assembly from your automated tests.

Besides using COM to automate tests from TestComplete, you can also use Automated Build Studio for managing virtual machines. Using Automated Build Studio is the easiest way to automate actions over virtual platforms. It contains a lot of high-level operations intended for working with different VM software. For instance, to reboot a virtual machine from an AutomatedBuildStudio macro you can just add the appropriate operation to the macro. Then, for instance, you can schedule the macro execution to run your virtual machine at a specific time. The best practice is to use Automated Build Studio and TestComplete together. This allows you to automate both virtual machines and automated tests in a virtual lab easily.

Running Automated Tests in Virtual Labs with TestComplete

Once you have created automated tests with TestComplete, you should configure the needed virtual machines in your virtual lab. To prepare virtual machines for testing, determine the following requirements:

• Which testing platforms do you need?
• Which VM software should you choose?
• Which automated testing tool do I need to install on virtual machines?

These requirements are important for software testing and you are free to choose any platform and use TestComplete to test your application. You also need to determine how you will work with the tested platforms:

• Will virtual machines run on a single or on several computers?
• Which utility should I use for controlling virtual machines?

Preparing Virtual Labs for Automated Testing

As we mentioned above, you are free to choose almost any testing platform and any VM software available on the market to use in your virtual testing labs.

After you have installed the particular VM software, you can prepare your virtual machines according to the following steps:

1. Create a new virtual machine. For information on how to do this, see the VM software documentation.
2. Configure the virtual hardware.
3. Install the needed operating system.
4. Install TestComplete or TestExecute (see Installing Automated Testing Tool section).

Installing an Automated Testing Tool

There are several tools that can be used to execute automated tests on virtual machines in your virtual lab. For example, you can install TestComplete or TestExecute on the virtual platforms:

• Executing automated tests with TestComplete in virtual labs

  Install TestComplete on a virtual machine. You can then launch TestComplete and execute your automated tests in your virtual lab. Note that all TestComplete plug-ins required for your tests to also be installed and enabled in TestComplete in order for you to use them in your virtual testing lab.

• Executing automated tests with TestExecute in virtual labs

  Install TestExecute on a virtual machine. You can create automated tests with TestComplete on a real computer and then copy these tests to several virtual machines in your virtual testing lab. Then you can install TestExecute with all required plug-ins and run your automated tests in your virtual lab. TestExecute is a resource-friendly utility that executes TestComplete tests and logs its results on machines that do not have TestComplete.

• Organizing the application’s distributed testing

  Test applications through your network. If you test a client-server application, you can install one part of the application on a virtual machine and then test it with TestComplete from another machine via the HTTP protocol. The only thing you need is to establish a network connection between the machine that has TestComplete and the virtual machines with the tested application. For instance, you can test web servers this way.

Controlling Automated Tests in Virtual Labs

To be able to launch automated tests on virtual machines in your virtual testing lab, they should be running and you should have control over them, whether direct or remote control. There are two possible ways to work with virtual machines in your lab:

• Virtual machines residing on a local computer

  Virtual machines can be installed and run on a local computer, where TestComplete or TestExecute is also installed. You can control different virtual machines just by switching between the windows with virtual computer desktops.

  When running the GUI tests in a virtual machine console, do not minimize the console window. When a virtual machine window is minimized, the operating system does not draw the GUI so TestComplete cannot perform user actions over the tested application.

  As for other kinds of tests: unit tests, load tests and others, you can minimize the virtual machine console; however, do not shut down the machine until the automated tests are finished.

• Virtual machines residing on a remote computer

  You can access remote virtual machines via a special utility, for instance, the Remote Desktop Connection tool embedded in Windows XP or third-party utilities, VNC, DameWare and others. In this case the virtual machine must be preliminary prepared for controlling it remotely and executing automated tests.

The Remote Desktop is a built-in Windows feature that lets you access a Windows session running at one computer from another computer. In order to use Remote Desktop, the workstation that has TestComplete or TestExecute must have special server-side software. This software may be included in the operating system (like it is included in Windows XP), or it can be a third-party software like VNC, DameWare or NetOp.

The Remote Desktop feature lets you perform a variety of tasks. For example, you can use the Remote Desktop if you need to:

• Observe the tested application’s behavior on remote computers.
• Run tests on remote computers.
• Run tests on computers that are locked due to security purposes (see Locking Computers in Your Virtual Lab).
• View and analyze test results on one or several remote computers, and save them to a local computer.

To run automated tests with Remote Desktop, follow these steps:

• If needed, lock the computer where TestComplete (TestExecute) is installed.
• Log on to the TestComplete (TestExecute) workstation via the Remote Desktop Connection from another computer.
• Start running tests with TestComplete (TestExecute).

TestComplete (TestExecute) will run tests in the Remote Desktop session, the same way as if they were running from the remote computer itself.

When running GUI tests via the Remote Desktop, do not minimize the Remote Desktop window; when the Remote Desktop window is minimized, the operating system does not draw the GUI so TestComplete will not be able to “see" the tested application’s interface and thus will fail to interact with it.

For the same reason, do not disconnect from the Remote Desktop Connection session or lock the current computer while running tests that interact with GUI, otherwise the automated tests will fail.

These restrictions apply only to tests that involve interaction with an application’s GUI. As for other kinds of tests, for example, load tests, they can be successfully run if the Remote Desktop window is minimized or if you disconnect from the Remote Desktop.

Automated Testing in a Virtual Lab - Scheme

Now that the virtual machine configurations are ready in your virtual testing lab and you know how to control the automated test execution, you can start software testing. The following automated testing scheme allows you to take full advantage of your QA testing in a virtual lab:

1. Create a copy of the virtual machines with the desired system configuration. Only test with this copy from this point forward.
2. Run the virtual machine copy.
3. Install the tested application.
4. Copy all automated tests to the virtual machine in your virtual testing lab.
5. Run automated tests under the virtual machine.
6. Transfer all test results to the physical computer, where they can be analyzed.
7. Delete the virtual machine copy.
8. Perform steps 1-7 for each automated test in your virtual lab.

With this scheme, you can test different application builds on different operating systems with different configurations.

Determining the Type of Virtual Machine in Automated Tests

TestComplete provides special properties that determine whether TestComplete is running under a virtual machine or a physical computer. With TestComplete, if the Sys.OsInfo.VirtualPC property is True, this indicates that TestComplete is running under a Virtual PC machine. If the Sys.OsInfo.VMWare property is True, TestComplete is running under a VMWare machine. You can use these properties to provide conditional testing in your automated tests. For example, you can execute certain automated tests on specific virtual machines and other automated tests on physical machines, based on particular property values. Also, you can execute only those tests that do not require an Internet connection on virtual machines, and so on.

The following picture demonstrates how we can find that TestComplete is running under a virtual machine. The Sys.OSInfo.VirtualPC property is equal to True, so TestComplete is running under a VirtualPC machine.

Picture – Determining the Type of Virtual Machine

You can also determine whether TestComplete is running under a remote session via the Sys.OsInfo.RemoteSession property. If this property is True, TestComplete is running under a remote session, for instance, a session launched via the Remote Desktop utility. Using this property you can determine, for instance, whether a tester logged into a virtual workstation via a remote control utility and execute all tests, except GUI tests, on this virtual machine.

Distributed Testing in Virtual Labs with TestComplete

Distributed testing means that a test consists of several parts and each part is performed on a separate computer. This testing is often performed with Client-Server type applications. With a virtual lab on one computer, you can perform distributed testing of your Client-Server applications without the need for multiple computers.

With TestComplete, you can automate the whole distributed testing process. Distributed tests can be organized as a master and slave projects in TestComplete, and then run in a virtual lab.

The following image demonstrates how two parts of a distributed test are executed on a single computer. The master project is running on the physical computer and the slave project is running on a virtual machine that runs on that same computer:

Picture – Distrubuted Testing in Virtual Labs

Testing Web Applications in a Virtual Lab with TestComplete

Using virtual machines is becoming a good practice for testing different web applications. For instance, you can test web applications simultaneously running in different browsers in a virtual lab. In this case the browsers are completely isolated from each other and this testing approach gives the most reliable results.

The following image demonstrates two instances of a Flash application running in two browsers – Internet Explorer and Firefox. Internet Explorer is launched on a virtual machine, Firefox - on a real machine. So we can execute tests for the application on both machines simultaneously and independently from each other.

You can also test web servers running under virtual platforms via the HTTP protocol with TestComplete. In this case, you can install and configure the tested web server on a virtual machine, and then test it with TestComplete, installed on a virtual or real computer. The only thing you need – a network connection between these computers

Virtual Machine Limitations

There are some obvious benefits of testing applications under virtual platforms; however, they have some limitations too. The virtual operating system can work slower than a real computer, but this depends on the situation: the virtualization technology, hardware, the number of simultaneously running virtual platforms and others. Besides the speed limitations, some hardware functionality can only be used by native operating systems directly working on a physical computer. So, you should take into account these virtual platform limitations, especially if you are testing with specific hardware features, multiprocessor systems and other important factors. Of course, these limitations depend on the particular VM software and hardware and you should check their documentation first before performing automated testing.

Supported VM Software

There is a lot of VM software available on the market. TestComplete can work with any virtual machine software, including the most popular:

• Microsoft Virtual PC 2004 and 2007
• Microsoft Virtual Server 2005
• VMware Server 1.0 or later
• VMware Workstation 5.5 or later

Conclusion

Automated testing in virtual labs and working with virtual machines allows QA testers to get the maximum benefit out of their testing with the lowest overhead cost. TestComplete’s ability to work with both virtual machines and real computers gives testers an edge and can even increase your testing coverage and efficiency. TestComplete also helps you achieve your maximum ROI when using automated testing in virtual labs. If you haven’t tried TestComplete, be sure to download and try it today.