Automated Testing of Win32 Applications on Multiple Operating Systems

Abstract

This article tells about application testing on different versions of Windows operating systems via virtual machines working under Microsoft Virtual Server 2005, Automated Build Studio, which is AutomatedQA's build and release management system, and TestComplete, an automated testing tool by AutomatedQA.

In this article

Testing on Different Operating Systems

Any software developer may face a situation when his or her application works fine on one operating system, but closes abnormally or doesn’t work as expected on another operating system. Why does this happen? The reason is that the Windows operating system changes from version to version. The changes concern the application programming interface, security parameters, principles of working with memory, etc; moreover, an application may work incorrectly on localized versions of the operating system. That is why it is very important to test your application on different operating systems to get a reliable software product.

So, what should you do to start this kind of testing? First, you need to run several operating systems on your computer(s). Next, you need to create an application or script that will test your application. You can create a test script in TestComplete, but in this case, you will have to install the test engine on the computer that has the operating system for which you wish to test your application. Sometimes, it is inconvenient to launch a test manually, so you may need to automate testing. For this purpose, you need to have a testing system that will allow you to control processes on a remote machine.

It is often a good strategy to use several physical computers for this kind of testing. The computers must be within the same network environment. Here two questions arise: is it always convenient to use several computers? And is it always possible to have several computers at hand? It may happen that there are not enough computers on which you can perform testing. Even if your company has a lot of computers that can be used for testing purposes, they still may not be available to the tester, since they may have different locations.

One of unpleasant drawbacks that you might come across when testing on several physical computers is that if you test an application repeatedly, each installation may modify the operating system, so it becomes impossible to test a new version of the application correctly. For example, if the installation program works incorrectly, it may not remove some registry keys or files when uninstalling the application. If this happens, you will have to reinstall the operating system or abolish the changes manually, which is time-consuming and will slow down the testing process.

Testing on Virtual Machines

A very good alternative to using several computers for testing is the use of virtual machines. A virtual machine is programmatic simulation of a computer’s hardware environment. It is possible to install the needed operating system on a hard drive of the virtual computer. You can install any operating system on a virtual machine - not just Windows only.

One of the most popular applications used to work with virtual machines is Microsoft Virtual Server 2005. It uses two files to store a virtual machine with one hard drive on it – a configuration file and a file containing an image of the hard drive. Storing an image of the hard drive in a file allows you to test your application on the operating system that was not changed by previous installations. For this purpose, it’s quite enough to copy both files of the virtual machine from some common repository in the network to a computer that has Virtual Server installed.

Virtual Server allows you to launch up to 256 virtual machines simultaneously. This doesn’t require a lot of powerful computers – you just need to have one high-end physical computer to work with Virtual Server and launch the needed number of virtual machines. A running virtual machine becomes available in the network automatically, so it’s possible to get access to its resources through the net. The rest of this article describes testing via virtual machines.

Remote Control Over Resources

Virtual Server has no means to control processes running on supervised virtual machines. However, the fact that remote work with a virtual computer’s resources does not differ from the work with resources of a physical remote computer allows you to use standard means of remote control over processes on virtual computers. One of the most reliable tools, which is used on computers with Windows NT/2000/XP/2003 Server, is Windows Management Instrumentation (WMI). WMI is a Microsoft implementation of the WBEM protocol (Web-Based Enterprise Management) that defines the standards of access to data in enterprise networks. WMI allows you to safely control a number of resources of a remote computer: its processes, services, registry, etc.

For reasons of security, Microsoft did not implement the ability to remotely launch an interactive process via WMI. At first, this doesn’t appear to allow you to test the application’s user interface, but the RunInteractive.exe application shipped with Automated Build Studio makes it possible to launch interactive processes even if the application is running as a non-interactive process.

Testing on Virtual Computers

AutomatedQA’s Automated Build Studio 1.4+ supports WMI and Microsoft Virtual Server 2005, which makes it possible to create macros that will perform automatic testing on several virtual machines.

Automated Build Studio's operations that support WMI allow you to control processes (application launching, closing, enumeration, status checking) and services (status checking and changing, enumeration), and to get information about registered COM classes on both local and remote computers.

Fig. 1. ABS operations used to work with WMI.

Operations of the Virtual Server category include the operations used to control the state of a virtual computer, register it, get information about available machines, simulate keypresses (this is needed when entering the user name and password and when loading the operating system). Virtual Server can be installed on a computer that has Automated Build Studio installed or on a remote computer.

Fig. 2. ABS operations used to work with Virtual Server.

So, as you can see, using Automated Build Studio, you can easily test your applications on physical computers and on any number of virtual machines.

In order to organize testing on virtual machines, you will need the following:

• A client computer with Automated Build Studio 1.4+ installed on it.
• A server computer with Windows 2003 Server (recommended) or Windows XP installed and with Virtual Server 2005.
• The needed number of images of virtual machines with the application you are going to use for testing (for example, AutomatedQA TestExecute) and shared folders.

If Virtual Server is installed on the computer on which Automated Build Studio is installed, only the resources of one physical computer are needed to perform testing.

To prepare for testing, you need to do the following:

• Create a test and, if necessary, modify the application code.
• Prepare virtual machines (you will have to create virtual computers, install the operating system on them as well as TestExecute, customize the security system of WMI and share resources).
• Create a macro for Automated Build Studio that will comprise the sequence of actions.

In the next section, we will tell you how to create a test system on several virtual computers using only one physical computer.

Sample Test

Suppose the logic of your application is built on whether the application is running on Windows XP or not. Of course, the code that checks the version of the operating system must be perfect. The project IsWinXP is a project of an MFC application (Visual Studio .NET) that can check the operating system version.

The CIsWinXPDlg::CheckWinXP() function of the project’s IsWinXPDlg.cpp module generates exceptions if the current operating system is not Windows XP. We are going to test this behavior, for example, on virtual machines with Windows XP and Windows 2000. On Windows XP the CheckWinXP function is supposed to be executed with no exceptions and the function is supposed to generate an exception on Windows 2000.

We will test the function using TestComplete’s unit testing capabilities. The predefined test project TestIsWinXP will be launched via TestExecute on each virtual machine.

To run the test, we will create a testing macro for Automated Build Studio named MacroTestIsWinXP.bpp (you can find it in Macro.zip). We’ll describe creation of such a macro in detail in the Creating the Macro in Automated Build Studio section of this article.

To launch the macro on virtual machines, we will use the RunTest.bat file (you can find this file in Macro.zip).

Preliminary Operations

First, we will prepare the physical computer for testing. For this purpose, we’ll do the following:

• Install Automated Build Studio 1.4 or later.
• Install Microsoft Virtual Server 2005.
• Create two virtual machines, test_xp and test_2k, on the server and install Windows XP and Windows 2000 on them.
• Add these virtual machines to the network domain or to the working group available to the physical computer.
• On the physical computer, we will create a folder in which all the files needed for testing will be stored. Let it be a folder with the C:\Testing path. It will have the following structure:

Fig. 3. The folder with files used for testing.

The IsWinXP subfolder contains the files of source texts of the tested project.

The TestIsWinXP folder contains the files of the test project for TestComplete with the same name.

The Dlls folder will contain the files of dynamically loaded MFC libraries (mfc70.dll, mfc70d.dll, msvcr70.dll, msvcr70d.dll, msvcp70.dll and msvcp70d.dll) needed to launch and test the IsWinXP.exe application.

The MacroTestIsWinXP macro will copy all the files needed for testing to the VMFolder folder. This folder will also contain the following:

• A new version of the application under test (IsWinXP.exe, IsWinXP.pdb).
• MFC libraries.
• The RunInteractive.exe application taken from the folder in which Automated Build Studio is installed.
• The batch file RunTest.bat that will launch the macro.
• TestComplete’s project files.

So, the VMFolder folder will be used as a temporary buffer when a new version of the application under test is being copied to the virtual machine along with its tests.

Fig. 4. The folder containing the files that will be copied to the virtual machine.

Now we are going to prepare each virtual machine for testing. On each machine, we will install TestExecute 3 and Virtual Machine Additions. (Virtual Machine Additions are shipped with Virtual Server. ABS needs them to be able to work with Virtual Server.)

To launch a process on a virtual machine, you will have to customize the WMI security system on it. To do this:

• Select the Administrative Tools item in the Control panel on the virtual computer.
• Choose the Computer Management.
• Find the Services and Applications | WMI Control node, right-click it and
• Select Properties from the context menu.
• Find the Security tab of the ensuing dialog
• Select the Root|CIMV2 node and grant full access to it to allow creating a Windows account.

In our example, we will refer to WMI of both virtual machines using an account with the name tester.

Now we need to set up the security policy as the RunInteractive.exe application requires. For this purpose —

• Open the Local Security Settings dialog (Control Panel | Administrative Tools | Local Security Policy);
• Select Local Policies | User Rights Assignment from the tree view on the left side of the dialog and add the account that will be used to access WMI to the following lists of security settings:
  • Act as part of the operating system;
  • Create a token object;
  • Replace a process level token.
• After the above-mentioned settings are selected, restart the virtual machine.

To enable the client computer to exchange file with the virtual machine, you need to create and share the C:\Testing folder on the virtual computer. Before testing, the MacroTestIsWinXP macro will copy the files taken from the VMFolder folder (in our case, it is C:\Testing\VMFolder) located on the physical computer.

Creating the Macro in Automated Build Studio

Now it’s time to create the MacroTestIsWinXP macro. As we have already said, the macro will initiate the testing process on the two virtual machines.

To be more exact, this macro will sequentially perform the following operations:

• Compile the IsWinXP project.
• Prepare files to be copied to the virtual machines.
• Perform testing on the virtual machine with Windows XP.
• Perform testing on the virtual machine with Windows 2000.
• Send test results to developers.

To store testing results obtained on each of the virtual machines, we will use the TestResults macro variable. To create it, choose Variables|Variables from the main menu of Automated Build Studio and use the ensuing Variables dialog:

Fig. 5. The Variables dialog.

One of the main parameters of the macro will be the path to the folder that contains files needed for testing on the physical computer. Now we need to specify the path to this folder using the TestingFolder constant. (To add this constant, choose the Variables|Constants item from Automated Build Studio's main menu.) We have already said that this folder is C:\Testing. We will specify this path as a value of the TestingFolder constant:

Fig. 6. The Constants dialog.

Specifying the path as a constant and using this constant instead of the real path allows you to easily change the path when it is needed.

Compiling the Application to be Tested

Often, an automated test is launched after a new version of the product is compiled; our macro will compile the application before testing as well. To enable it to do this, we will add the Compile Visual C++ .NET 7.x Project operation to it and specify the project to be compiled - IsWinXP.vcproj.

Fig. 7. The Project page of the “Compile Visual C++ .NET 7.x Project” operation’s properties window.

Note that the path to the project contains the %TestingFolder% substring. To “tell” Automated Build Studio that part of the entered string is the name of a variable or constant, you need to enclose the name in the % symbols. This means that during the macro execution, the substring enclosed in %% will be replaced with the constant value or with the current value of the variable. In our case, the path to the compiled project will look like this during the macro execution:

C:\testing\IsWinXP\IsWinXP.vcproj

Preparing Files to be Copied to the Virtual Machines

After you get a new installation, you should copy all the needed files to the VMFolder folder located on the physical computer, but do not forget to delete all the files from that folder before you do this.

Fig. 8. The macro snippet that copies all the files needed for testing to the VMFolder folder.

Once the files are copied, we are ready to run tests on the virtual machines.

Before you start testing, you need to initialize the TestResults variable with an empty string. For this purpose, use the Set Variable operation.

Fig. 9. Initialization of the TestResults variable.

Creating the Testing Submacro

Now we are ready to test the application. In our case, it will be tested on both virtual machines in a similar way, that is, the sequence of performed operations will be the same. In order not to overload the physical computer, we will run the tests one after another. This guarantees that only one virtual machine will be working at a time. All the needed operations will be included in the submacro called VMTest. Upon initializing the TestResults variable, we will call the VMTest submacro twice – it will test the application on the virtual computers.

Submacro Parameters

In Automated Build Studio, a submacro is like a routine in programming languages. You can parameterize a submacro, allowing you to call it from the macro body with appropriate parameters. The VMTest submacro has a lot of parameters that you can change when testing your application on multiple virtual machines.

Fig. 10. The Parameters page of the “Submacro” operation’s properties window.

Here is a brief description of each parameter of the VMTest submacro:

Parameter	Description
ConfigName	The virtual computer name – test_xp or test_2k.
NetworkName	The network name of the virtual computer (this is the name under which the computer is registered on the domain or in the working group). For example, vm_test_xp or vm_test_2k
WMIUser	The user account that will be used to refer to WMI on a virtual machine.
WMIPassword	The password used to authorize a WMIUser user when referring to WMI.
VMUser	The user name that will be entered when activating the virtual machine.
VMPassword	The password of the VMUser user that will be entered when starting the virtual machine.
WMIQueryTime WMIQueries	They are used as parameters when the macro is waiting for the end of the test run.
VMQueryTime VMQueries	They are used as parameters when the macro is waiting for the invitation to enter the user name and password when loading a virtual machine.
DelayAfterLogon	The time (in milliseconds) needed to load Windows completely after the user name and password were entered.

Parameters are created in the Submacro operation, while the parameter values are specified in the Run Submacro operation that will execute this submacro. The values used to call the submacro depend on which virtual machine you will run. You can find them in the MacroTestIsWinXP.bpp macro.

Submacro Structure

The VMTest submacro can be divided into the following logical parts:

• First, we start a virtual machine and wait until it is fully loaded.
• Second, we launch tests.
• Third, we wait until the test runs are completed.
• Fourth, we get test results from the virtual computer.
• After all this, the virtual computer must be switched off in order not to use the resources of the physical computer.

Now let’s take a closer look at each step.

Loading a Virtual Machine

First, we need to activate the virtual computer using the Run Virtual Machine operation. In our case, Virtual Server is on the computer on which Automated Build Studio is installed, so we will specify the Use local server option on the Connection Options page of the properties window here and in all other operations that work with the virtual machines.

Fig. 11. The Connection Options page of the “Run Virtual Machine” operation’s properties window.

We’ll specify the ConfigName parameter passed to the submacro as the name of the virtual machine.

Fig. 12. The Virtual Machine Options page of the “Run Virtual Machine” operation’s properties window.

Once the virtual machine has been switched on, the operating system starts loading. It is very important to correctly determine the moment when you can enter the user name and password. There are several ways to wait until the operating system invites you to enter them. The easiest way is to use the Delay operation with a large waiting period. After this time expires, the macro starts simulating keypresses, but this approach is not very effective.

On virtual machines that work under Windows NT/2000/XP/2003 Server with Virtual Machine Additions installed, it is possible to “catch” the window that offers you to press the Ctrl+Alt+Delete shortcut using the If OS Is Loaded operation. This relative operation is a container. In other words, its sub-operations are executed only if the operating system, running on the virtual computer specified in the operation parameters, can accept commands from the user. After the operation returns True, you can simulate pressing the Ctrl + Alt + Delete shortcut.

The waiting process can be organized as a loop with the If OS Is Loaded operation in it. The exit from the loop occurs when the operation returns True. Make sure the loop is not endless; otherwise, if an error occurs on the virtual machine or on Virtual Server, the macro execution may never come to an end.

Perhaps one of the best ways to make the operating system wait for that window is to organize a loop with a specified number of steps – this can be done with the For Loop operation. The number of steps is passed within the VMQueries parameter of the submacro. At each step, after the timeout specified by the VMQueryTime variable has expired, the If OS Is Loaded operation is executed. If this operation returns True, the Go to Label operation is executed and the operation execution goes to the StartLogon label, which is beyond the loop.

Fig. 13. The macro snippet that waits for the operating system to load.

If the submacro parameters VMQueries and VMQueryTime have values 36 and 5000 respectively, in the worst situation, the loop will be executed 36 times and during each execution the If OS Is Loaded operation will be preceded by a 5-second delay. The values of these parameters mostly depend on the performance of your physical computer.

The next step is simulation of keypresses. To load the operating system, it is necessary to press the Ctrl + Alt + Delete shortcut, enter the user name and password, and simulate pressing the Enter button. Automated Build Studio includes a convenient tool used to simulate keypresses on virtual computers. The Send Key Sequences operation suggests you use either a standard sequence of keypresses or specify the needed sequence explicitly.

Fig 14. Types of key sequences that can be simulated.

At this point, we need to “press” Ctrl + Alt + Delete. This is a standard combination of keys, so we will select it from the menu and place it in the script that will simulate the keypresses. To focus the field in which the user name will be typed, we will use another standard combination - Shift + Tab. Once the window is focused, we can enter the user name: choose the ASCII Key Sequence item from the context menu and specify the submacro’s %VMUser% variable in the appeared window:

Fig 15. The Input ASCII Key Sequence dialog.

To go to the field in which the password is specified, choose the standard Tab key, then enter the password the way you entered the user name by specifying the submacro’s %VMPassword% parameters as an ASCII sequence. Finally, you can simulate pressing the Enter key.

Fig 16. Shortcuts that can be simulated when the operating system is being loading.

The Send Key Sequences operation will successively simulate all the keypresses on the current virtual machine.

Once the user name and password have been entered, you need to wait until the operating system is loaded. We will use the Delay operation in which the length of the waiting time will be specified in accordance with the value specified in the DelayAfterLogon parameter. This value will depend on the performance of your physical computer and, perhaps, on the operating system installed on the virtual computer.

Testing

The operating system is now running on the virtual computer and the shared folder Testing is available on the network. Our task is now to copy all the files needed for testing to that folder. The files are copied via the Copy File(s) operation from the VMFolder folder located on the physical computer:

Fig. 17. The properties of the Copy File(s) operation.

On the network, we use a symbolic name of the virtual computer. It is passed as the submacro’s NetworkName parameter. The path to the source folder, VMFolder, is specified via the TestingFolder constant.

Now everything is ready for testing. The command file RunTest.bat that we copied to the virtual computer will launch the RunInteractive.exe application which in its turn will run the test in TestExecute in interactive mode:

RunInteractive.exe "C:\Program Files\Automated QA\TestExecute 3\TestExecute.exe" "C:\Testing\TestIsWinXP.mds" /UUnit1 /Pmain /SilentMode /RunAndExit

Note: We use the RunTest.bat file just to show that you can execute not only .exe files. In your macros, you can also execute RunInteractive.exe and pass the path and parameters of the testing application to it.

To execute the RunTest.bat file remotely, we can use the Run Process operation. On the Connection Options page of the operation’s properties window, we’ll specify the parameters of connection to WMI:

Fig. 18. The Connection Options page of the Run Process operation’s properties window.

Here, we specified the submacro’s parameters WMIUser and WMIPassword as the user name and password. The name of the WMI server is the value of the NetworkName parameter.

The parameters of the launched process must be specified on the Run Options page of the operation properties window:

Fig. 19. The Run Options tab of the Run Process operation’s properties window.

Waiting for the End of Testing

In most cases, testing of complex applications takes a lot of time. If the tester does not use some special tools, it’s hard to evaluate when the test results become available. Finalization of the TestExecute.exe process signals that the test run is over. Automated Build Studio includes a relative operation, If Process is Running, that checks whether there is some process running on a remote computer. You can specify parameters of the process to be checked on the Process Options page of the operation properties window. To identify the process, we will use its name – TestExecute.exe.

Fig. 20. The Process Options page of the If Process Is Running operation’s properties window.

All we have told you about how to make the operating system wait until keypresses are simulated at startup of the virtual computer can be applied to the finalization process. The submacro snippet below demonstrates how you can make the operating system wait for the TestExecute.exe process to finish:

Fig. 21. The macro snippet that waits for the end of the TestExecute process.

The number of loop steps is passed within the submacro’s WMIQueries parameter. The length of the timeout on each step is determined by the WMIQueryTime parameter. If the “If Process is Running” operation returns True, the loop is executed; otherwise, the loop is broken, the Go to Label operation is executed and the execution goes to the TestOk label, which is beyond the loop.

Processing Test Results

After the test run, you have test results to process. We will transfer the test results from TestExecute to ABS using an intermediate file: the test project executed by TestExecute saves brief results to the TestResults.txt file, and the ABS macro reads data from this file and processes it. We will add the data stored in that file to the TestResults variable that we will use to collect results.

We will save the results in the following format:

Below is the JScript procedure that reads data from the TestResults.txt file and adds it to the TestResults variable:

function Main()
{
   Variables.TestResults =
      Variables.TestResults +
      "-----------------------------------------\n" +
      " Virtual machine: " +
      Variables.ConfigName +
      "\n-----------------------------------------\n";
   strResultFileName =
      "\\\\" +
      Variables.NetworkName +
      "\\Testing\\TestResult.txt";
   FileSystem = new ActiveXObject("Scripting.FileSystemObject");
 try
   {
      FileStream = FileSystem.OpenTextFile(strResultFileName, 1);
 try
      {
         Variables.TestResults =
            Variables.TestResults +
            FileStream.ReadAll()
            + "\n";
      }
 finally

      {
         FileStream.Close();
      }
   }
 catch(e)
   {
      Variables.TestResults =
         Variables.TestResults +
         "I/O ERROR\n\n";
   }
}

The script adds the caption, which includes the name of the virtual machine (the submacro’s ConfigName parameter) on which the testing was performed, to the global variable TestResults as well as the contents of the TestResult.txt file which is in the shared Testing folder located on the virtual computer. For example, after a test run on the test_xp virtual machine, the following text will be added to the TestResults variable:

-----------------------------------------
Virtual machine: test_xp
-----------------------------------------
CheckWinXP - Ok!

After you read the test results, you can delete all the files from the C:\Testing folder located on the virtual machine. For this purpose, use the Delete File(s) operation; you can specify the \\%NetworkName%\Testing\*.* string as a file mask. If you plan to use a “clean” image of the operating system for each test, there is no need to remove these files.

Switching off a Virtual Machine

After a test run, you can switch the virtual computers off. To do this, make use of the Reset\Shutdown\Turn Off Virtual Machine operation. To switch off the virtual machine, we will choose Shutdown guest OS.

Fig. 22. The Reset/Turn Off Options page of the Reset\Shutdown\Turn Off Virtual Machine operation’s properties window.

Launching a Submacro

Now let’s get back to creation of the basic macro. Upon first initialization of the TestResults variable, we will call the VMTest submacro to perform testing on the virtual machines test_xp and test_2k. To call the submacro, use the Run Submacro operation, which is in the macro body.

Fig. 23. The options of the Run Submacro operation.

Sending Notifications

After the two submacros launched sequentially are executed, you can notify the developers and testers about the test results (they are kept in the TestResults variable). You can send notification via email by using the Send E-Mail operation. As shown on the image below, we specified the value of the TestResults variable in the message body.

Fig. 24. The Message page of the Send E-mail operation's properties window.

After the testing is over, the results will be sent to the recipients.

The Whole Macro

The whole macro we have created looks like this:

Fig. 25. The MacroTestIsWinXP macro.

Download the macro

The Results

The parameters that we used when calling the VMTest submacro are in the table below:

Parameter	Operating Systems
	Windows XP	Windows 2000
ConfigName	test_xp	test_2k
NetworkName	vm_test_xp	vm_test_2k
WMIUser	vm_test_xp\tester	vm_test_2k\tester
WMIPassword	tester	tester
VMUser	tester	tester
VMPassword	tester	tester
WMIQueryTime	5000	5000
WMIQueries	36	36
VMQueryTime	5000	5000
VMQueries	36	36
DelayAfterLogon	15000	15000

We got the following results in the TestResults variable:

-----------------------------------------
Virtual machine: test_xp
-----------------------------------------
CheckWinXP - Ok!

-----------------------------------------
Virtual machine: test_2k
-----------------------------------------
CheckWinXP raised an exception

As you can see, the test results are the ones we expected. The CheckWinXP() function did generate an exception when working on the operating system other than Windows XP.

Summary

In this article, we described how to test applications on virtual computers using Automated Build Studio and TestComplete. You can customize your test system by modifying the afore-mentioned macro. This will make it more flexible and it will be able to meet many of your testing needs. Testing on virtual machines allows you to quickly find possible errors in your applications; this kind of testing is not resource-consuming, which is also very important. Using Automated Build Studio, you can automate the testing process and make it even more flexible.