Case Study: Gateway Improves Time to Market With TestComplete

Gateway Ticketing Systems, Inc has a wide customer base and their application has to maintain a high level of flexibility in its initial configuration. Because of this need, even for standard ticketing transactions, there are a large number of different case scenarios that need to be run. Gateway Ticketing System's Robert Martin explains how TestComplete helps them test their Galaxy Point of Sale system.

TestComplete Handles Thousands of Case Scenarios

The Galaxy Point of Sale system has 4 different ways to calculate tax on ticket sales. In addition to this, revenue for a single item may be split into multiple ledger accounts, each split possibly subject to different taxation rules. Discounts can be applied to any of the above combinations. Also, their application supports the ability to complete a transaction using a foreign currency exchange rate. "So, any time a core section of code involving financial calculations of our application is changed, thousands of case scenarios need to be run in order to ensure full regression for our customer base," said Robert Martin from Gateway Ticketing Systems, Inc.

Gateway Ticketing Systems also has a lot of commonly shared code in their applications, where an object is defined for use in one application but may be used in other applications. "While unit testing is sufficient for testing common objects, the various ways these objects interface in different applications needs to be subjected to regression anytime they are changed to ensure that any corrections or enhancements do not adversely impact other applications. Add the interfaces to the other applications in the system and the need to ensure that data is properly stored for universal use and you have a recipe for a very labor intensive regression test any time a new software release is prepared for delivery."

TestComplete Supports a Wide Range of Testing

"One reason for choosing TestComplete is the wide range of testing that TestComplete can perform. Our application needs to be able be tested for standard user operation via the GUI as well as data storage tests, web site testing (for e-commerce web stores), http load testing, integration testing, and system testing. With a single application, TestComplete allows us to create automated tests for any number of testing scenarios without having to utilize a different application," said Robert.

"Support for multiple scripting languages like C# script, DelphiScript and Javascript was very important for us. Since we are a Delphi shop, this allows us to continue to use the same language base across the board and even allow developers to write scripts rather than just leaving it up to the QA Analysts to develop the tests."

TestComplete Finds Problems that Manual Testing Can't

The Gateway Ticketing Systems eCommerce solution is a multi-process solution. With a solution like this, single threaded manual testing is insufficient for detecting the types of problems that happen under real-world loads. Gateway's QA department instituted automated load tests to detect these issues. Robert described their difficulties: "Analysis showed that these problems were volume related and that, under a heavy request load, the application would occasionally fail with memory address conflicts and improper thread handling. Manual testing was insufficient for reproducing the problem in the test lab because of how the timing of requests contributed to the problems."

Gateway Ticketing Systems created two different sets of TestComplete tests to recreate the problem in the test lab. "The first set involved the website itself and the .NET application that the site used. An HTTP load test was constructed using TestComplete’s load testing plug-in to send a large number of virtual users to the website to process orders rapidly to determine what sort of problems occurred at the website that contributed to the volume problems. Another HTTP Load test was constructed to fire a large number of XML requests directly to the eGalaxy service to determine what sort of load problems would occur at that point of the interface."

"These tests were extremely informative in that they pointed to a number of memory management problems within the .NET code that were easily corrected to prevent malformed requests from the website. The eGalaxy application also showed a number of thread handling issues where objects were being created and destroyed in a non-thread safe solution. With the help of the logs generated by TestComplete we were able to pinpoint and correct application problem areas. These corrections helped us release a stable solution to our user base."

TestComplete Saves Man Hours and Improves Time to Market

Robert was enthusiastic about TestComplete's influence on Gateway's development process: "TestComplete has saved us a lot of man hours that would be needed to manually execute our regression tests. As stated above, we have a very flexible system with a multitude of configuration scenarios, each of which may be subject to regression testing after code changes. I have estimated that, in order to run all necessary tests to regress a single change to our tax calculation code, it would take, manually, at least 3 days, 8 hours a day, to run all the tests. That is just for the POS functionality and would most likely take more time than I estimated. This does not include the integration testing of the tax code into the other applications that utilize it. Also, manual testing is inaccurate in these cases because mistakes are easily made by the user and they may miss a configuration change or a test case simply because of the sheer volume. Automation of these tests has reduced the time to 3 hours to complete the test cases. That’s an 86% improvement of time spent simply for one portion of our software."

"This time improvement has also decreased our time to market for our application. While our coverage of our code base is not 100%, our testing projects have targeted the most critical code sections currently, covering approximately 60% of our code base. So, for every software release, we run a suite of projects that encompasses this entire set of tests. Within 5 hours, we have complete regression of our most critical code and can release the software, confident that the number of defects is significantly lower. Again, the estimation of time saved would be approximately 80%-90% of our man hours that, instead of spent on regression, can be spent on other tasks such as functional validation of new features, documentation of new test cases, and continued development of the automated regression."