Case Study: Scorpion Software Saves Money and Frees Staff with TestComplete

Scorpion Software Corp. a leading provider of security analytic tools for small businesses, describes how they use TestComplete to move their company forward in software development.

Overview

Scorpion Software uses TestComplete on two primary projects: Carina Intrusion Prevention System and Firewall Dashboard.

Carina Intrusion Prevention System is a host-based intrusion prevention system that provides a mandatory access control system that can be deployed into Microsoft Windows environments to drastically reduce, and in many cases completely remove many security related risks that are exposed to vulnerable servers and workstations.

Firewall Dashboard is a firewall analytics tool that proactively identifies threats before they become problems for businesses. It changes raw firewall logs into meaningful and interpretable threat assessments.

As a very small ISV, Scorpion Software wanted to ensure that they maximized every resource possible to meet the challenge of competing with other larger security vendors. "I began looking at automated testing when I realized that I was overwhelming the limited staff I have with monotonous manual testing to meet our quality control standards when shipping software" says Dana Epp, President and Computer Security Software Architect at Scorpion Software Corp. "Repeating the same tests over and over manually seemed to be a huge investment of time and money, and prone to human failure. Moving to automated testing freed the company from these shackles and let us focus on other areas where staff contributions could help move the company forward towards other business objectives."

One of the weaknesses in their old manual test process was that it was prone to break when staff would rush to get things fixed and shipped. Although they were never negligent in their duties, it was far too easy to miss steps in the testing process when normal business interruptions took place. Further more, there was a large disconnect between development and testing, where it was far too easy to miscommunicate what defects existed in their products. "Moving to automated testing allowed QA to be able to build tests that could be given to a developer to reproduce defect conditions in our software. This overcame the burden of the ‘show me’ attitude most developers have in the industry, as the test does just that before their very eyes. And of course, it immediately gives the developers a baseline to test against. Now developers cannot even check in a fix until it passes the QA test script."

"And finally, I was tired of seeing the same defect types creeping into our software. When we come across a type of defect (an input validation failure for example) we want to be able to reproduce those conditions from that point forward in any software that we write that might use that codebase. Although it’s easy enough for a person to do manually, computers are much better suited to run the gambit of all possible inputs that could be provided. We use this to be able to perform such actions as fuzz testing and fault injection, where we can now literally fire thousands of possible pseudo-random inputs at our applications in an effort to judge our software resiliency and maintain application reliability."

Solution

Dana originally heard about TestComplete while reading an article by Joel Spolsky on Joel on Software. "The timing was impeccable as I had just finished evaluating WinRunner from Mercury Interactive just days before, and was already getting phone calls from their sales team. As a very small business that is funded from my own pockets, I just couldn’t afford the huge investment that WinRunner required. I knew that the ROI was there for the investment into automated testing, but I also knew that the state of the company's cash flow wouldn’t allow me to outlay the costs just yet. Believing there was little hope for me to add automated testing to our software development lifecycle any time soon, I was ecstatic to learn about TestComplete. It offered many of the features I needed from WinRunner without a lot of the other complexity burdens I didn’t care for, and at a price I could afford."

One of the immediate challenges that TestComplete helped Scorpion Software overcome was functional testing of their kernel mode security driver for the Windows Server platform. "We have so many test conditions in which the security driver could function, and the nature of the scripting engine in TestComplete allowed us to build automated test scripts to test every code path in the security driver, on every single kernel that Microsoft provides to us. Through the use of TestExecute and some custom tools that can check out the tests from our source control server, we can automate the testing of different security policy enforcement rule sets on different operating systems in different VMWare images without human interaction. Since all access control policies are stored securely on the file system, we were able to leverage the scripting in TestComplete to automatically alter the policy files and test that the security driver met all policy enforcement conditions. This set up has already helped pay for TestComplete ten-fold when a critical defect was found in our software; a unique test case exposed us to an issue which would have caused many customers to BSOD if we had ever merged that code into the production codebase."

Unexpected Advantages

"What surprised me about TestComplete was some of the unexpected advantages we never thought about before, but became apparent when using the software." Dana expressed that one of the primary examples is how their software development's lifecycle has changed as it relates to defects:

Other Benefits

"Other benefits include the fact that we are now able to add a lot of security testing without having to invest in new tools and technologies, and the education/learning curve that comes with it. We also have a better understanding of the quality of our software at any given time due to the confidence we have in our tests."

"It’s weird, but we routinely find new ways to use TestComplete to automate some new task as part of our development process."

Dana states that, "primarily TestComplete has freed up staff to work on more important things that require their critical thinking, rather than focusing on the tedious repetition of mundane tasks. I enjoy watching staff spending hours developing a new automated test script that will literally save them days or months of man hours that can be performed on a daily basis."

"TestComplete has caught critical defects before they got into the mainstream production codebase. Traditional defect cost analysis shows the difference in monetary value in fixing bugs before they are shipped verses when you fix them in the field. On top of that, the money saved in staffing requirements has been very beneficial to the company. Although our pool of tests continues to grow along side of our products, we haven’t had to hire additional staff to work in QA yet. We can do more with less! That’s something extremely critical for many small software businesses like my own."

Higher Confidence and Reduced Defects

Dana explained that they have increased the time it takes to release a product because they have mandated new workflow processes that take more time to build tests. "However, in doing so, we have benefited from a higher level of confidence in the product at time of release than ever before. And it has reduced the defects that we are seeing in the production software. In the end, the extra time spent SAVES the company a lot in future customer service and support costs."

While Dana stated that he has lost count of how many tests Scorpion Software has, he stated; "We literally have hundreds of test conditions that are tested in various TestComplete projects. As we learn how to optimize tests more we routinely deprecate old tests in favor of new streamlined tests. We don’t follow a traditional documented test case process in favor of automated scripts being tagged to functionality within the product. However, I can tell you that in some cases, some tests take over 24 hours to run on a P4 3GHz machine with 2GB of ram. That’s a lot of different scenarios being tested!"

"We run regression testing daily at noon with TestExecute on a dedicated build and test system. This allows developers to check in work before noon and know before the end of the day if their submissions for fixed defects or new feature have been accepted against the entire product codebase. We also run fault injection, fuzz testing and security testing once a week over the weekend, where the test system can run for 24 to 48 hours straight without worrying about new test cases being introduced."

"Before TestComplete, we would run through all tests manually, if we were lucky, before shipping a new version, which would take weeks of man hours to complete. As such, we only did it on a limited basis, as we simply couldn’t afford the investment in time to perform all manual testing."

Dana explains, "TestComplete's best quality is the simple recording of pre-defined actions that can be augmented with simple but powerful scripts. The test accuracy that is afforded to my company through the use of TestComplete has eliminated manual testing that was prone to human failure and which was expensive to the business."

"Further more, It is my opinion that quality assurance test specialists have traditionally had to be better programmers than the developers of the original codebase to properly build the test harnesses needed to build quality software. With TestComplete, this no longer has to be the case. However, if they are proficient in programming, you benefit immensely with in depth testing that can rival any testing team from big software companies."

Highly Recommended

"Being that I recommend TestComplete to other software CEOs on a pretty regular basis, I often tell them how much test surface we now have on our products. Not only do I recommend that they look into buying the product, I suggest they seriously look into the efficiencies they are missing by NOT taking advantage of automated testing. It’s well worth the investment, especially if you are a smaller ISV and have to compete with the big boys."