Case Study: QlikTech Uses TestComplete to Test QlikView

QlikTech International AB develops data analysis software that is easy to use, but very difficult to test. QlikTech used TestComplete to quickly perform tests that were almost impossible for a human and saved time and money while doing it.

Overview

QlikTech's software, QlikView, is a state of the art data analysis tool that can collect and associate data for analysis from multiple sources, no matter if the data is stored in a database, Excel, html, xml or text files. QlikView is based on a new revolutionary technology that enables multi dimensional data analysis far quicker and customizable than the old fashion OLAP technology and makes sophisticated analysis dramatically easier to deploy, use and maintain. QlikView’s click driven, visually interactive interface is simple for end users to learn and use, but is very hard to test.

"The QlikView development process is optimized for quick and frequent releases. Given this and the number of new product features that are continuously increasing, manual testing of all functionality has become way too time consuming and complex", says Niklas Lagesson, Automated Test Engineer for QlikTech, "As an example you can imagine how it would be to look at a painting of 500 different charts and comparing them to how they where painted in the previously released version. I can assure you, after comparing 25 different charts you don’t see much difference."

Solution

In 2004 QlikTech had a conference in Cortina, Italy and one of the main discussions at the development meeting was whether or not it was possible to implement automated testing with QlikView.

Lars Anderson from WinSider, QlikTech's neighbor company, recommended that they try TestComplete. Niklas says, "After evaluating it for a few weeks, and having a test project that was well on its way, we decided TestComplete was right for us. This has proven to be a very good decision since TestComplete now handles almost all of our pre-release testing."

QlikView is very focused on user interaction and most functionality is handled from the user interface. "Our problem before we started using TestComplete was that we needed to ensure that new functionality or bug fixes didn’t affect or brake any of the old functionality. This especially applies to chart painting, as all charts are painted differently depending on the input data and code changes can affect it in an unpredictable way. TestComplete’s ability to compare images, text files and dialog objects has really helped us overcome this obstacle."

Bugs that are not found or those that are found late in the release cycle can be very expensive and might lead to lost deals and delayed project releases. "With TestComplete this is drastically minimized. To be able to run a complete automated test in a few hours compared to three days for the manual test procedure, gives us the possibility to do last minute bug fixes without the risk of breaking all or part of our application."

Before QlikTech had TestComplete, all testing was performed manually. "Manual testing has a tendency to be very scattered, while an automated test always performs the action in the same way each time. Test coverage increases with every new release since we started developing automated tests hand-in-hand with the QlikView features. And we now run a full test suite every time we compile instead of performing incremental manual protocol testing after each bug fix."

QlikTech currently has one main TestComplete project with six test areas that are constantly increasing. Each test area comprises of a number of specific test items which total to be around a thousand.

QlikTech continuously runs tests to ensure that the development changes haven’t broken anything. "Before we implemented automated testing we weren’t able to run the full suite of manual protocol based testing more than once or twice per release cycle. When a bug was found and fixed we had to incrementally run a specific protocol for the affected area. Today with automated testing, we can run the full test suite over and over again after each bug fix. TestComplete’s ability to compare dialog objects, images and text files is a very important in all of our test areas."

Niklas Recommends TestComplete

Niklas recommends TestComplete to other software developers, "With TestComplete you get a testing tool that gives you the most “bang for the buck”. You’ll be up and running in no time and when you want to take your tests to the next level the functionality is there."