A simple idea

Testing is essential. Testing is essential in any engineered product. Software development has the advantage here, in that systematic testing can find ahead of time if an application does what is expected of it, and does not do what is not expected of it. In the real world, this isn't a burden, it's a boon.

Software testing is essential at the user level - functional testing. It is essential at the unit level as well -- test out each code unit before you release it to your project, know that it does what it is meant to do, avoids the unexpected - even in unexpected conditions, and you have the assurance of working on a solid project, each step of the way.

Yet, more often than not, applications and units are insufficiently tested, for the reason that testing is too much of an interruption in the development process.

Worse, the results of tests often remains unexploited. By the time the "test phase" comes around, it is simply too hard to find exactly what gives rise to observed defects. Debugging costs days or weeks, and typically at the worst time in the project's life.

None of this is necessary today. In fact, this approach to testing is becoming obsolete. Testing should be a natural part of the development day, where the team gauges the work it has just done, and makes sure it fits specifications before proceeding further.

Every year, we are expected to deliver finished software in more quantity, with richer capabilities, and on shorter schedules. Some development offices will grow the aptitude of doing this in workman-like fashion, reliably day in and day out, and with gratified customers on delivery. Others will keep "controlling" code quality through more and more increments of sheer fright. The future is not hard to read.

What stops testing from being exploited as a boon, rather than borne as a burden, stems from problems of test organization and test management - problems which, on today's computers, should be handled by automated tools without distracting developers.

When such tools are used properly, testing becomes the stand-back-and-see moment at the end of each development step. Once the test results are finally correct, the code is more than solid: it truly fits the need it was written to meet.

AQtest can serve many purposes. But the idea behind the tool is this - automation-based daily regression testing.

From the start of the project, each small increment of functionality gets a small test written for it. Once the test generates correct results, the results are stored as the standard against which to compare output henceforth. Every time a new build is complete, both the latest tests for the newest features and all the tests created since the start of the project are executed. If all is well, these tests merely report that current output matches the expected standard. Only the newest tests need human review.

If existing code has been "broken," one of the existing tests will find it. Because the test applies to one small functionality, it's easier to trace what's broken. Better yet, whatever broke it is something that was done since the last run on the test battery (normally twenty-four hours or less), which is something easy to remember.

In any case, as soon as the tests are in, any needed correction is attended too immediately, the tests are rerun and the cycle repeated until they come out clean. Imagine there is one build a day, near quitting time. Before leaving for home, the automated test is launched. On starting work the next day, the automatically-filtered results are checked. Either everything is ok, or a quick correction cycle is needed. After that, developers can proceed with the knowledge that, up to that moment, their code passes every test they have thought up for it. When they leave, they know the day's work will be checked, automatically, against the same standard.

An advanced example of the same idea that stands behind AQtest is Extreme Programming, where builds and automated regression tests run several times a day. Recommended reading - Kent Beck, Extreme Programming, Addison-Wesley, 2000.

In a nutshell

AQtest is an automated testing environment for development projects in Microsoft Visual C++ and Visual Basic, and in Borland Delphi and C++Builder. It manages scripts to test applications externally, self-testing by the applications themselves, and many inside-outside forms of tests. It is oriented equally to unit testing and to functional testing.

Its ideal use is to manage daily or more frequent regression tests from the beginning of the project to its final delivery, as illustrated above. Many prospective customers, however, see it first as a means of automated support for traditional QA -- functional testing by a team separate from the developers. AQtest is excellently suited for this as well.

AQtest's unit of management is the project. This includes:

The results of a test are the messages it posts to the log. Of course, they can be commented later, or pruned out.

In the normal run of things, a test just compares current output against the stored standard and reports "all-ok". This is what you want - thousands and thousands of all-ok's through the life of the project.

AQtest's reporting is based on applying filters to the log. Normally, for instance, "all-ok" is assumed, and filtered out of the report. As there are five kinds of messages that can be posted (including images), and each can be posted with one of five priorities, it is very easy to tune AQtest's reporting to just about any requirement.

The AQtest engine is an OLE server. Everything that is available through the AQtest user interface, is equally available to any application that links in one file, AQtestConnect, making it a "connected" application. AQtest is totally agnostic as to whether the test code runs as a script, interpreted by AQtest, or as compiled code in the application. A connected application may for instance test itself. In that case, its test code has full access to internal interfaces. This allows a unit test harness to test out the unit and simply use AQtest to log its results.

An Object Model for application control

Except for the case of self-testing applications, where test code also has internal access, all AQtest test code -

-- addresses applications and their environment through one COM-based object model.

This object model is exposed not merely through the script text as it is recorded, but more usefully through an Object Browser that (selectively) shows all processes running on the system, every on-screen object they involve and the public properties and methods of these objects. Properties are marked to show whether they can be changed externally or not. Their current value can be clipped from the Browser and pasted into test code.

Properties shown in the Browser and accessible in test code may be visible or not, as long as they belong to an on-screen object. Properties marked as "changeable" can be changed directly in test code, independent of the presence or absence of any user control to do it on-screen.

As delivered, the AQtest engine and browser extend this object model to -

In the following recorded script, for instance, on the last line AQtest recognizes the Select method of a TPageControl window in a Delphi application -

w.Window('TPageControl').Select('Notes');

The following code is hand-written (OLE objects are not per se on-screen windows). It manipulates a Word document -

o := Sys.GetOleObject('Word.Application');

o.Visible := True;

o.Documents.Add;

Document := o.Documents.Item(1);

Document.Paragraphs.Add;

Document.Paragraphs.Add;

The capacity of the object model to reach into applications can be extended in two ways.

First, an application may be compiled with AQtestClient linked in. It will then run as a child process. Depending on the development tool used (Borland Delphi or C++Builder, Microsoft Visual C++ or Visual Basic), a number of internal elements will be exposed to the AQtest server, and hence to any external test code. In the case of Borland applications, all published methods and properties become visible, and can be acted upon. The same is true for public methods and properties of the VCL, and also for those of other classes, once the source has been given a wrapper by a provided utility. These applications are called "open applications". AQtest itself is extensible through plug-ins, and access into open applications is likely to evolve further in the coming months.

Second, access to uninstrumented applications can also be extended through plug-ins. Already, object-model support for the Win 32 API, ADO and BDE comes through plug-ins delivered and installed with AQtest.

Script basics

A "script", by convention, is a single, external test procedure or function (depending on language). There are three scripting languages available, DelphiScript (a subset of Object Pascal), VBScript and JScript.

Because recording is a very useful facility in itself, C++ is defined as fourth, output-only language. The resulting code cannot be interpreted by AQtest itself, but it can be integrated into C++ connected applications.

The script, or test, must do three things:

AQtest provides full recording facilities at the level of Windows controls. More exactly, this is at the level of the visible elements in the common object model.

Where the application uses visible elements that are not "true" controls - that do not have their own window class - the recorder resorts to the next-best reference system, mouse and keyboard commands, and positions relative to the enclosing control. For instance, the following recording inputs a text field and clicks a button in a VCL TGroupBox belonging to a Delphi application. TGroupBox is a Windows control, but its elements are not.

w := p.Window('TForm1', 'HtmlSho: Create html file showing images');

w.Activate;

w.Window('TGroupBox').Click(328, 106);

Sys.Keys('[BS][BS]any');

w.Window('TGroupBox').Click(184, 107);

Further simulated input can be coded in directly. A full-featured editor is provided.

For special cases, separate "low-level procedures" can be recorded, within a main script. These use absolute screen positions, time all delays and record events at the highest Windows definition. A low-level version of the code above would have no reference to any Windows control. It would record a series of events, such as keyDown, keyUp, buttonDown, etc., with all relevant parameters, including delay since previous event, key or button and absolute screen position. A separate, special editor is provided for low-level code because such code would be simply too tedious to edit in the normal manner. The low-level editor, for instance, allows any group of selected events to have its delays expanded or compressed, or its positions modified spatially.

The same exact facilities are provided for output gathering, as for simulating input. Controls can directly be identified in recording mode, or from the Object Browser, and their properties read directly. Pixel-precise screen clips, the output equivalent to low-level input, can also be defined as objects in the code, and saved to file if needed.

Finally, the object model provides a full library of comparison functions for many purposes, right down to searching for one image within another. Using library functions and the usual programming operations, the script defines the messages it will send to the log, sends them and then terminates - the normal purpose of running any script is getting messages into the log.

As explained in the Object Model section, any script or test code has access to the entire internal structure of AQtest. But the minimum it needs are the message methods of the Log object. Of course, comparisons and evaluations cannot be recorded, they must be hand-coded. It is easy to divide the task, and have one script (procedure) take care of comparisons for any other script that calls it.

Some exceptional uses of the script recorder do not imply comparison and message posting. For instance, one might want to record precisely what a user is doing. Or a low-level recording can be used as a simple robot driver for software demonstrations.

When the script is run in the AQtest interpreter, the user is provided with the same kind of debugging facilities (conditional breakpoints, evaluations, etc.) as are available in professional development tools.

Systematic openness

Openness has been a primary architectural target of AQtest from the start. Proprietary interfaces and formats are avoided, and everything possible is documented. This underlying, implicit openness broadens the usability of all features, and extends the circumstances under which AQtest can be exploited.

The following lists outlines some of the features that make AQtest an open and flexible framework for your testing needs:

For more information on AQtest and other Quality Assurance products from AutomatedQA, write to us at: sales@automatedqa.com