Abstract
This article describes the one of the most intractable
problems facing software development teams today; the cost
of testing. According to most pure Systems Engineering
Analysis experts the testing of a System requires at least
as much effort as its development. In reality, the budget
for testing a system is a great deal less than the budget
created to develop it in the first place. In many instances,
fully testing or retesting a system is simply not feasible.
It would take too much time and cost too much to fully test
the system. Therefore the challenge of the test team is to
find a way to detect and fix
most of the major problems in a system prior to its
deployment. The problems of testing fall upon two actors in
a system’s development, namely, the engineer and the
integration tester. For the engineer the challenge is that
as the information system becomes better defined, its specification
becomes more disconnected from the original requirements.
For the integration tester the problem is that tests must be
developed to test the requirements and
threads of the system rather than insuring that all the
developed features are exhaustively tested. This usually
occurs despite an ever shrinking testing budget. In
analyzing the problems faced by both engineers and testers a
number of key facts emerge that could help the test
engineer. Testing to requirements is only a valid test of
the software if the requirements have been kept in
synchronization with the code as it is developed. Reducing
test personnel will not go far enough in reducing testing
costs because most test teams are at their minimum staffing
level already. The key is automatic generation of tests
based on a static analysis of the system’s source code.
In this
informational paper we
introduce the concept of a Test Boundary. A Test Boundary is
a virtual wall around a portion of a software system that
represents the portion of the code that is evaluated during
a specific run of a test. It provides a conceptualization of
the system not only along its programmatic details, but also
around the elements that are executed during a particular
test. Some of these Test Boundaries already identified
include the “Unit Test Boundary”, the “Change Test
Boundary”, and the “Change Impact Boundary”. The objective
is to determine ways to generate these boundaries such that
a minimum number of tests exercise the maximum amount of
code in the software system. Trident Systems has introduced
a technology called Safe Test Boundaries that is designed to help
both developers and test engineers manage and generate test
boundaries for a software system.
Testing
Software in the Current Environment
The
technology of today's software is becoming more complex as larger,
componentized, and multithreaded/multi-processor systems are
developed. This complexity is even more pronounced as the
requirement for interoperability is demanded both from new
systems and legacy applications. In many of these systems
the ability to fully re-test when upgrades are made is
simply not feasible. Budgets for integrated testing are
constantly being cut while the time required to fully
test/re-test a large system is simply not feasible. As such
integrated test teams must make tradeoffs, hoping to detect
the most pressing deficiencies while leaving the more
obscure problems for a later maintenance upgrade.
However in
reality a maintenance upgrade often is bundled together with
new functionality that often introduces more high priority
deficiencies that have to be re-tested and dealt with. In
many cases the low priority problems that are detected are
never repaired.
The question
becomes how to test more while at the same time reduce the
time and expense of testing complex applications as they
become larger and more complex. In order to answer this
question we’ll need to look at the current practices that
make the testing of large complex systems so difficult and
expensive. First we’ll look at the problem faced by many
engineers. Next we’ll look at the problems faced by many
testers.
In this
document Trident Systems will present a number of diagrams
representing a portion of a software system. These diagrams
are directed graphs in which the nodes represent some source
code element such as a class or a function and the edges
represent software dependencies between two nodes.
The Engineers Problem
Most large and complex software projects undergo a rigorous
engineering process that defines the data elements and the
program in every detail prior to the beginning of its
development. This process begins with the definition of the
requirements of the software. Then the process proceeds to
the software architecture models, followed by the design and
finally the detailed design. Once the detailed design
documentation is completed the Systems Engineering is said
to be complete and the software development can begin.
The problem here is that the Systems Engineering evolves
through a number of steps that creates an ever-increasing
level of detail. Each step in the process becomes further
removed from the original requirements definition. And it is
from the requirements definitions that the tests are
developed. The development of the architecture models may
drive the creation of new requirements as should the design
and detailed design phases of the project. However, for
complex projects it is difficult to insure that all the
issues determined during the detailed design get pushed back
into the requirements
specification.
Once software development begins this “disconnect” between
requirements, design and actual code developed becomes even
more pronounced. Often individuals who have no experience
with systems engineering at all are the ones who perform the
software development. This means that these individuals will
not be able to recognize when they, in solving a
particular problem have created a situation that requires a
change in the requirements.
The key to solving this problem is to either insure that the
development and maintenance of the program stays connected
to the engineering documents or to regenerate the
engineering documents based on analysis of the software. The
industry has been unsuccessfully trying to develop a process
to keep engineering documents up to date for the last twenty
years. One method to keep these documents up to date is to
perform software analysis through white-box dependency
testing. That is, the development of a custom source code
parser to produce a dependency graph so that additional
analysis may be done on the actual code that was developed.
This has certain advantages over manually keeping
documentation up to date in that it relies on the product of
the development process. It also has advantages over using
exiting compilers in that the custom parser/analyzer may
perform optimizations not available in a standard source
code parser.
The Testers Problem
Once an application is developed an integration phase
usually begins in which all the various pieces of the
application are fused together into the final application.
This phase generally exists for all large software
development efforts including large upgrades and
maintenance. Up to this point developers and development
teams have been conducting ad-hoc unit testing of their
assigned bit of the application using whatever data sets
they could cobble together during their development.
Integration is usually as complex as the application under
development. In today’s environment, in which applications
are being re-engineered to be componentized so that they may
provide services to clients, this level of complexity is
even greater. Components are being integrated that had no
connection to each other when the technology was originally developed.
Multi-threaded, multi-processor, and, multi-computer
applications also are difficult to test and debug because of
the asynchronous nature of the environment. Processes that
work just fine in a synchronous environment may fail
unpredictably when placed in an asynchronous environment.
During the Integration Testing Phase an independent test
team usually conducts tests of the project based on the
detailed requirements. Here is one point at which
inaccuracies can crop up between software and testing. If
the Engineering documentation is not kept up to date, then
there is no way for the test team to accurately know what
features to test for. Automatic Test software typically
passes data through components or creates events that
simulate/stimulate the selection of user interface controls
to see how the applications behave under specific
conditions. However, this method of testing often does not
exercise all the code developed for an application.
The big question facing business today is how to reduce the
cost of testing while maintaining good quality of software
produced. The easiest way to reduce the cost of testing is
to reduce the number of personnel engaged in the testing
process. However, in most cases, the test team is already at
a minimum and reducing the size of the team may not be
feasible or advisable. It may, in fact be a single
individual developing and running tests for a large and
complex application. The job of the test team is to develop
the tests that will be performed on the application, insure
that all branches of the developed system are tested
adequately, develop the data that must be used to test the
system, and
finally, to run the tests on the system. This process is
time consuming and labor intensive.
Automated generation of these artifacts is the key to
reducing the cost of testing. If the tests and the test data
can be generated automatically by some application to suite
of applications, then a large part of the effort in creating
the tests is reduced. However, in order to generate the
tests that an application needs to perform, a detailed
understanding of the software structure is required. We
come, then to a common need for an analysis of the structure
of the software application in order to regenerate our
engineering documentation and to generate complex tests from
simple ones.
Some approaches for analyzing a software application involve
reverse engineering the object code or reverse engineering
the application itself. While these activities are useful
when the need exists to discover the structure of an
application when source code is not available, there is one
primary deficiency with this approach in the area of
test/retest generation. Reverse engineering is not able to
distinguish between the portions of the software that were
developed and those that were integrated. Developed software
means the actual code that is what makes the application
unique and solves the particular problem or performs the
particular function that the application was designed to do.
Integrated software means software that is part of the
compiler environment, or the operating system, or third
party libraries that were incorporated into the application
to provide the infrastructure on which the application was
developed. While some may argue that many of the operating
system features or integrated third party libraries are
provided as run-time libraries many of the features of the
compiler environment are linked directly into the
application making it impossible to distinguish where the
functions of the software leaves off and the compiler
environment begins.
|
