Dimensions of Software Testing

April 29, 2004 | Tags:

A few days ago I was poking around the web for ideas about how to test software, and I saw Scott Ambler’s article about “Full Life Cycle Object-Oriented Testing (FLOOT).” The article includes a list of common testing techniques. As I looked over the list, I noticed that there is a small set of key dimensions that distinguish one testing technique from another. For example, unit testing and system testing differ in the kind of component they test. Stress testing and usability testing differ in the quality attribute that they test for. Unit testing and acceptance testing differ in the nature of the decisions that are made based on the test results.

I love looking for patterns like that, so I spent an hour analyzing Scott’s to identify the dimensions. Here are thirteen dimensions I found, and a few examples that show how different testing techniques vary along each.

Unit Under Test. What type of component being tested?

Test Case Scope. What is the scope of the interaction tested by each test case?

Unit Coverage. What subset of the unit under test is exercised by the test suite?

Behavioral Scope. What subset of the unit-under-test’s behavior is being tested?

Unit Relationships. What are the relationships among the units whose interactions are being tested?

Quality Attribute. What type of quality attribute is being tested?

Stakeholder. Whose interests are the focus of the testing?

Liveness. How closely does the test environment mimic the operational environment. Or perhaps this dimension is better characterized as Safety: To what extent are the testers using the system to do the real work for which the system was intended?

Visibility into Unit Under Test. To what extent does the tester exploit knowledge about the internals of the unit under test?

Tester. What is the relationship of the tester to the software under test?

Processor. What type of “processor” will “executes” the “software” during the tests?

Pre-Test Confidence. How confident are we about the software before we begin the testing?

Decision Scope. What kinds of decisions will we make based on the outcome of the test?

This list is based on only an hour’s work, and on my analysis of only a single list of testing techniques (Scott’s), so I don’t claim that it is anywhere near complete or correct. It might be useful, though, for people who want to expand their repertoire of testing techniques, or to locate a technique that fits a given purpose or context.

I wonder what would happen if we created a thirteen-dimensional matrix. What parts would of the matrix would be crowded with testing techniques? What parts would be empty?

Thirteen dimensions is more than I can handle. So what would happen if we took two or three dimensions at a time and explored all of the values along those dimensions? Would that be interesting? Would it be useful? Would it help us to identify testing techniques that fit our specific situations? Might we notice holes in the matrix for which we want to invent useful techniques?