A few days ago I was poking around the web for ideas about how to test software, and I saw Scott Ambler’s article about “Full Life Cycle Object-Oriented Testing (FLOOT).” The article includes a list of common testing techniques. As I looked over the list, I noticed that there is a small set of key dimensions that distinguish one testing technique from another. For example, unit testing and system testing differ in the kind of component they test. Stress testing and usability testing differ in the quality attribute that they test for. Unit testing and acceptance testing differ in the nature of the decisions that are made based on the test results.

I love looking for patterns like that, so I spent an hour analyzing Scott’s to identify the dimensions. Here are thirteen dimensions I found, and a few examples that show how different testing techniques vary along each.

Unit Under Test. What type of component being tested?

• In Class Testing or Unit Testing, the unit under test is a class.
• In Method Testing, the unit under test is a method of a class.
• In System Testing, the unit under test is the system.

Test Case Scope. What is the scope of the interaction tested by each test case?

• In Use-Case Scenario Testing, the scope of the interaction tested by each test case is a user goal.
• In Unit Testing, the scope of each test case is a method invocation.
• In Integration Testing, the scope is a transaction.

Unit Coverage. What subset of the unit under test is exercised by the test suite?

• In Coverage Testing, the subset being exercised by the test suite is code statements.
• In Path Testing, the coverage is logic paths.
• In Regression Testing, the coverage is code changes.
• In Boundary-value Testing, the coverage is limits.

Behavioral Scope. What subset of the unit-under-test’s behavior is being tested?

• Installation Testing tests the system’s installation procedure.
• Functional Testing tests the system’s business functionality.
• Integration Testing tests interactions among subsystems.

Unit Relationships. What are the relationships among the units whose interactions are being tested?

• In Inheritance-regression Testing, the relationship between units is inheritance.
• In Integration Testing, the relationship is collaboration or peers.

Quality Attribute. What type of quality attribute is being tested?

• In Stress Testing or Volume Testing, the quality attribute being tested is throughput or latency or capacity.
• In Usability Testing, the quality attribute being tested is usability.

Stakeholder. Whose interests are the focus of the testing?

• Acceptance Testing focuses on the interests of users.
• Operations Testing focuses on the interests of operators.
• Support Testing focuses on the interests of support staff.

Liveness. How closely does the test environment mimic the operational environment. Or perhaps this dimension is better characterized as Safety: To what extent are the testers using the system to do the real work for which the system was intended?

• In a Pilot, the test is the actual operational environment, perhaps limited in scope (e.g. a small subset of users, or for a limited time).
• In Beta Testing, the environment is a fully operation environment, but perhaps used only for non-critical functions.
• In Acceptance Testing, the environment is a non-operational similar to the operational environment.
• Unit Testing is done in the development environment.

Visibility into Unit Under Test. To what extent does the tester exploit knowledge about the internals of the unit under test?

• In Black-box Testing, the tester exploits no knowledge knowledge of internals of the unit under test.
• In White-box Testing, the tester exploits full knowledge of internals.
• In Grey-box Testing, the tester exploits some knowledge of internals.

Tester. What is the relationship of the tester to the software under test?

• For Acceptance Testing or User Testing, the tester is a user of the software.
• For Unit Testing or Developer Testing, the tester is a developer of the software.

Processor. What type of “processor” will “executes” the “software” during the tests?

• In most kinds of testing, a computer executes the software.
• In Code Inspections and Design Reviews, developers “execute” the software.
• In Prototype Walkthroughs, user “execute” the “software.”

Pre-Test Confidence. How confident are we about the software before we begin the testing?

• Before Alpha Testing, our confidence in the software is lower (compared with Beta Testing).
• Before Beta Testing, our confidence in the software is higher (compared with Alpha Testing).

Decision Scope. What kinds of decisions will we make based on the outcome of the test?

• For Acceptance Testing, the key decision is shell to release the product.
• For Integration Testing, the decision may be whether to begin system testing.
• For Unit Testing, the decision is whether the current coding task is complete.

This list is based on only an hour’s work, and on my analysis of only a single list of testing techniques (Scott’s), so I don’t claim that it is anywhere near complete or correct. It might be useful, though, for people who want to expand their repertoire of testing techniques, or to locate a technique that fits a given purpose or context.

I wonder what would happen if we created a thirteen-dimensional matrix. What parts would of the matrix would be crowded with testing techniques? What parts would be empty?

Thirteen dimensions is more than I can handle. So what would happen if we took two or three dimensions at a time and explored all of the values along those dimensions? Would that be interesting? Would it be useful? Would it help us to identify testing techniques that fit our specific situations? Might we notice holes in the matrix for which we want to invent useful techniques?

Employment relationships are based on an exchange of value. Jane provides products and services to her employer, Phil, in exchange for money. Phil provides money to Jane in exchange for her products and services. (There may be other values involved, such as an opportunity for Jane to do interesting work, but let’s keep this simple.) The exchange works as long each party values what they receive at least slightly more than what they give.

If one party or the other is not gaining in the exchange, either can choose (among other options) to end the employment. If Jane isn’t satisfied that the money is worth her time and energy, she can leave the company. If Phil isn’t satisfied that Jane’s products and services are worth the money, he can fire Jane. These choices give each party power with the other. Each has control of something the other wants, and the choice of whether to provide or withhold it. As long as the value they provide each other is reasonably similar, each party has similar power in the relationship.

Though Phil and Jane each have equal choice about continuing the employment relationship, there is a factor that swings power in the direction of the employer: aggregation.

Aggregation is one of the fundamental survival strategies of living systems (and non-living systems, too). An aggregate is a system that is made up of a critical number of more or less uniform pieces. A colony of ants is an aggregate made up of more or less uniform ants.

The power of aggregation as a survival strategy comes largely from what Jerry and Dani Weinberg, in their brilliant book
General Principles of Systems Design
, call
The First Law of Aggregates: When it comes to survival, aggregates outlive their worst members. If the weakest ant dies, the colony survives.

Aggregates abound in living systems. A sea turtle lays hundreds (or is it thousands?) of eggs. If only one of her eggs survives to produce offspring of its own, her lineage survives the death of thousands. If none of her eggs survives, thousands of other sea turtles are laying eggs. If enough of their offspring survive, the species continues despite the deaths of millions.

That is the power of aggregation. That power is based in massive redundancy.

Human engineers use the power of aggregation when they build technical systems. The lighting for a football or baseball stadium, for example, consists of a dozen banks of lights, each bank including dozens of bulbs. If one bulb burns out, the light from the bank diminishes slightly. Suppose that instead of banks, a stadium were lit by twelve enormous, high-intensity bulb. The failure of a single bulb would reduce the quality of lighting significantly, perhaps putting the safety of the players at risk. In addition to survival, aggregation helps systems to degrade gracefully rather than catastrophically.

Organizations tap the power of aggregation. Phil has lots of employees in addition to Jane. If Jane decides to leave the company, the performance of Phil’s group will decrease, but not collapse.

Jane, on the other hand, is less likely to have aggregation on her side. Where Phil has lots of employees providing products and services, Jane has a single source of income. If Phil decides to fire Jane, she will lose that single source. Where Phil may suffer a bearable loss of production, Jane suffers a possibly catastrophic loss of income.

Now, Jane may have a few aggregates of her own. She may have three other companies bidding to hire her. She may have a home business on the side. She may have a husband with income. She may have a small fortune in the bank. She may have a large pile of stock options, giving her what Silicon Valley engineers in the dot com boom called “fuck you money.” If she has these things, her income (or wealth) may degrade merely significantly instead of catastrophically.

It is rare for employees to have aggregates that match those of their employers. Aggregation tips the balance of power in favor of employers. Aggregation is the mechanism that creates the imbalance of power between employers and employees.