By Morgan McCollough, Senior Software Engineer
Testing software can be a very complicated, time-consuming process. The number of possible inputs to a system and the ways in which they interact can quickly create a situation where the set of possible test cases is, for all practical purposes, unbounded.
In order to find the problems in a piece of software, a tester’s job is in large part about finding intelligent ways to pare down the possible test scenarios in such a way that the majority of problems are found in a reasonable amount of time.
To this end many techniques have been proposed over the years as shortcuts to finding the best set of test cases for locating the most software bugs in a time-efficient manner. One such technique that seems to be gaining in popularity is Pairwise Testing, also referred to as “All Pairs”.
Pairwise testing is a technique for generating test cases based on the idea that most software faults are the result of the interaction of at most two factors. It is an extension of the technique called domain partitioning in which each input to a test is imagined as a set of domain values, and test scenarios are generated by using the principles of combinatorics to produce a test suite where all combinations of two values are represented.
The resultant test suite is much smaller than one containing the exhaustive set of combinations. The theory is that the smaller set is still very efficient at finding bugs and is orders of magnitude quicker to execute. For example, a software system with 10 parameters that each has 10 possible values would require 10,000,000,000 test cases to try all possible combinations. Testing all possible pairs would only require 177 test cases.
The good news is that there are a number of both open source and commercial tools that can calculate the actual Pairwise test cases given the input parameters and the possible values. The bad news is that choosing the possible values for each parameter may not be a simple task.
Applying this technique as a “best practice” to every combinatorial testing problem may not yield the results you would expect. Specifically, pairwise testing suffers from the following problems:
- The nature of the interdependency of the variables in the product may be difficult to determine. In other words, the values may or may not be independent and possible relationships may be much more complicated.
- The occurrence of specific combinations in the real world is not evenly distributed, and this type of testing may not put enough emphasis on the most common user scenarios.
- The severity of the problems found may be limited, while catastrophic failures might be among those missed.
- The specific variables that the tester decides to combine may not end up exercising the software in the desired ways.
- The specific values selected for testing may not represent those with the potential to cause the most problems; or, they may not be the ones most commonly encountered in the real world.
- The combinations of values actually tested could possibly miss major failures depending on the nature of the interdependencies between variables.
- The technique is dependent on the tester’s ability to actually detect a bug when it happens.
The reality is that any specialized testing technique or shortcut must be understood, selected carefully, and applied wisely. There is simply no magic bullet for good testing. But Pairwise testing may be one good tool to incorporate into your testing protocol. To learn more about the technique, here are some links that provide a good introduction to this testing method:
http://en.wikipedia.org/wiki/All-pairs_testing
http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
