Is Pairwise Testing All It’s Cracked Up To Be?

Leave a comment

By Morgan McCollough, Senior Software Engineer

Morgan McCollough Bridge360Testing software can be a very complicated, time-consuming process. The number of possible inputs to a system and the ways in which they interact can quickly create a situation where the set of possible test cases is, for all practical purposes, unbounded.

In order to find the problems in a piece of software, a tester’s job is in large part about finding intelligent ways to pare down the possible test scenarios in such a way that the majority of problems are found in a reasonable amount of time.

To this end many techniques have been proposed over the years as shortcuts to finding the best set of test cases for locating the most software bugs in a time-efficient manner. One such technique that seems to be gaining in popularity is Pairwise Testing, also referred to as “All Pairs”.

Pairwise testing is a technique for generating test cases based on the idea that most software faults are the result of the interaction of at most two factors. It is an extension of the technique called domain partitioning in which each input to a test is imagined as a set of domain values, and test scenarios are generated by using the principles of combinatorics to produce a test suite where all combinations of two values are represented.

The resultant test suite is much smaller than one containing the exhaustive set of combinations. The theory is that the smaller set is still very efficient at finding bugs and is orders of magnitude quicker to execute. For example, a software system with 10 parameters that each has 10 possible values would require 10,000,000,000 test cases to try all possible combinations. Testing all possible pairs would only require 177 test cases.

The good news is that there are a number of both open source and commercial tools that can calculate the actual Pairwise test cases given the input parameters and the possible values. The bad news is that choosing the possible values for each parameter may not be a simple task.

Applying this technique as a “best practice” to every combinatorial testing problem may not yield the results you would expect. Specifically, pairwise testing suffers from the following problems:

  1. The nature of the interdependency of the variables in the product may be difficult to determine. In other words, the values may or may not be independent and possible relationships may be much more complicated.
  2. The occurrence of specific combinations in the real world is not evenly distributed, and this type of testing may not put enough emphasis on the most common user scenarios.
  3. The severity of the problems found may be limited, while catastrophic failures might be among those missed.
  4. The specific variables that the tester decides to combine may not end up exercising the software in the desired ways.
  5. The specific values selected for testing may not represent those with the potential to cause the most problems; or, they may not be the ones most commonly encountered in the real world.
  6. The combinations of values actually tested could possibly miss major failures depending on the nature of the interdependencies between variables.
  7. The technique is dependent on the tester’s ability to actually detect a bug when it happens.

The reality is that any specialized testing technique or shortcut must be understood, selected carefully, and applied wisely. There is simply no magic bullet for good testing. But Pairwise testing may be one good tool to incorporate into your testing protocol. To learn more about the technique, here are some links that provide a good introduction to this testing method:

http://en.wikipedia.org/wiki/All-pairs_testing

http://www.pairwise.org/

http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

Author: bridge360blog

Software Changes Everything.... Bridge360 improves and develops custom application software. We specialize in solving complex problems at every phase of the software development lifecycle, removing roadblocks to help our clients’ software and applications reach their full potential in any market. The Bridge360 customer base includes software companies and world technology leaders, leading system integrators, federal and state government agencies, and small to enterprise businesses across the globe. Clients spanning industries from legal to healthcare, automotive to energy, and high tech to high fashion count on us to clear a path for success. Bridge360 was founded in 2001 (as Austin Test) and is headquartered in Austin, Texas with offices in Beijing, China.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s