By Paul Bostrom, Senior Software Engineer
Do your software testing teams ever discover bugs that seemingly should have been found by the developers’ unit tests? Quite often, the developer actually did unit test the software, but perhaps simply failed to think of scenarios using the problematic inputs. What if we could tell the computer to “think” of all the values used in our unit tests? This is the approach of property-based testing.
Instead of specifying a limited number of inputs and outputs for testing a unit of software, property-based testing specifies properties that a unit of software must hold, and then relies on the computer to generate the test values.
The authoritative library for property-based testing is called QuickCheck (http://en.wikipedia.org/wiki/QuickCheck), created for the Haskell programming language, but implementations of the library exist for many other popular programming languages. To illustrate the differences in these two testing approaches, we will use a simple example — testing a square root function.
A typical unit testing approach involves using a testing framework that allows us to state assertions that our function, ‘sqrt()’, should satisfy for specific inputs and outputs. Our assertions might be in the form of ‘sqrt(16) == 4’, and ‘sqrt(36) == 6’. Our test framework will automatically run all of our assertions, and give us some amount of confidence that our function is implemented correctly.
The downside of this approach is that we must manually specify our assertions, which takes time and mental energy to think of so-called “boundary” or “corner” cases that our function might have trouble handling. We may also have a tendency to focus on getting our finite set of tests to pass without thinking of the broader requirements that our function should hold. Another limitation of this unit testing approach is that we don’t gain any advantage by running our test cases multiple times, as it will always return the same results given our specific inputs.
When we incorporate property-based testing into our development process, we greatly increase the range of inputs that our software is tested against. Instead of specifying a set of inputs and outputs for testing our function, we would use a property-based testing library to specify a set of properties that the inputs and outputs must hold. The library will then generate a much greater number of test cases than we are able to specify in our unit testing framework. These test cases will also be randomly generated, so running our test multiple times will increase the variation of the inputs tested. I will use the ‘sqrt’ function to demonstrate how this works.
A typical property-based testing library should include functionality that generates random positive integers. Using the library we can specify a property that for every generated integer, we can multiply it by itself to generate the square, and then pass the squared value to the ‘sqrt’ function. The code for specifying this property may look like ‘i == sqrt(i * i)’ for every positive integer ‘i’ generated. Once we have defined our property, we can run the test specifying the number of random inputs to generate. The testing library will then handle the work of generating the inputs and verifying that the property holds for each input. Since the inputs are randomly generated, running the test multiple times increases the chances that we will find a value that causes the property to fail.
Most property-based testing implementations also have a feature that will attempt to ‘shrink’ a failing test case to a minimal example. This is a useful feature when debugging that allows us to isolate problematic inputs.
Property-based testing is useful when we can easily define the properties that our software should hold. It requires some additional effort over traditional unit testing, but the increased testing variation should uncover more bugs than unit testing alone.