Permutations & Probability - A Revisit to your Math Class - Part III (Do it Like the Hippies)
15.01.12 - 11:13 - Filed in: Software Testing
Imagine a commune of five long haired hippies. Let’s call them Andy, Bruce, Claire, Denise and Emma. And - hey - it is the time of free love so they live in alternating relationships. No room for conservative thoughts here, the hippies were free thinkers, so all pairs work. Let’s have the following pairs as a start: Anthony & Claire and Bruce & Denise. Poor Emma is not in a relationship and therefore somewhere with with the earth goddess pacha mama dancing barefootly while gently humming a tune.
But, we’re drifting away here.
Now the question: How many possibilities are there for relationships, before our five hippies get bored and become law abiding white protestant middle-class work drones?
You remember the lottery from last time? 45 choose 7? Ok, this here is a simple 5 choose 2. And there are 10 possibilities for relationships. See below.
This leads us to something powerful used in Software Testing: Pairwise Testing
But what is it?
Pairwise testing is a combinatorial testing method where within a given set of parameters with each a finite set of values all value combinations are tested and where you want to keep the total number of tests to a minimum.
When testing exhaustively with the array below the total number of tests would be five to the power of five = 3125
Hm, well, with decent automation this does not sound that unfeasible, does it?
You’re right. But now lets have a look at this. Does not look that much bigger. But actually we now have a total of 10’000’000’000 possible test cases. This is obviously far too many.
It is often stated (and there is some evidence here, see chapter 3 “Empirical Data” ) that errors happening in a program are most often in the combination of two parameters and become less likely when combining three and more. Therefore it would make sense to test covering all pairs.
With each test run you cover several pairs. So for instance with our hippies A, B, C, D, E you would already cover 10 different pairs (as exemplified above).
Assuming that the combination of A1 and H9 leads to a bug, with pairwise testing - wait, now it comes - you win your own special little lottery because you actually find the bug. That’s how you earn respect with the developers. Isn’t that cool?
Information becomes knowledge when applied. Therefore I would recommend to try it out for yourselves. Here are some tools I find useful:
Now a word of warning: Some might say: “Let us declare pairwise as best practice, cast it in stone and apply it readily everywhere”. No, not at all. Just because using explosives is excellent for tunnel building (or more abstractly “to make a hole” ) you do not blow up your house every time you need to have a whole for a screw to hang up a picture. Pairwise testing is useless in some contexts. I am not going to explicitly name any, as it should be obvious.
One more thing: I would like to be the advocate of our sapient testers (often still called “manual testers” as if writing the code for automated tests was not done using the hands of the developers). Whenever possible, automate this kind of testing.
What more is there?
Pairwise Testing: A Best Practice That Isn’t
pairwise.org
Pair Wise Testing
And there is even a little conference:
Workshop on Combinatorial Testing
Now go and spread the word. Send out the links. Let the RSS feed rock n’roll. And comment like you haven’t commented ever before. I’m ready to rumble in the jungle.
But, we’re drifting away here.
Now the question: How many possibilities are there for relationships, before our five hippies get bored and become law abiding white protestant middle-class work drones?
You remember the lottery from last time? 45 choose 7? Ok, this here is a simple 5 choose 2. And there are 10 possibilities for relationships. See below.
This leads us to something powerful used in Software Testing: Pairwise Testing
But what is it?
Pairwise testing is a combinatorial testing method where within a given set of parameters with each a finite set of values all value combinations are tested and where you want to keep the total number of tests to a minimum.
When testing exhaustively with the array below the total number of tests would be five to the power of five = 3125
Hm, well, with decent automation this does not sound that unfeasible, does it?
You’re right. But now lets have a look at this. Does not look that much bigger. But actually we now have a total of 10’000’000’000 possible test cases. This is obviously far too many.
It is often stated (and there is some evidence here, see chapter 3 “Empirical Data” ) that errors happening in a program are most often in the combination of two parameters and become less likely when combining three and more. Therefore it would make sense to test covering all pairs.
With each test run you cover several pairs. So for instance with our hippies A, B, C, D, E you would already cover 10 different pairs (as exemplified above).
Assuming that the combination of A1 and H9 leads to a bug, with pairwise testing - wait, now it comes - you win your own special little lottery because you actually find the bug. That’s how you earn respect with the developers. Isn’t that cool?
Information becomes knowledge when applied. Therefore I would recommend to try it out for yourselves. Here are some tools I find useful:
Now a word of warning: Some might say: “Let us declare pairwise as best practice, cast it in stone and apply it readily everywhere”. No, not at all. Just because using explosives is excellent for tunnel building (or more abstractly “to make a hole” ) you do not blow up your house every time you need to have a whole for a screw to hang up a picture. Pairwise testing is useless in some contexts. I am not going to explicitly name any, as it should be obvious.
One more thing: I would like to be the advocate of our sapient testers (often still called “manual testers” as if writing the code for automated tests was not done using the hands of the developers). Whenever possible, automate this kind of testing.
What more is there?
Pairwise Testing: A Best Practice That Isn’t
pairwise.org
Pair Wise Testing
And there is even a little conference:
Workshop on Combinatorial Testing
Now go and spread the word. Send out the links. Let the RSS feed rock n’roll. And comment like you haven’t commented ever before. I’m ready to rumble in the jungle.
blog comments powered by Disqus