pair wise testing
Permutations & Probability - A Revisit to your Math Class - Part III (Do it Like the Hippies)
15.01.12 - 11:13 - Filed in: Software Testing
Imagine a commune of five long haired hippies. Let’s call them Andy, Bruce, Claire, Denise and Emma. And - hey - it is the time of free love so they live in alternating relationships. No room for conservative thoughts here, the hippies were free thinkers, so all pairs work. Let’s have the following pairs as a start: Anthony & Claire and Bruce & Denise. Poor Emma is not in a relationship and therefore somewhere with with the earth goddess pacha mama dancing barefootly while gently humming a tune.
But, we’re drifting away here.
Now the question: How many possibilities are there for relationships, before our five hippies get bored and become law abiding white protestant middle-class work drones?
You remember the lottery from last time? 45 choose 7? Ok, this here is a simple 5 choose 2. And there are 10 possibilities for relationships. See below.
This leads us to something powerful used in Software Testing: Pairwise Testing
But what is it?
Pairwise testing is a combinatorial testing method where within a given set of parameters with each a finite set of values all value combinations are tested and where you want to keep the total number of tests to a minimum.
When testing exhaustively with the array below the total number of tests would be five to the power of five = 3125
Hm, well, with decent automation this does not sound that unfeasible, does it?
You’re right. But now lets have a look at this. Does not look that much bigger. But actually we now have a total of 10’000’000’000 possible test cases. This is obviously far too many.
It is often stated (and there is some evidence here, see chapter 3 “Empirical Data” ) that errors happening in a program are most often in the combination of two parameters and become less likely when combining three and more. Therefore it would make sense to test covering all pairs.
With each test run you cover several pairs. So for instance with our hippies A, B, C, D, E you would already cover 10 different pairs (as exemplified above).
Assuming that the combination of A1 and H9 leads to a bug, with pairwise testing - wait, now it comes - you win your own special little lottery because you actually find the bug. That’s how you earn respect with the developers. Isn’t that cool?
Information becomes knowledge when applied. Therefore I would recommend to try it out for yourselves. Here are some tools I find useful:
Now a word of warning: Some might say: “Let us declare pairwise as best practice, cast it in stone and apply it readily everywhere”. No, not at all. Just because using explosives is excellent for tunnel building (or more abstractly “to make a hole” ) you do not blow up your house every time you need to have a whole for a screw to hang up a picture. Pairwise testing is useless in some contexts. I am not going to explicitly name any, as it should be obvious.
One more thing: I would like to be the advocate of our sapient testers (often still called “manual testers” as if writing the code for automated tests was not done using the hands of the developers). Whenever possible, automate this kind of testing.
What more is there?
Pairwise Testing: A Best Practice That Isn’t
pairwise.org
Pair Wise Testing
And there is even a little conference:
Workshop on Combinatorial Testing
Now go and spread the word. Send out the links. Let the RSS feed rock n’roll. And comment like you haven’t commented ever before. I’m ready to rumble in the jungle.
But, we’re drifting away here.
Now the question: How many possibilities are there for relationships, before our five hippies get bored and become law abiding white protestant middle-class work drones?
You remember the lottery from last time? 45 choose 7? Ok, this here is a simple 5 choose 2. And there are 10 possibilities for relationships. See below.
This leads us to something powerful used in Software Testing: Pairwise Testing
But what is it?
Pairwise testing is a combinatorial testing method where within a given set of parameters with each a finite set of values all value combinations are tested and where you want to keep the total number of tests to a minimum.
When testing exhaustively with the array below the total number of tests would be five to the power of five = 3125
Hm, well, with decent automation this does not sound that unfeasible, does it?
You’re right. But now lets have a look at this. Does not look that much bigger. But actually we now have a total of 10’000’000’000 possible test cases. This is obviously far too many.
It is often stated (and there is some evidence here, see chapter 3 “Empirical Data” ) that errors happening in a program are most often in the combination of two parameters and become less likely when combining three and more. Therefore it would make sense to test covering all pairs.
With each test run you cover several pairs. So for instance with our hippies A, B, C, D, E you would already cover 10 different pairs (as exemplified above).
Assuming that the combination of A1 and H9 leads to a bug, with pairwise testing - wait, now it comes - you win your own special little lottery because you actually find the bug. That’s how you earn respect with the developers. Isn’t that cool?
Information becomes knowledge when applied. Therefore I would recommend to try it out for yourselves. Here are some tools I find useful:
Now a word of warning: Some might say: “Let us declare pairwise as best practice, cast it in stone and apply it readily everywhere”. No, not at all. Just because using explosives is excellent for tunnel building (or more abstractly “to make a hole” ) you do not blow up your house every time you need to have a whole for a screw to hang up a picture. Pairwise testing is useless in some contexts. I am not going to explicitly name any, as it should be obvious.
One more thing: I would like to be the advocate of our sapient testers (often still called “manual testers” as if writing the code for automated tests was not done using the hands of the developers). Whenever possible, automate this kind of testing.
What more is there?
Pairwise Testing: A Best Practice That Isn’t
pairwise.org
Pair Wise Testing
And there is even a little conference:
Workshop on Combinatorial Testing
Now go and spread the word. Send out the links. Let the RSS feed rock n’roll. And comment like you haven’t commented ever before. I’m ready to rumble in the jungle.
Permutations & Probability - A Revisit to your Math Class - Part I
08.01.12 - 12:43 - Filed in: Software Testing
What’s the relationship between Testers and Mathematics? I don’t know. But I know what it should be. It should be one of affectionate care and love. Let’s have a short experiment: Upon hearing the following two words: Permutations & Probability. Don’t you immediately start to smile? I do.
The subject of permutations came up when I recently had a coaching session with James Bach and we had some discussions on pair wise testing and the selection of test cases.
For today let’s start simple:
If you are a team of five fabulous testers and you are keen to have a picture taken as a memory of the equally fabulous project you just finished. In how many different orders can you place yourselves smiling towards the camera?
The blunt answer without explanation is: 5!
The ones among you who have never liked mathematics may ask: “Why do you shout? Why do you put an exclamation mark after the answer that is not such a huge number after all?”
And I reply: That’s because it is bigger than you think and because the answer is not “a shouted 5”, but 5 factorial, which is 5 x 4 x 3 x 2 x 1 = 120 possible configurations of a picture of a fabulous team of 5 testers.
Why is that so? Well for the picture of 5 testers you have five slots: _ _ _ _ _ . And you have all 5 testers eagerly waiting to get on one of the slots. One of the five takes the first slot (so there are 5 possibilities for the first slot): 5 _ _ _ _ . Then the remaining 4 testers viciously fight for the second slot: 5 x 4 _ _ _. And so on until we have: 5 x 4 x 3 x 2 x 1
That leads us to a meaner example: Let us therefore assume you were not 5 testers on the project but 6. And now it comes: One of the six testers did not contribute to the success of the fabulousness at all. Obviously, you would not want to have that 6th tester on your memory picture. You now have 6 testers of which only 5 will eventually be on the picture: But, how many possibilities?
The answer is: 720
No shouting, no exclamation marks, but an amazing amount of possible constellations for bullying one of the 6 testers out of a memorial picture. The calculation goes as follows: 6 x 5 x 4 x 3 x 2.
That’s all for today. Next time there will be some hairy examples of ordered and not ordered sets and the calculation of the probability of winning the lottery.
Do I hear your voices demanding: “Come on, give us at least one more useful link!”?
Ok, here you go: Khan Academy
The subject of permutations came up when I recently had a coaching session with James Bach and we had some discussions on pair wise testing and the selection of test cases.
For today let’s start simple:
If you are a team of five fabulous testers and you are keen to have a picture taken as a memory of the equally fabulous project you just finished. In how many different orders can you place yourselves smiling towards the camera?
The blunt answer without explanation is: 5!
The ones among you who have never liked mathematics may ask: “Why do you shout? Why do you put an exclamation mark after the answer that is not such a huge number after all?”
And I reply: That’s because it is bigger than you think and because the answer is not “a shouted 5”, but 5 factorial, which is 5 x 4 x 3 x 2 x 1 = 120 possible configurations of a picture of a fabulous team of 5 testers.
Why is that so? Well for the picture of 5 testers you have five slots: _ _ _ _ _ . And you have all 5 testers eagerly waiting to get on one of the slots. One of the five takes the first slot (so there are 5 possibilities for the first slot): 5 _ _ _ _ . Then the remaining 4 testers viciously fight for the second slot: 5 x 4 _ _ _. And so on until we have: 5 x 4 x 3 x 2 x 1
That leads us to a meaner example: Let us therefore assume you were not 5 testers on the project but 6. And now it comes: One of the six testers did not contribute to the success of the fabulousness at all. Obviously, you would not want to have that 6th tester on your memory picture. You now have 6 testers of which only 5 will eventually be on the picture: But, how many possibilities?
The answer is: 720
No shouting, no exclamation marks, but an amazing amount of possible constellations for bullying one of the 6 testers out of a memorial picture. The calculation goes as follows: 6 x 5 x 4 x 3 x 2.
That’s all for today. Next time there will be some hairy examples of ordered and not ordered sets and the calculation of the probability of winning the lottery.
Do I hear your voices demanding: “Come on, give us at least one more useful link!”?
Ok, here you go: Khan Academy
Permutations & Probability - A Revisit to your Math Class - Part II (Winning the Lottery)
11.01.12 - 16:21 - Filed in: Software Testing
What is the equivalent of winning the lottery in the software tester’s context? How do you as a tester win your own special little lottery? In order to answer these questions let’s have a look on what the chances are that you win the “real” lottery. We can do that backwards, starting with the solution:
The chances for you to win the lottery (select 6 out of 45 numbers) is exactly 1 : 8’145’060
But, how do we know?
Ok, we have 45 different numbers or slots:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
For those of you who have not skipped my last post know that for each of the slots we have the following amount of declining possibilities to select numbers:
45 x 44 x 43 x 42 x 41 x 40 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
And because we only have to choose 6, we stop at 40.
This can also be expressed as:
Like: 45 x 44 x 43 x ….. x 3 x 2 x 1 divided by anything in the row below 40 as they are not needed. That is the elegance of math. Isn’t it beautiful?
I know, you have already quickly calculated the result in your head and you must have arrived with the number 5’864’443’200, which, obviously, is not equal to 8’145’060. And you are right. And I am right, too. I have not said I am finished yet. Sneaky me!
We now have the calculation of: “In how many ways can a set of 6 be chosen out of a stack of 45 including different orders of the six cards”. But the lottery does not care if one of the winning numbers is selected as a first number or as a last. Neither does the winner care in his bliss of the millions whether or not he crossed one or the other number first on his tiny receipt.
In how many ways can a set of 6 be grouped? You remember the evil mobbing bunch of memorial picture taking testers from the last post? No? Ok, here is the explanation once more: Six factorial, or 6 x 5 x 4 x 3 x 2 x 1 = 720
That is the number we have to divide the whole thing by, before we hop off to cash the winner’s cheque with the grumpy old lady behind the cashier’s desk.
Or written as a formula:
Or even more elegantly:
Or even more generally
-> BTW: If you want to actually say that to - let’s say: to a person who has very dark sun glasses and therefore cannot see the formula - then say: “n choose k” and everybody will immediately believe you are a highly professional software tester.
So
= 8’145’060
Which stated as a probability is:
QED
This leads us to the intriguing question: Why on earth am I dwelling on math formulas? That is because after having recommended Khan Academy in the last post, I ate my own dog food. And now I am hooked. I joyfully participate in Khan Academy’s exercises. And man, I wanna get one of those Black Hole Badges. Any would be ok, but “Tesla” is the one I fancy.
I need to add a 3rd part to this series in order to close the loop back to actionable value for software testing. Next time will be about pair-wise testing and why it is not “Best Practise”. Sometimes it is not even “Good Practise”. It may indeed be “Totally Crappy Practise” in some context.
Ah, almost forgot: In the next post I’ll also tell you how to win your own special little lottery (Justin: I’ll address your requests, too). So, stay tuned!
The chances for you to win the lottery (select 6 out of 45 numbers) is exactly 1 : 8’145’060
But, how do we know?
Ok, we have 45 different numbers or slots:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
For those of you who have not skipped my last post know that for each of the slots we have the following amount of declining possibilities to select numbers:
45 x 44 x 43 x 42 x 41 x 40 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
And because we only have to choose 6, we stop at 40.
This can also be expressed as:
Like: 45 x 44 x 43 x ….. x 3 x 2 x 1 divided by anything in the row below 40 as they are not needed. That is the elegance of math. Isn’t it beautiful?
I know, you have already quickly calculated the result in your head and you must have arrived with the number 5’864’443’200, which, obviously, is not equal to 8’145’060. And you are right. And I am right, too. I have not said I am finished yet. Sneaky me!
We now have the calculation of: “In how many ways can a set of 6 be chosen out of a stack of 45 including different orders of the six cards”. But the lottery does not care if one of the winning numbers is selected as a first number or as a last. Neither does the winner care in his bliss of the millions whether or not he crossed one or the other number first on his tiny receipt.
In how many ways can a set of 6 be grouped? You remember the evil mobbing bunch of memorial picture taking testers from the last post? No? Ok, here is the explanation once more: Six factorial, or 6 x 5 x 4 x 3 x 2 x 1 = 720
That is the number we have to divide the whole thing by, before we hop off to cash the winner’s cheque with the grumpy old lady behind the cashier’s desk.
Or written as a formula:
Or even more elegantly:
Or even more generally
-> BTW: If you want to actually say that to - let’s say: to a person who has very dark sun glasses and therefore cannot see the formula - then say: “n choose k” and everybody will immediately believe you are a highly professional software tester. So
= 8’145’060Which stated as a probability is:
QED
This leads us to the intriguing question: Why on earth am I dwelling on math formulas? That is because after having recommended Khan Academy in the last post, I ate my own dog food. And now I am hooked. I joyfully participate in Khan Academy’s exercises. And man, I wanna get one of those Black Hole Badges. Any would be ok, but “Tesla” is the one I fancy.
I need to add a 3rd part to this series in order to close the loop back to actionable value for software testing. Next time will be about pair-wise testing and why it is not “Best Practise”. Sometimes it is not even “Good Practise”. It may indeed be “Totally Crappy Practise” in some context.
Ah, almost forgot: In the next post I’ll also tell you how to win your own special little lottery (Justin: I’ll address your requests, too). So, stay tuned!