image credit: http://j.mp/wRWveO

Interviewer:
So, how do you handle your bugs?
Interviewee:
Finally, we don’t have to talk to each other anymore. Why should we? We have a bug tracking tool now. It’s called “Brave New Bug 1984” . Everything goes in there. Even the slightest suspicion is reported. Unit and integration tests that fail result in a report in the tool. That is something we the noble quality police fought hard for. And we have set high goals for ourselves. The next step will be that even if a developer e.g. makes a typo - something like: accidentally typing two closing brackets - he/she can no longer just correct it. It must go in the bug tracking tool. And then we will issue metrics and reports. About every single developer. We exactly know who the bad apples are, and then we can talk to HR directly, and we will fire people, and…[Interviewee starts to salivate]
Interviewer:
Have you lost your mind?

That’s right. This is the voice of madness. And hopefully there is no company that practices bug tracking that way. But - who knows - if it exists and you work there, then do yourself a favor and quit your job today.

Most of us use a tool to track bugs and I have a question: Why? What is the reason to use a tool for bug tracking? Hmm?

Our brains are very good at being creative, socializing with our friends, playing cards and drinking a beer. Not so good at keeping updated lists in our heads. That is where a bug tracking tool comes in handy. It outsources our list keeping. So, the bug tracking system prevents us from forgetting about things. Bugs, in this case.

But does that mean that every single little thing in every single little situation must go in the tracking tool? What about tester feedback from exploratory testing sessions at the daily stand ups in an agile environment? Is it really necessary to have it all in a tool? Maybe so, maybe not. Just decide for yourselves. Maybe even have some of the bugs tracked and others not. I don’t see it as an either this or that game.

Another made up reason for being strict about handling all bugs in the tool could be that you as a test manager are fond of reporting the total number of bugs in a project. “We have found 53’732 bugs during this project”. My reaction to that: “So what?”

Compulsive counting is something we used to do as school boys. In our case we used to brag about being in possession of 53’732 different posters from discotheques. You know, we used to drive around with our mopeds and take discotheque posters off the walls. They looked like this:



Ridiculous, isn’t it? But excused, because a weak beard growth does not really help being sensible.

So don’t become this compulsory counting freak in your company and if you feel like reporting any number then be sure it has some meaning. However, the number of bugs found does not inherently have meaning. It does not say anything about the quality of the product.

BTW: This very old article by Joel Spolsky is quite a good read and still valid.

tags: bug tracking, testing

Somewhere in between two releases we are born. At that milestone there is a whole suite of tests done with us. Boot-up complete - tested, 10 fingers - check, 10 toes - verified. Then of course our mother immediately introduces us into the mysteries of software testing by feeding us gently. We get to find our first bugs as toddlers, in the nursery school we draw our first test plans and in elementary school there is a comprehensive software testing curriculum.

During adolescence we naturally have to rebel against good practices and secretly meet with not so trustworthy people with whom we do some testing certifications. Later in life, we have to admit to ourselves that we were just young and foolish. Anyway, off we go into the glamorous work world, a master’s degree in software testing in our pockets and filled with a lot of energy to tackle even the most difficult testing problems.

All this rich life of working the software forms our fond memories we then tell the nurses in the elderly home, who are not really that interested in what we say, because software - of course - is an outdated concept and no longer being used in this future world. But that is something we won’t notice because our perceptional capabilities have already become very limited. Therefore we just sit there in our rocking chair and smile.

image credit: http://j.mp/wXYDLr

tags: skill, education

Try this: Go buy any new book. Then de-shrink-wrap it (i.e.: remove the book from its enveloping plastic awfulness). Place the book in front of you. Take it in your hands. Open the book somewhere. Meditate for a second. Then put your nose deep into the book and inhale.



Now comes the triage part. What is your reaction to that proposal? (mark your selection with a check mark)

1. That’s crazy talk! Why would I do that?
2. Yes, I know. Done it before. Ahhh, I just love the smell of new books.

If your answer was similar to 1. then stay with me and keep on reading. If you feel more intimate with answer 2. then you will probably stay with me anyway because you are a book lover and like to read.

It is amazing how many people in software testing have not read one single book on software testing. Or have even hardly read anything at all. Mostly the reason given is lack of time. But that is simply not true. Nobody can have lack of time because the flow of time is just as it is. There is no such thing as “lack” of it. It is 24 hours every single day. No deviation to that (given that you are a habitant of planet earth). The only difference is the priorities assigned to different activities and for some strange reason reading books seems to be de-prioritized by so many.

Just image e.g. a physicist, or a medical doctor stating: Bah, I haven’t had any time to read books on molecules or the human heart respectively. Would you trust such a physicist with a Large Hadron Collider? Would you trust your grandmother with such a physician?

Knowledge that may be helpful:

• Software Testing (ok, that was obvious)
• Software Engineering
• Psychology
• The art of drawing
• Philosophy
• Rhetoric and Persuasion
• How to describe well
• Literature on observation
• Epistemology
• Heuristics

Who should we take as role models? What’s the highest we could aim for? I propose the Uomini Universali. Or as Wikipedia has it:

The common term Renaissance man is used to describe a person who is well educated or who excels in a wide variety of subjects or fields.

As software testers you might be exposed to a wide variety of different programs. If you hop from project to project it is very likely that you need to educate yourself in wide varieties of domains. Furthermore, you as a software tester are better off with a questioning and doubting mindset. This is something that can be acquired. Books are a brilliant tool for that. Not the only one of course because practice should never be underestimated. Nor should tacit knowledge, that may only be acquired through socialization processes.

And: do you think that one of the Uomini Universali would have ever been caught uttering the statement: “I haven’t read a single book”?

Ok, my friends. Please leave your comments on your favorite books that helped you become better as software testers. If you convince me that your recommendation is good, I will read the book, too. And we could start a conversation. And then the world would become a better place. And happiness would spread.

Today we finish with a poem by Thomas Moore:

Off I fly, careering far
In chase of Pollys, prettier far
Than any of their namesakes are
—The Polymaths and Polyhistors,
Polyglots and all their sisters.

tags: books, education

I love what I am doing. If I wasn’t I would do something else. Like gardening or follow a career as a beach bum. Sun tan and a lot of booze. But no, it is software testing and managing software testers and talking to developers and ship tested software that I like.

But from time to time I observe a pattern that seems to be especially endemic among software testers. It is the tendency to whine about one’s own work situation while blaming everybody around us for causing it.

Let’s bullet point what some of the testers I have met at conferences not only think but also say aloud (The last one is slightly modified. Instead of the url’ed word, another one was used):

• I am fine, they are not
• I do a pristine job, they are slobs
• I could be so much better if only they did their job
• I would immediately start testing if only they’d provide me with perfect, unambiguous, neatly versioned, accurate, testable, re-usable, superman-feature-like requirements
• I am a saint, they’re rectal cavities (caution: link not safe for work)

Isn’t it interesting to observe how this follows the usual bipolar structures usually found in western cultures? (e.g. BS like: “Either you’re with us or you’re against us”). Never a continuum nor a trinity. Or calm acceptance without assessment.

Now, please! Software Testing is not a terrible job and the circumstances are not bad at all. In order to illustrate go and read Viktor Frankl’s Man’s Search for Meaning.

I’ll wait.

See, that is a horrible situation. For those of you who couldn’t get the book or cannot read here a short abstract: The book describes Viktor Frankl’s devastating time as a prisoner in Auschwitz and how he managed to not fall in total despair and keep up his spirits to survive his personal disaster.

Whenever you find yourself in a situation, just compare to that. I’ll guarantee that you’ll always be fine.

And here’s a belief shattering secret: The world around you was not built to please your own little egoistic demands.

And here’s another: If you are one of those whining people, your are not fun to spend time with. People will avoid you. Then you will be alone. And even more miserable.

Therefore:

• smile every day
• never develop an inferiority complex
• don’t become a zombie tester
• avoid process freaks
• fight sub standard work ethics where ever you find it
• have a serious discussion with testers not willing to read books or learn anything (I think I am going to have a special post on that)

Let’s finish this post in a little bit more upbeat mood. I propose to go watch this hilariously funny short clip on youtube.

tags: testing, testers, whining

Imagine a commune of five long haired hippies. Let’s call them Andy, Bruce, Claire, Denise and Emma. And - hey - it is the time of free love so they live in alternating relationships. No room for conservative thoughts here, the hippies were free thinkers, so all pairs work. Let’s have the following pairs as a start: Anthony & Claire and Bruce & Denise. Poor Emma is not in a relationship and therefore somewhere with with the earth goddess pacha mama dancing barefootly while gently humming a tune.

But, we’re drifting away here.

Now the question: How many possibilities are there for relationships, before our five hippies get bored and become law abiding white protestant middle-class work drones?

You remember the lottery from last time? 45 choose 7? Ok, this here is a simple 5 choose 2. And there are 10 possibilities for relationships. See below.



This leads us to something powerful used in Software Testing: Pairwise Testing

But what is it?
Pairwise testing is a combinatorial testing method where within a given set of parameters with each a finite set of values all value combinations are tested and where you want to keep the total number of tests to a minimum.

When testing exhaustively with the array below the total number of tests would be five to the power of five = 3125
Hm, well, with decent automation this does not sound that unfeasible, does it?



You’re right. But now lets have a look at this. Does not look that much bigger. But actually we now have a total of 10’000’000’000 possible test cases. This is obviously far too many.



It is often stated (and there is some evidence here, see chapter 3 “Empirical Data” ) that errors happening in a program are most often in the combination of two parameters and become less likely when combining three and more. Therefore it would make sense to test covering all pairs.

With each test run you cover several pairs. So for instance with our hippies A, B, C, D, E you would already cover 10 different pairs (as exemplified above).

Assuming that the combination of A1 and H9 leads to a bug, with pairwise testing - wait, now it comes - you win your own special little lottery because you actually find the bug. That’s how you earn respect with the developers. Isn’t that cool?

Information becomes knowledge when applied. Therefore I would recommend to try it out for yourselves. Here are some tools I find useful:

• Hexawise
• James Bach’s ALLPAIRS

Now a word of warning: Some might say: “Let us declare pairwise as best practice, cast it in stone and apply it readily everywhere”. No, not at all. Just because using explosives is excellent for tunnel building (or more abstractly “to make a hole” ) you do not blow up your house every time you need to have a whole for a screw to hang up a picture. Pairwise testing is useless in some contexts. I am not going to explicitly name any, as it should be obvious.

One more thing: I would like to be the advocate of our sapient testers (often still called “manual testers” as if writing the code for automated tests was not done using the hands of the developers). Whenever possible, automate this kind of testing.

What more is there?
Pairwise Testing: A Best Practice That Isn’t
pairwise.org
Pair Wise Testing

And there is even a little conference:
Workshop on Combinatorial Testing

Now go and spread the word. Send out the links. Let the RSS feed rock n’roll. And comment like you haven’t commented ever before. I’m ready to rumble in the jungle.

tags: mathematics, permutations, pair wise testing

What is the equivalent of winning the lottery in the software tester’s context? How do you as a tester win your own special little lottery? In order to answer these questions let’s have a look on what the chances are that you win the “real” lottery. We can do that backwards, starting with the solution:

The chances for you to win the lottery (select 6 out of 45 numbers) is exactly 1 : 8’145’060

But, how do we know?

Ok, we have 45 different numbers or slots:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
For those of you who have not skipped my last post know that for each of the slots we have the following amount of declining possibilities to select numbers:
45 x 44 x 43 x 42 x 41 x 40 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
And because we only have to choose 6, we stop at 40.

This can also be expressed as:
Like: 45 x 44 x 43 x ….. x 3 x 2 x 1 divided by anything in the row below 40 as they are not needed. That is the elegance of math. Isn’t it beautiful?

I know, you have already quickly calculated the result in your head and you must have arrived with the number 5’864’443’200, which, obviously, is not equal to 8’145’060. And you are right. And I am right, too. I have not said I am finished yet. Sneaky me!

We now have the calculation of: “In how many ways can a set of 6 be chosen out of a stack of 45 including different orders of the six cards”. But the lottery does not care if one of the winning numbers is selected as a first number or as a last. Neither does the winner care in his bliss of the millions whether or not he crossed one or the other number first on his tiny receipt.

In how many ways can a set of 6 be grouped? You remember the evil mobbing bunch of memorial picture taking testers from the last post? No? Ok, here is the explanation once more: Six factorial, or 6 x 5 x 4 x 3 x 2 x 1 = 720

That is the number we have to divide the whole thing by, before we hop off to cash the winner’s cheque with the grumpy old lady behind the cashier’s desk.

Or written as a formula:

Or even more elegantly:

Or even more generally -> BTW: If you want to actually say that to - let’s say: to a person who has very dark sun glasses and therefore cannot see the formula - then say: “n choose k” and everybody will immediately believe you are a highly professional software tester.

So = 8’145’060

Which stated as a probability is:

QED

This leads us to the intriguing question: Why on earth am I dwelling on math formulas? That is because after having recommended Khan Academy in the last post, I ate my own dog food. And now I am hooked. I joyfully participate in Khan Academy’s exercises. And man, I wanna get one of those Black Hole Badges. Any would be ok, but “Tesla” is the one I fancy.



I need to add a 3rd part to this series in order to close the loop back to actionable value for software testing. Next time will be about pair-wise testing and why it is not “Best Practise”. Sometimes it is not even “Good Practise”. It may indeed be “Totally Crappy Practise” in some context.

Ah, almost forgot: In the next post I’ll also tell you how to win your own special little lottery (Justin: I’ll address your requests, too). So, stay tuned!

tags: mathematics, permutations, pair wise testing

What’s the relationship between Testers and Mathematics? I don’t know. But I know what it should be. It should be one of affectionate care and love. Let’s have a short experiment: Upon hearing the following two words: Permutations & Probability. Don’t you immediately start to smile? I do.

The subject of permutations came up when I recently had a coaching session with James Bach and we had some discussions on pair wise testing and the selection of test cases.

For today let’s start simple:

If you are a team of five fabulous testers and you are keen to have a picture taken as a memory of the equally fabulous project you just finished. In how many different orders can you place yourselves smiling towards the camera?

The blunt answer without explanation is: 5!

The ones among you who have never liked mathematics may ask: “Why do you shout? Why do you put an exclamation mark after the answer that is not such a huge number after all?”

And I reply: That’s because it is bigger than you think and because the answer is not “a shouted 5”, but 5 factorial, which is 5 x 4 x 3 x 2 x 1 = 120 possible configurations of a picture of a fabulous team of 5 testers.

Why is that so? Well for the picture of 5 testers you have five slots: _ _ _ _ _ . And you have all 5 testers eagerly waiting to get on one of the slots. One of the five takes the first slot (so there are 5 possibilities for the first slot): 5 _ _ _ _ . Then the remaining 4 testers viciously fight for the second slot: 5 x 4 _ _ _. And so on until we have: 5 x 4 x 3 x 2 x 1

That leads us to a meaner example: Let us therefore assume you were not 5 testers on the project but 6. And now it comes: One of the six testers did not contribute to the success of the fabulousness at all. Obviously, you would not want to have that 6th tester on your memory picture. You now have 6 testers of which only 5 will eventually be on the picture: But, how many possibilities?

The answer is: 720

No shouting, no exclamation marks, but an amazing amount of possible constellations for bullying one of the 6 testers out of a memorial picture. The calculation goes as follows: 6 x 5 x 4 x 3 x 2.

That’s all for today. Next time there will be some hairy examples of ordered and not ordered sets and the calculation of the probability of winning the lottery.

Do I hear your voices demanding: “Come on, give us at least one more useful link!”?
Ok, here you go: Khan Academy

tags: mathematics, permutations, pair wise testing

Quite interestingly in the quiz of Part I on observation proficiency, there was a surprising additional answer among the extraordinarily numerous comments - that there was a sleeping young woman on the picture.

By looking at it again I can now actually identify at least 2 sleeping women: a big one on top right of the picture and a small one that is lying on the belly and looking in the direction of the observer. Who knows, there might be even more! Just apply some more pattern matching mechanisms and pair it with a little bit of fantasy - and there you are!

We humans could also be defined as pattern matching machines. In my opinion that is how we survived to eventually become software testers. Let’s pause here and say a sincere “Thank you!” to our ancestors who constantly asked themselves: “Could this shade there be a nasty smilodon salivating in my direction?”

Anyway, read Francisca’s comment to the last post to also see the answer I was initially looking for. (And I say “Hello Francisca” here, because I happen to know her)

Now, in this post let’s have a walk-through with how observation and pattern matching of such an image actually works. To start with, here’s the picture:



There are different things happening in your brain now:

Stage 0: Surprise - Oh! Look at that! A picture with a mess of funny dots!
Stage 1: Reflection - Hm, what could that be?
Stage 2: Analysis - Ok, let’s see, there are some black dots and some shadier areas.
Stage 3: Hypothesis - That sure looks like a UFO up there, doesn’t it?
Stage 4: Disillusion - Oh, no! That’s not it. It is just a big mess of dots here.
Stage 5: Enlightenment - Ah, look at that! It’s a Dalmatian sniffing on the floor

And then you’re done!
And what does all this sound like? Let’s translate to an other domain:

Stage 0: Surprise - Oh! Look at that! A nice piece of software!
Stage 1: Reflection - Hm, what could that be?
Stage 2: Analysis - Ok, let’s see, there are some buttons to press and some UI elements to look at.
Stage 3: Hypothesis - That sure looks like an entry field up there, doesn’t it?
Stage 4: Disillusion - Oh, no! That’s not it. It is just a big mess of dots here.
Stage 5: Enlightenment - Ah, look at that! It’s a bug sniffing on the floor

Yes, I know, it’s obvious. This is Exploratory Testing!

tags: testing, observation, proficiency, exploratory testing

One important element of software testing is applying the right set of methods, approaches and actions in the right context in order to gain data. Another important element then is observation of the data. By observing well it is that you find potential bugs. Therefore - as a self-respecting tester - it is probably a good idea to become a good observer.

I like Wikipedia’s definition of observation:

“Observation is either an activity of a living being, such as a human, consisting of receiving knowledge of the outside world through the senses, or the recording of data using scientific instruments.”

But, how to become a better observer?

Deliberate practice leads to proficiency, and because I enjoy providing a good service - voilà - here’s an opportunity to hone your skill. What do you see on the picture below?



If you feel like it, you may let me know your solution here.

Games have always been the most efficient (and at the same time most joyful) approach to learning. It is by playing games that kids learn about life. Lumosity is an excellent website that I highly recommend if you want to train your observation skills. I like the games Bird Watching, Space Junk or Top Chimp.

tags: testing, observation, proficiency