Reading Time: 6 minutes

In my quest for testing knowledge in which hindsight lessons were born, I am trying to dig deeper in the thinking process behind testing. It’s not an easy job, I’d say, and for sure not so shiny and popular as showing automation how tos with various tools, but I love it and I think it’s what our testing craft is lacking a lot.

One thing that is missing from the landscape of testing is the explanation about heuristics and how the heuristic problem-solving approach guides our journey as testers. There’s fair number of testers and consultants that speak of heuristics or use heuristics in their materials, but based on my experience and subjective evaluation, few of them perform a good job explaining what it is. I believe this an integral part of testing and testing mindset and I even included it in lectures about exploratory testing in Pragmatic.

In the end of this article I will try to gather up some interesting and useful links about heuristics.

What is a heuristic?

I am not a huge fan of providing definitions, but I realize heuristic is a scary word. I also believe that firm knowledge of the basics is sometimes better than “some” advanced knowledge about something.

So, here’s a formal definition for heuristic:

any approach to problem solving or self-discovery that employs a practical method, not guaranteed to be optimal, perfect, logical, or rational, but instead sufficient for reaching an immediate goal

Heuristic has the same root as in Greek word Eureka, which means “to discover”, so heuristics are, shortly put – means to discover information by employing practical approach that’s fallible, please take a note on that last part, as we will go back to it couple of times.

Where can we observe heuristics?

Although you may be hear this word rarely, or not so rarely if you monitor the context driven testing or Rapid software testing communities, but if you read this word for a first time, you might be surprised that heuristics are may be the most common human way of discovering.

Imagine one of these logical puzzles, for example a Rubik’s cube.

You can’t solve it by simply following a rule, I don’t know, may be people who solve Rubik’s cube for world records just memorize all the possible positions and the movements to arrange it, not sure. If you solve it for the first time, the best bet you have to solve it is trying a directed series of fast experiments that aim to explore how it works, but also provide fast feedback whether or not you are on the right track. So, we call these fast experiments heuristics.

As I said, heuristics are something very common, although we might not be aware of that or we might not be aware we are calling them heuristics. In fact, they are something used in everyday life, all the time. I bet the readers of this blog are using heuristic approach in determining if this post is worth reading or not – because they will read part of it, trying to figure out if it makes sense or if it will be something they can use in practice.

There’s also plenty of heuristics used in everyday life, here are few examples.

• Rule of thumb – commonly used to represent a technique, advice or piece of wisdom, acquired by practical knowledge.
• Stereotyping – this is very common heuristic we use and sometimes to make wrong decisions, for example – many people use stereotyping saying things like – low price products are low quality, high price products are high quality.
• Educated guess – deciding based on the knowledge we acquired, previously.
• Draw a model – also used in software development – when we want to describe something or understand better how it works, we often use to draw a model or a flow chart of it, in order to layout what’s complex and try to understand it.
• Trial and error – commonly used in problem solving, when no viable solution is available, we have no other way, but try and learn from our mistakes. Also, commonly used in testing.

 If you are interested in short intro in heuristics, you can look at the video here:

What is essential for heuristics?

Normally, heuristics are alternative to using a strict rule or an algorithm. Here’s an interesting observation that Michael Bolton made in his post Heuristics for Understanding Heuristics:

“When you know how to solve a problem, you might follow a rule. When you’re not so sure about how to solve the problem, following a rule won’t help you. Not knowing how to solve a problem means not knowing which rule to apply, or whether there’s a rule at all. When you’re in uncertain conditions, or dealing with imperfect or incomplete information, you apply heuristics—methods that might work, or that might fail.”

I’d like to put the focus on that last part – the fact that heuristic might fail, and it is actually a pretty good chance that it will fail. It is something that is often omitted about heuristics, but is very crucial about them. This is a trade-off that we are ready to make, because we hope at best to receive relevant information, related to our problem fast.

And here’s another difference with rules – when applying rules, if they fail, this means we screwed up bad, it’s either that rule is wrong, or we interpreted the problem wrong and applied the wrong rule.

With heuristics, it seems like a win-win situation – they fail fast, if they succeed – they provide us with vital information for solving the problem, but if they fail – even better they provide us with vital information for our approach and its efficiency.

Another essential key aspect of heuristics is that there’s hardly one single heuristic useful to solve a problem, but the real power of heuristics come from combining them to multiply their effect.

How heuristics are related to testing?

I give lecture to my students in heuristics and I believe they are essential for testing, because the approach we use while testing seems to me to be purely heuristically formed.

We might pretend we follow rules, defining “practices” that work always, but we’ve seen enough of these to fail when they get out of “fairytale land” and hit the horror of practice.

The reason why I see the problem solving in testing to be based on heuristics is mainly that the problems we solve, although framed always in the same manner, are always diverse.

Normally we face challenges like:

• Perform consistent testing programmatically (automation, they call it)
• Find as many valuable bugs, for a short time (but what bugs are valuable, and how “short” that time is)
• Find a show stopper
• Reproduce that weird behavior that’s causing our client so much pain, etc.

All of these are problems that everyone solved or is solving, yet the best anyone can provide as a solution is their own personal, subjective experience  of it.  We can’t apply a rule, because the business case is so different from company to company, or domain to domain or even two different teams within the same company. It can also vary enough between versions of the same product.

Heuristics in testing

Luckily, I am not the first to think that heuristics are grounded in testing and that’s a good thing.

If you are interested in finding out more about heuristics and their application in testing, you can start with:

SFDIPOT – heuristic for creation of meaningful and in dept coverage outline of your product by Bach and Bolton

Heuristic cheat sheet – document of testing heuristics for commonly used functionalities, created by Elisabeth Hendrickson, James Lyndsay, and Dale Emery

RCRCRC – heuristic for regression testing by Karen Johnson

Also, if you are interested in taking a deep dive in testing heuristics, I’d recommend reading “Explore it: Reduce risk and increase confidence with exploratory testing” by Elisabeth Hendrickson, not only it has a collection of great heuristics, but also one of the most in-depth books for exploratory testing I’ve read, so far.

“Best” practices – distorted heuristics

I have already addressed the “best” practices fallacy in a talk, called ““Worst” practices of software testing” I gave at Ministry of testing Testbash Munich. By the way, it was selected as one of the most watched videos from Testbashes, so now I am going to brag about it on every possible occasion, so there. 😀

What we must clear out is – above all, “best” practices are heuristics – they are what we earlier referred to as “rule of thumb” – a practical advice, often obtained by a person or a collective and is empirically justified. So far, so good, the reason why I claim they are distorted heuristics is that “best” part. Calling a practice “best” implies that it is either always working or proven to be more efficient than any other. What this wording omits, in fact, is the fallible nature of heuristics – every heuristic is fallible, “best” practices included.

Having in mind the complexity of the testing domain, diversity of contexts, all the different business cases, there’s pretty good chance that any “best” practice will fail in a specific combination of the above. The best you can hope for is that a practice works best, for you, in your specific case and nothing more. Do not try to extrapolate overall efficiency, based on limited amount of representative data.

Advice, instead of conclusion

I believe the topic of heuristics is an interesting one and learning about them can be a useful refresher of our testing approach.

My personal advice is – do not try to learn heuristics or use them as rules. If you follow a heuristic strictly, you are turning it into a rule, try instead to build your own heuristics, put your creative thinking into the process – that’s where their true power is maximized.

If you have any other ideas or thoughts on heuristics, I’d kindly read and discuss them with you. Thanks for reading!