Beating robots down with your bare hands: Quality

Reading Time: 8 minutes

I’ve recently had a conversation with Maaret Pyhäjärvi related to a topic I’ve submitted to Eurotest Conf, it was about AI and humans, and quality of testing and all that hype around us loosing our jobs to machines. Well, the talk didn’t quite make it to the conference schedule, but it was great I had the opportunity to speak with Maaret at first and talking with her actually gave me some interesting ideas to think of. So, thanks for that, Maaret! 🙂

The bottom line of our talk was – we often see people talking how powerful and omnipotent automation is. I see it starts to transfer to AI as well, I mean I still haven’t heard of AI powerful enough to replace a human being and yet, there’s “prophecies” of the end of the days for human workforce.

Also, we see other testers talking how imperfect automation is and how we can not replace human testing with machine one, and yet we fail so miserably (me included) to demonstrate and prove with examples why is this so, or at least we are too lazy to do it.

So, an idea came to my mind for post(s) named “Beating down robots with your bare hands” which will try to explain what sort of knowledge, skills, tactics methods we have on the human side of testing that make us relevant. Yes, human testing, not “manual testing”, stop using “manual testing”, ffs 🙂

bostom dynamics robot wtf

So, the first thing that I will focus at is esential for software testing. Not only, it’s something that we do waaay better than robots and automation. In fact we are the only ones to do it, at all …

Beating robots down with … Quality

So let’s start with something easy – we work on these everyday, we have this in our “title” – Quality assurance, yet we don’t normally assure it, but, at least, we argue a lot about it.

What’s quality?

Not so long ago I read an interview with David Greenlees by Rich Rogers – in the last section the guest (David) has to ask the next guest a question, so David asked the following:
“Can you measure quality, and if so, how? Oh, counting bugs doesn’t count (pun intended)!”

A beautiful question, I loved it, so many great ideas can come out of it. When a couple of seconds later I was like… hold on a minute, how do you measure quality, what is quality in first place?

So, what I would do is, just like any average human being, I would go to google, ask it, then come here and write it in my blog, just to look smart ass :D. So, I open the first result and it said:

In manufacturing, a measure of excellence or a state of being free from defects, deficiencies and significant variations. It is brought about by strict and consistent commitment to certain standards that achieve uniformity of a product in order to satisfy specific customer or user requirements. ISO 8402-1986 standard defines quality as “the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs.” If an automobile company finds a defect in one of their cars and makes a product recall, customer reliability and therefore production will decrease because trust will be lost in the car’s quality.

Read more:

Well, sh*t. There’s a lot of complicated terms here, so let’s try to get deeper and analyze some of these and try to find out whether they apply to software testing:

  • free from defects – I guess we all agree that there’s no state “free from defects” and if anyone ever tells you he will produce a “defect free” software, you should be seriously concerned about that person’s credibility and honesty. The best we can aim for is – “free of critical issues that  we anticipated” or “free of issues that were detected”.
  • strict and consistent commitment to certain standards – Well, that doesn’t quite work in software manufacturing, in fact there is regulated areas where testing is performed in compliance with standards and laws and yet, this only is not enough to produce a high quality product.
  • specific customer or user requirements – That might seem pretty straight forward, but in fact it’s the hell on Earth. Every time you start working on software project you’ll be given (if you are lucky enough) “strict and specific customer or user requirements” which will probably change on daily basis, due to:
    • dynamic business niche changes
    • ongoing changes in our clients infrastructure (re-branding, restructuring, budget cuts, etc.)
    • emerging new trends
    • or simply clients not being able to articulate exactly what do they want from the product, which is absolutely normal, they are not software professionals, if I was not in the software craft and I ask to have a software product developed for me and somebody asks me what are my quality criteria, I’d probably answer something like: “I don’t know, I guess it should work and not be terrible pain in the ass.”
See also  Hindsight lessons about API testing

What does all of this has to tell us? That dealing with quality is hard. Providing meaningful information about it is even harder.

Can we measure quality?

Short answer – No!

Long answer – No, if we want to be honest with ourselves, our clients, our customers and our management. No, if we want to derive from practice and not from abstract, unrelated to our craft theory, made up by someone who wants to sell “solutions” for pennies.

How are we being fooled we can measure quality?

There’s tons of literature and blog posts trying to make you believe how you can measure therefore automate certain aspects of testing and therefore measure quality. Here are the most common ones, along with my comment how they aren’t actually being helpful in providing quality related info:

  • Measuring pass/fail/left test cases to run.
    This is one of my favorite, I still can’t wrap my head around the concept that there are people actually trusting count of test cases matters in any way. I mean, let’s put it this way, if I were a bodybuilder and I wanted to track my progress, the number of times that I successfully/unsuccessfully entered the gym, would be the less relevant thing to my progress, this is how test cases counts are relevant to quality.
    Whenever people are counting test cases they are missing one important thing – the quality of the tests. Not all tests that you perform are equal as execution time, configuration time, knowledge involved, prerequisites, test data involved and many other variables and we are not even getting to the part of are we testing the right thing, is my testing meaningful and providing meaningful information.
    The moment we start to count test cases, we only drive testers to start twisting these metrics the way they want, by producing huge amounts of low quality tests, because the only thing that matters are the numbers.
    I am strong enemy of the blind trust with which many people seem to approach counted tests as “a priori” useful tests, rather than review each test and make sure it’s actually making sense in the current context.
  • Measuring bugs
    This is another metric which is yet relevant to the project and not directly related to the product quality. Counting bugs might provide us with a lot of useful information when we are talking about the process and the quality of testing that we perform, but it has few major flaws:

    • We can only count bugs that we already found.
    • Not all of our bugs have the same impact.
    • Not all bugs involve the same amount of effort to find them or complexity to reproduce them.
    • By no means count of bugs found is relevant to total count of bugs, nor to critical bugs found in the product.
See also  Some kick ass blog posts from last week #6

Machines know shit about quality.

And that’s true, they are totally unaware about it and, in my humble opinion, currently, with our current state of knowledge and intelligence and technology we can not translate quality concepts into machine language, automate it, learn an AI to do it, or train a magical robot that would do it.

We can use machines and scripts and AI and robots and whatever technologies are available to us to enhance our testing and gathering of different information relevant directly or indirectly to what we perform, but it is a human decision at the end if the product reached certain quality criteria or not, therefore it’s up to humans to decide and give the final verdict.

What is quality after all?

But why is this so, why is it such a problem to translate quality to machine language?

I think in order to answer the question we need to revisit other definitions about quality that we have:

Quality is value to some person.
Jerry Weinberg

and its modernised version:

“Quality is value to some person who matters“
James Bach

Do you see a pattern here? “Person” – quality is value to a person, therefore quality is strictly subjective to a single person and varies with all the important individuals that we have in our project. This means what I find being of high quality might totally suck according to a colleague of mine, or according to the CEO or the client or his marketing or sales representative.

That’s why quality is totally untranslatable to machine language – it is purely human concept and as almost everything human it is not definite, it varies and it varies a lot.

What we can do about quality better than machines?

What I can offer as advice how to generate quality related knowledge is – learn. Learn to learn and learn to improve you learning. Software testing is all about learning, improving and all of these skills will be vital in gathering quality related information about the product.

So, what to focus on?

Learn different types of quality.

Many of the metrics described above were actually related to project quality and not to product quality – number of found bugs that are invalid, number of bugs with low priority, number of bugs with “can’t reproduce state”, etc. All of these provide information of “how are we developing and testing the thing”, but not directly to the thing that we want to deliver to our clients.

So it is important to make the distinction:

Product quality – Quality of the deliverable that we wish to provide to our clients with all its assets, help docs, user guides, support services, future updates, etc.

See also  Hindsight lessons about automation: The mastery of data seeding

Project quality – the efficiency related to all of the actions that we perform during the development process, practicesmethodology, time related tasks, repetitive activities, wasteful activities, facilitating activities, etc.

Both of these are different things and hold different types of risks. Which brings me to my next point.

Learn about risks.

In my lessons in software testing to novice testers I spent a serious amount of time teaching people how important it is to be aware of the risks related to our product. How important it is to make sure we understand not only the technological risks that we normally deal with, but also the business risks.

It is an important thing to show to the people we work with, that we are not simply coders/testers trying to deal with their daily tasks, but we are involved and interested in their business and we want to help it succeed, because their success means a job well done for us, as well.

Also, knowing risks is a good way to figure out our quality criteria. How are we supposed to know what’s quality if we are unaware of what has the potential to harm it?

Learn what other testers have to say about quality.

As I concluded, quality is subjective, but different ways to explore and search information about quality can be very useful. That’s why talking to other testers about quality, learning what is their strategy of uncovering information might be a powerful tool to develop your approach.

I already mentioned two definitions about quality, but if you are interested I would definitely recommend taking a look at Heuristic Test Strategy Model by Bach and Bolton, especially the “quality criteria categories” section of the document.

Recently, I’ve also read a nice article by Rich Rogers with some nice visuals related to quality, which might also bring some  light over that complex and profound topic.

But most important…

Talk to your team and clients about quality.

Software testing is not simply about code and test cases and documentation, as I stated before theres’ a huge amount of social knowledge that we have to obtain in order to produce high quality software.

This means we need to constantly discuss, ask, sometimes defend different positions related to quality, but above all, it’s of primary importance that we explore what our clients and colleagues have to say about quality and try to find the cross section of all their opinions that will be relevant enough to our context, deadlines and budget so we could focus on producing it and implementing it.


So, I believe that’s it. It’s a long and complex topic and I definitely missed something, don’t be shy and tell me about it. I would love to read your comments and thoughts on the topic.

Any shares and retweets are highly appreciated. Thanks for reading. 😉

Note: This topic was also somehow inspired by Michael Bolton’s Things Could Get Worse: Ideas About Regression Testing – EuroSTAR – Michael Bolton, if haven’t seen it, go do it now! 

Please follow and like us:


Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.

More Posts - Website

Follow Me:

5 thoughts on “Beating robots down with your bare hands: Quality”

  1. Fascinating article. I disagree with the statement that robots cannot replace testers. Robots do replace testers with test execution. I’m currently working on an initiative to let robots decide which tests to execute, as the decision should be influenced by so many parameters, that a machine can analyze better than any human being.
    I believe in the future there will be other aspects of testing replaced by “testbots”, such as test planning, bug analyze and more.
    Testing software is complex, but not more than driving a car…

    1. Hello, Ayal!
      Thanks for the comment!
      Let me question some of the statements that you make.
      1. “Robots do replace testers with test execution.”
      [Viktor’s comment]
      Great, that’s true, but yet, test execution is not all testing to be performed, yet in all following statements you extrapolate from the statement as “test execution” means “testing” and it doesn’t.
      2. “I’m currently working on an initiative to let robots decide which tests to execute, as the decision should be influenced by so many parameters, that a machine can analyze better than any human being.”
      [Viktor’s comment] Great is this project open-source, cause I would like to know more about it? Is there any open documentation for it?
      Another question – how do you measure “that a machine can analyze better than any human being” did you make a test with any human being? There’s plenty of situations where human can make a decision based on tons of sensory input, without even thinking about it. Driving is just one example of that. If you are interested take a look at Daniel Kahneman’s book “Thinking fast and slow”.
      3. “Testing software is complex, but not more than driving a car”.
      [Viktor’s comment] I am again interested how do you measure “complexity” of both and based on what qualities do you compare them?
      To me this doesn’t sound like a statement supported by some concrete evidence, but more of a personal opinion.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.