Quality’s dirty little secrets

Reading Time: 8 minutes

I have a severe cringe reaction every time I hear someone with “manager” or “director” in their title, speaking about quality. Why? Stop me if you know this one… You’re in one of these long boring company meetings and a “passionate” manager takes the stage to tell the “story about quality”, and you know what you are going to hear, and yet you listen, hoping it’s something new. Yet, the moment comes when all your hopes are shattered, and there’s the familiar refrain “we improved our quality by a huge percent… We increased our coverage and number of tests in that that and that area” and here you are, pulling your hair, banging your head against your desk, crying “No, no, no … numbers of tests and percentages have nothing to do with quality, when are you ever going to learn… Are you guys living in the forest?!”. And it goes on and on and on, few months later you will hear somebody else, repeating the same mantra. So, here’s a good question – what the fuck is wrong with quality and people’s perception of it?

I dare to say there’s nothing wrong with quality, after all that’s what all of us are looking for, is it? Quality of life, quality car, quality relationship, so we must be on to something, how come we are so confused in quality of software, then? Well, I believe quality has dirty little secrets, you will only know them if you’ve spent some time dealing with it.

Dirty little secret #1: Quality has nothing to do with numbers

You will hardly spend a conference on software testing or development without a certain breed of clowns that will cheer from the stage how important it is to improve your quality by reducing manual tests and increasing coverage, improving ratio of whatever or the total number of bugs escaped in production. You know them.

The problem I see with statements such as “improving quality” by increasing coverage above 80% or reducing X number of bugs is that they are totally irrelevant to the concept of quality itself. Quality is not easily measurable characteristic in first place and you in fact will be giving everything you love to get some feedback on the quality from your customers. So, all that talk on how our quality improved after we added X tests to our suite is merely wishful thinking. The fact you act accordingly with it, or pretend quality is related to the number of tests, or number of failed tests or percentage of coverage has nothing to do with it.

What we know about numbers after all is:

• It takes not more than 1 bug to put your reputation as a development company at jeopardy or cause financial harm to you or your clients.
• It takes 1 angry, disappointed client to blow the whistle hard enough and shatter your reputation to pieces.
• It takes 1 missed test that was important enough to lead to the two above.

There isn’t golden ratio or number for testing, or if there was one why isn’t everyone simply doing it and be done with it, once and for all?

And don’t get me wrong – these numbers that we account for, they are not useless – they are information and that’s what  we need when making important decisions about the software.

• Increasing number of escaped bugs give you good perspective that you’re missing bugs and your testing is inefficient.
• Increased number of regression bugs means you are having too much coupled functionality that breaks every time someone pulls a string in the code.
• Even percentages of tests and coverage are useful to some degree. The problem with automated tests is – how do you measure how useful they are? You have 1000 tests, so what, how do you know they actually test something? I mean, you can measure their bug catching ratio, but again this tells you nothing since interpolation on historical data isn’t sufficient for making prediction about the future… unless you’re predictably dumb about it.

In other words, these numbers are great, but they are not quality, they are not improving quality they are giving YOU the tools and visibility to act, start a conversation, raise a concern that might improve quality in the long run.

Quality is not measured in tests in the same manner that love is not measured in the number of “I love you” you will tell your partner.

I can’t understand why it’s so hard for people to wraps their heads around that.

Dirty little secret #2: Quality is as subjective as it gets

The largest problem that I see in outsider’s perception of quality is that they believe there is some hard-coded formula for quality that is some function of number of tests and coverage and bugs. When in fact quality is something quite subjective. Jerry Weinberg famously said that quality is value to someone, someone that matters as completed by James Bach. So, speaking of quality we inevitably should discuss the topic of values. Well, you’re not going to hear that a lot on conferences, are you?

Imagine you are a car fan, imagine you believe Mercedes make the greatest cars, but at some point, you meet a person that says BMW does, or any other brand, for the sake of the argument. Why so? After all, the data is there, you can discuss the specific models, horse powers, extras, transmissions, anything and yet, you like one, he likes the other. What’s the reason? Values – there’s simply one company’s values, that are either explicitly or implicitly build in their product that makes it more compelling for you to go with that brand. It’s the same with mobile phones, software, testing frameworks, toothbrushes, and socks, it is the way our human world works.

The game of quality as far as my current understanding goes is – simply pick the right values and make sure you build your product around them and if you walk it like you talk it and work hard it will happen. That inevitably includes listening to your customers carefully, having good ethic and discipline, since quality is not a sprint, but rather a marathon.

Doesn’t work if you’re not authentic, by the way, nor if you try to fool the rest of the world how your magic formula can solve all problems.

Dirty little secret #3: You don’t assure quality

When I joined the tech industry, software testers (or QAs) seemed to believe they are some sort of magical bees that eat software and poop quality. Nearly a decade later, we still do believe it. Some of us are even offended if you call us testers. Even I fell into that trap when I was fresh in the field, the word tester to me was kind of derogatory or disrespectful.

In time and with the help of some of my mentors, I found out quality cannot be tested forcefully into the product. In fact, testing and quality have no obvious correlation.
Yes, for sure we know successful companies test their software, but there’s tons of companies that have successful product, but a terrible process, so terrible you’d be scratching your head wondering – how the hell are you still in business? And let’s not forget even tech giants started as a garage companies with 5 guys coding into a basement, they had no Head of Quality nor “change agents” or “angels” or whatever hipsters doing testing like to call themselves.
Testing doesn’t assure quality, in the same manner that doing a Covid test doesn’t assure the fact you will get better, it will simply provide you with information about your current condition. In other words – testing is the diagnostics part that will give you important information that might influence your decision how to act upon quality.

It has the uneasy, non-trivial and often awkward task to uncover problems and provide that valuable information to people who matter in making important decision. Quality on the other hand is an effort that is far beyond one person’s capabilities – it’s a whole team responsibility, it’s everyone’s input that makes the difference.

So, I would like to frame this very clearly – there is not one action or lack of it that leads to “assuring quality”, if there was again – why aren’t we all doing it, already? After all that time and effort put into testing, if there was an easy way of doing it, someone would have “brute forced” it already.

Dirty little secret #4: You won’t prevent bugs from happening

Along with the all the talk on metrics and how they improve quality there’s a group of people from the agile world that believe they improve quality by “preventing” bugs from happening and therefore reducing the “overhead of unnecessary testing”. Now, that’s an amusing one.

I had this conversation with an engineering manager once that believed you can prevent bugs by writing your tests prior to developers implementing the feature, so they can account for them and this way we can prevent bugs from happening. What this manager was too careless to understand was that the decreased number of caught defects that she was so proud with, was in fact a result of the tester and developer crating an unwritten contract what’s going to be tested and testing only that. As a result, instead of making testing better, this was making it lazier.

Quite honestly there isn’t a way to “cheat” the game by preventing bugs from happening. Yes, there’s actions you can take that help you catch bugs faster, earlier, but that doesn’t mean you can prevent that from happening and even if you do, that doesn’t mean you can cut down on your testing. Even if you are successful, you will have to test what you believe you prevented.

How can we do anything about quality, given our limited reach

I am a huge fan of that Jordan Peterson quote, that you must first take care of your own yard, then try to change the world. So, I believe when we talk about quality important aspect of it and the only part we can in fact “assure” with relevant certainty is the quality of our own work.

How do we do that:

• Make sure you’re the first to come and the last one to leave, when it comes to doing something.
• Take the extra mile in testing, don’t settle with the bare minimum. This includes – question everything you see, break rules, think diverse, challenge yourself.
• Try to train yourself in the policy of zero tolerance to laziness and mediocrity.
• Don’t automate to reduce, reform or replace testing, but to extend it, make it more focused, to enrich it.
• Develop your exploratory and critical mindset, approach everything like a challenge that will make you learn something, instead of as a boring repetitive task.

If you do all the above, you are already doing better than 75% of the industry. So, it’s only upwards from that point.

What can you do to go even better?

• Learn programming – this will help you first be more competent in the world of software and second make you more capable to use tools as your helpers.
• Take the hardest task possible – there’s always a task that everyone would run from – cause it’s boring, too complicated, too unsexy or whatever. Take it – this will prove you’re trustworthy, if you can handle it, it will boost your career and you will learn a lot from dealing with. Even if you fail, the fact you took the courage to volunteer for it will teach you a lot.
• Be ready to act as an advocate of quality – suggest improvements, push your colleagues do to better, be their mentor and guide in doing better more efficient testing. You can do that in many ways – lectures, demonstrations, pair sessions, code reviews and many other.

Conclusion

Testing and quality are two different things. To say something like doing software testing improves quality is beyond absurd. Yes, testing has its part in quality, but it’s far not enough. Our reach as software testers is limited, so we should be very precise and careful how we perform what we can.

So, what is your view of quality? How do you see your role fit in it?
Do you want to tell me how wrong I am? Theere’s the comments 😉