I have a severe cringe reaction every time I hear someone with “manager” or “director” in their title, speaking about quality. Why? Stop me if you know this one… You’re in one of these long boring company meetings and a “passionate” manager takes the stage to tell the “story about quality”, and you know what you are going to hear, and yet you listen, hoping it’s something new. Yet, the moment comes when all your hopes are shattered, and there’s the familiar refrain “we improved our quality by a huge percent… We increased our coverage and number of tests in that that and that area” and here you are, pulling your hair, banging your head against your desk, crying “No, no, no … numbers of tests and percentages have nothing to do with quality, when are you ever going to learn… Are you guys living in the forest?!”. And it goes on and on and on, few months later you will hear somebody else, repeating the same mantra. So, here’s a good question – what the fuck is wrong with quality and people’s perception of it?
I dare to say there’s nothing wrong with quality, after all that’s what all of us are looking for, is it? Quality of life, quality car, quality relationship, so we must be on to something, how come we are so confused in quality of software, then? Well, I believe quality has dirty little secrets, you will only know them if you’ve spent some time dealing with it.
Dirty little secret #1: Quality has nothing to do with numbers
You will hardly spend a conference on software testing or development without a certain breed of clowns that will cheer from the stage how important it is to improve your quality by reducing manual tests and increasing coverage, improving ratio of whatever or the total number of bugs escaped in production. You know them.
The problem I see with statements such as “improving quality” by increasing coverage above 80% or reducing X number of bugs is that they are totally irrelevant to the concept of quality itself. Quality is not easily measurable characteristic in first place and you in fact will be giving everything you love to get some feedback on the quality from your customers. So, all that talk on how our quality improved after we added X tests to our suite is merely wishful thinking. The fact you act accordingly with it, or pretend quality is related to the number of tests, or number of failed tests or percentage of coverage has nothing to do with it.
What we know about numbers after all is:
- It takes not more than 1 bug to put your reputation as a development company at jeopardy or cause financial harm to you or your clients.
- It takes 1 angry, disappointed client to blow the whistle hard enough and shatter your reputation to pieces.
- It takes 1 missed test that was important enough to lead to the two above.
There isn’t golden ratio or number for testing, or if there was one why isn’t everyone simply doing it and be done with it, once and for all?
And don’t get me wrong – these numbers that we account for, they are not useless – they are information and that’s what we need when making important decisions about the software.
- Increasing number of escaped bugs give you good perspective that you’re missing bugs and your testing is inefficient.
- Increased number of regression bugs means you are having too much coupled functionality that breaks every time someone pulls a string in the code.
- Even percentages of tests and coverage are useful to some degree. The problem with automated tests is – how do you measure how useful they are? You have 1000 tests, so what, how do you know they actually test something? I mean, you can measure their bug catching ratio, but again this tells you nothing since interpolation on historical data isn’t sufficient for making prediction about the future… unless you’re predictably dumb about it.
In other words, these numbers are great, but they are not quality, they are not improving quality they are giving YOU the tools and visibility to act, start a conversation, raise a concern that might improve quality in the long run.
Quality is not measured in tests in the same manner that love is not measured in the number of “I love you” you will tell your partner.
I can’t understand why it’s so hard for people to wraps their heads around that.
Dirty little secret #2: Quality is as subjective as it gets
The largest problem that I see in outsider’s perception of quality is that they believe there is some hard-coded formula for quality that is some function of number of tests and coverage and bugs. When in fact quality is something quite subjective. Jerry Weinberg famously said that quality is value to someone, someone that matters as completed by James Bach. So, speaking of quality we inevitably should discuss the topic of values. Well, you’re not going to hear that a lot on conferences, are you?
Imagine you are a car fan, imagine you believe Mercedes make the greatest cars, but at some point, you meet a person that says BMW does, or any other brand, for the sake of the argument. Why so? After all, the data is there, you can discuss the specific models, horse powers, extras, transmissions, anything and yet, you like one, he likes the other. What’s the reason? Values – there’s simply one company’s values, that are either explicitly or implicitly build in their product that makes it more compelling for you to go with that brand. It’s the same with mobile phones, software, testing frameworks, toothbrushes, and socks, it is the way our human world works.
The game of quality as far as my current understanding goes is – simply pick the right values and make sure you build your product around them and if you walk it like you talk it and work hard it will happen. That inevitably includes listening to your customers carefully, having good ethic and discipline, since quality is not a sprint, but rather a marathon.
Doesn’t work if you’re not authentic, by the way, nor if you try to fool the rest of the world how your magic formula can solve all problems.
Dirty little secret #3: You don’t assure quality
When I joined the tech industry, software testers (or QAs) seemed to believe they are some sort of magical bees that eat software and poop quality. Nearly a decade later, we still do believe it. Some of us are even offended if you call us testers. Even I fell into that trap when I was fresh in the field, the word tester to me was kind of derogatory or disrespectful.
In time and with the help of some of my mentors, I found out quality cannot be tested forcefully into the product. In fact, testing and quality have no obvious correlation.
Yes, for sure we know successful companies test their software, but there’s tons of companies that have successful product, but a terrible process, so terrible you’d be scratching your head wondering – how the hell are you still in business? And let’s not forget even tech giants started as a garage companies with 5 guys coding into a basement, they had no Head of Quality nor “change agents” or “angels” or whatever hipsters doing testing like to call themselves.
Testing doesn’t assure quality, in the same manner that doing a Covid test doesn’t assure the fact you will get better, it will simply provide you with information about your current condition. In other words – testing is the diagnostics part that will give you important information that might influence your decision how to act upon quality.
It has the uneasy, non-trivial and often awkward task to uncover problems and provide that valuable information to people who matter in making important decision. Quality on the other hand is an effort that is far beyond one person’s capabilities – it’s a whole team responsibility, it’s everyone’s input that makes the difference.
So, I would like to frame this very clearly – there is not one action or lack of it that leads to “assuring quality”, if there was again – why aren’t we all doing it, already? After all that time and effort put into testing, if there was an easy way of doing it, someone would have “brute forced” it already.
Dirty little secret #4: You won’t prevent bugs from happening
Along with the all the talk on metrics and how they improve quality there’s a group of people from the agile world that believe they improve quality by “preventing” bugs from happening and therefore reducing the “overhead of unnecessary testing”. Now, that’s an amusing one.
I had this conversation with an engineering manager once that believed you can prevent bugs by writing your tests prior to developers implementing the feature, so they can account for them and this way we can prevent bugs from happening. What this manager was too careless to understand was that the decreased number of caught defects that she was so proud with, was in fact a result of the tester and developer crating an unwritten contract what’s going to be tested and testing only that. As a result, instead of making testing better, this was making it lazier.
Quite honestly there isn’t a way to “cheat” the game by preventing bugs from happening. Yes, there’s actions you can take that help you catch bugs faster, earlier, but that doesn’t mean you can prevent that from happening and even if you do, that doesn’t mean you can cut down on your testing. Even if you are successful, you will have to test what you believe you prevented.
How can we do anything about quality, given our limited reach
I am a huge fan of that Jordan Peterson quote, that you must first take care of your own yard, then try to change the world. So, I believe when we talk about quality important aspect of it and the only part we can in fact “assure” with relevant certainty is the quality of our own work.
How do we do that:
- Make sure you’re the first to come and the last one to leave, when it comes to doing something.
- Take the extra mile in testing, don’t settle with the bare minimum. This includes – question everything you see, break rules, think diverse, challenge yourself.
- Try to train yourself in the policy of zero tolerance to laziness and mediocrity.
- Don’t automate to reduce, reform or replace testing, but to extend it, make it more focused, to enrich it.
- Develop your exploratory and critical mindset, approach everything like a challenge that will make you learn something, instead of as a boring repetitive task.
If you do all the above, you are already doing better than 75% of the industry. So, it’s only upwards from that point.
What can you do to go even better?
- Learn programming – this will help you first be more competent in the world of software and second make you more capable to use tools as your helpers.
- Take the hardest task possible – there’s always a task that everyone would run from – cause it’s boring, too complicated, too unsexy or whatever. Take it – this will prove you’re trustworthy, if you can handle it, it will boost your career and you will learn a lot from dealing with. Even if you fail, the fact you took the courage to volunteer for it will teach you a lot.
- Be ready to act as an advocate of quality – suggest improvements, push your colleagues do to better, be their mentor and guide in doing better more efficient testing. You can do that in many ways – lectures, demonstrations, pair sessions, code reviews and many other.
Testing and quality are two different things. To say something like doing software testing improves quality is beyond absurd. Yes, testing has its part in quality, but it’s far not enough. Our reach as software testers is limited, so we should be very precise and careful how we perform what we can.
So, what is your view of quality? How do you see your role fit in it?
Do you want to tell me how wrong I am? Theere’s the comments 😉
7 thoughts on “Quality’s dirty little secrets”
Good article. Regarding #4, funny how someone thinks that writing tests prior to the functionality being test ready prevents bugs from happening. I mean, yes, that’s possible but this usually occurs during static reviews of the user story or whatever. Doing this on a regular basis will increase the devs and testers productivity in finding potential problems but usually it concerns the whole team not just the dev working on the functionality and the tester who will later run the tests against it.
What “unnecessary testing” is, I don’t even know what means in this context. As we know, testing never “ends” we just “stop” testing because of constraints, we are all aware of.
Thanks for the kind words! 🙂
I enjoyed the article – I also agree that testing does not equate to quality. The one comment I have is about #4. I am one of those folks who talk about preventing bugs by talking about testing early, and what I mean when I say that, is preventing misunderstandings, and highlighting assumptions that are made. I have found so many bugs in code that could have been prevented if questions had been asked earlier, and clarifications made. I never mean you do not have to do exploratory testing, or any other kind of testing after code is written. That said, I do know that many ideas are taken out of context and people take their own meaning.
Hello, Janet and thanks for spending the time to read my blog and comment on it.
I agree with you testing is necessary at the earliest possible stage, that might be even idea storming.
“I have found so many bugs in code that could have been prevented if questions had been asked earlier, and clarifications made.”
That sounds a bit like hindsight bias – yes, when things hit the fan it always looks like there have been an easier way preventing them, it never works the other way around. 😉
The bottom line here being – testing early doesn’t mean less testing later, it just means more persistent testing
Very good article!
I agree that software testing generally doesn’t improve quality but I have done so according to my experience of testing different kind of systems in different companies: Almost everything I have found and reported have been fixed and that has improved quality! But there’s no guarantee that it will continue to do so: I’m not sure I will find anything more or if things will be fixed on what I report BUT it is very likely as I am continuously testing systems ways to find problems BUT there’s no guarantee it will be fixed But according to how I report things, it will most certainly be fixed BUT you can never be sure and that’s the beauty of it!
Hello, Chris! Sure, that’s the beauty of it.
As stated above – there’s ton of companies that do zillions of tests and have the most complicated testing process and still fail. On the other hand you have companies that barely test and still succeed, it’s a mistery. We know for sure testing helps us identify problems and problems are bad for success. Quality on the other hand something waay much more complicated.