Test automation – the bitter truth

Reading Time: 7 minutes

Recently, I came across this awesome blog post by Mike Talks where he tells the story of his personal experience with automation and all the false expectations that he and his team had about it, and it is really an awesome post, if you hadn’t read it, go do it now.

And I really got inspired by it, I recently write a little post concerning automation and I figured out I had much more to say. I was really busy doing the “Software testing is not….” series, so I didn’t really feel like going off topic. But now I feel the time came to tell each other some bitter truth about “test automation”. It is bitter, because people don’t like it, they don’t like talking about it, they don’t like promoting automation like this, but, yet it’s the truth.

It all starts with love.

I have to say I love automation, I really do. And the reason why I write this post is, because I love testing and automation and I want to give the best perspective how they can supplement each other. Yes, it is related to saying some uncomfortable truths about it, but it has to be done.

So, I said I love test automation, not because I believe it is the one and only solution to all my testing problems, yet I am sure when I was new I had that delusion about it, probably. The reason I like it, is because I like to code. I like writing cryptic “spells” and see them turn into magic and I really don’t care if it’s a source file, an interpreter or a console, I just love it.

I see a lot of people being excited about automation, but for a different reason – because they see it as the ultimate solution to all testing problems, the silver bullet, the philosopher’s stone or you name it, it seems to them that human testing might be  replaced with automated scripts and everything will be great, we will ship the product in no time and the world will be a better place.

I have to object this. And trust me, this is not a hate post against automation, in fact it is post aiming to justify why automation is useful, but useful in the right way, with the right expectations, not the way that we want it to be. So, in order to achieve this, I will have to write for some bitter truths, about automation.

Bitter truth # 1: It’s not test automation, it is tool assisted testing.

James Bach and Michael Bolton did an awesome job, making the difference between “testing” and “checking” , so I think it will be useless for me to repeat what they already said. If you are  not familiar with their work, I strongly recommend you do check “Testing and checking refined” and their white paper on automation in testing – A Context-Driven Approach to Automation in Testing.

In a nutshell, the testing that we perform as humans is not easy to translate in machine language, we can not expect from a machine to act like a human. So, they make the difference between “checking” – shallow form of testing that could be translated into set of instructions and  expected conditions and “testing” – describing the process of human testing with all its complexity and depth. Therefore, the term “test automation” carries a false meaning – we can not automate testing, we can automate checking and it is more correct to say “automated checking”. Yet, I prefer another  term they used, for the reason it sounds more elegant and provides more information – “tool assisted testing”. Because that’s what we do, when we automate something – we use tools to assist our testing, that’s it, we don’t want to replace it, or get rid of it. We want to use tools to enable us, to do more, rather than to replace the human effort in testing.

See also  The state of testing 2013 - survey

Bitter truth # 2: Automation doesn’t decrease cost of testing.

I am really interested how does that equation work?. People claim automation reduce cost of testing. Let’s see.

  • You hire an additional person to deal with building the framework and write the tests. (If you think your existing testers can do it part-time, while doing their day-to-day tasks, my answer is … yeah, right ) so that’s additional cost
  • You probably will spend additional money for the tool licence, or otherwise you will use open source tools which means that guy will have to spend additional man hours in order to make that framework work for your specific testing needs.
  • You will have to pay that guy to write the actual tests, not only the framework, so that’s additional cost.
  • The code that you write isn’t “pixie dust”, it’s not perfect, it turns into one additional code base that has to be taken care of, just as every other production code – maintenance, refactoring, debugging, adding new “features” to it, keep it updated. Guess what, it all will cost you money.
  • And of course, let’s not forget about all the moments of “oh” and “ah” and “what was the guy writing this fucking thinking” that you will run into, related to the framework that you use and its own little bugs and specifics, that will also cost you additional money and time investment.

I think that’s enough. The main reason that people give for the “low-cost” of automation is – it pays back in long-term. Now that’s awesome. And it would be true, if we accept that once written a check is a constant that never changes, that it works perfectly and our application never changes until the end of the world. Well, that would be nice, but there’s two problems:

  1. We all know from our testing experience that things written in code don’t actually work perfect, out of the box. In fact it takes a lot of effort to make them work.
  2. If your code base never changes, if it stays the same, you don’t add anything, you don’t change anything, you don’t redesign anything, it is pretty possible that you are going out of business. We are working in a fast evolving environment and we need to be flexible to changes.
See also  Some kick ass blog posts from last week #9

So, again. I don’t get it, how having all of the above leads to cost reduction.

Bitter truth # 3: Testing code isn’t perfect.

I see that very often at conferences and in talks among colleagues. It seems that test automation scripts are some kind of miracle of the nature that never breaks, never has bugs, once you write it and it’s a fairy tale. Well, unfortunately it is not.

As stated above already, it is code, normal code just like any other, it has “its needs” – to be maintained, to be refactored and guess what … to be tested :O Why is nobody talking about testing the automated checks, I mean are these guys testers, did they left their brains in the fridge or something?! It’s code, it has bugs, it has to be tested, so we make sure it checks what we want it to check, wake up! Not only that, I bet you, that doing your automation, you will be dealing with bugs introduced by your checks, much more than bugs introduced by the code of the app that you test.

Bitter truth # 4: Automation in testing is more reliable… it’s also dumb as fuck

One of the many reasons why testers praise automation is – it is more reliable than human testing. It can insure us that every time we execute exactly the same check, with exactly the same start conditions, which is good, at least it sounds good. The truth is, it’s simply dumb and I mean machine-like dumb. It’s an instruction, it simply does what it is told, nothing more, nothing less. Yes, it does make the same thing every time over and over, until you don’t want it do that thing any more. Then it turns into huge pain in the ass, because someone will have to spend the time to go and update the check and make sure it checks the new condition correctly. In other words, automated checks are not very flexible, when it comes to changes and changes are something we see pretty often in our  industry.

Bitter truth # 5: Automated checks don’t eliminate human error

For reasons that I stated above I claim that automated checks can not eliminate human errors. Yes, they eliminate natural variety that human actions have, but that doesn’t mean we don’t introduce many more different new human errors while writing the code. The conclusion is simple, while the code is produced by a human being, it is possible it has errors in it, it’s just life.

See also  Software testing is not... finding bugs part 4.

Not only that, but having our checks automated, we introduce the possibility for machine error. Yes, our code has all the small bugs and weird behaviors that Java, C#, Python, Php or any other language have. The framework that we use also might have these, the interaction that it has with the infrastructure it runs on, might also introduce errors. So, we must be aware of that.

Bitter truth # 6: Automated checks are not human testing automatically

I see and hear that pretty often, everyone saying – “we know we can’t automate everything” and yet they continue to talk as if they could. Not only that, they talk like if they could mimic exactly the same process that a human is doing. And it is not possible. There is no human activity that is cognitively driven, that includes analysis, experience and experimentation to work together in order to achieve a goal, that could be automated, not now, not with the current technology. Yes, AI and robotics are constantly moving forward, if in some beautiful day that happens, I would love to see it. Until then, human testing can not be automated, at least not the one I understand as high quality human testing, what we can automate, though, is shallow testing and act like it will do the job, just right.

Conclusion

Again, this is not a hate post. It is informative post, informative of the risks that automation has. Clever testers know these risks and base their testing on that knowledge. And yet there’s a lot of people running excited from conference to conference, explaining how automation is the silver bullet in software testing and it will do magic for your testing. This is wrong.

Tool assisted testing is useful, it is very useful. And it is fun to do, but we have to use it in the right way – as a tool, as an extension to our testing abilities, not like if it’s their replacement. We should know why and when it works, how it works and what traps it might have for us. In other words, when we want to use it, we should know why and if we are using it in the right way.

And most important, human and machine testing, simply don’t mix. One can assist the other, but they are not interchangeable.

Hope you liked the post. If you did, I would appreciate your shares and retweets. If you didn’t, I would love to see your opinion in the comments. Thanks for reading 🙂 Good luck.

Mr.Slavchev

Mr.Slavchev

Senior software testing engineer at https://siteground.com.
Experience in mobile, automation, usability and exploratory testing.
Rebel-driven tester, interested in the scientific part of testing and the thinking involved.
Testing troll for life.
Retired gamer and a beer lover.
Martial arts practitioner.

More Posts - Website

Follow Me:
TwitterFacebookLinkedInGoogle Plus


13 thoughts on “Test automation – the bitter truth”

  1. Bas says:

    Hey Viktor, this is spot on in so many ways. Love the way some people are seeing the light just as I am starting to. Thanks for sharing.

    1. Mr.Slavchev Mr.Slavchev says:

      Thanks for the comment, Bas. I am really happy to say, that many of these thoughts were inspired by posts I read in your blog. Thanks for that, too.
      Good luck. 🙂

  2. Ash Winter says:

    Cool post, brought together lots of thinking around the community. Silver bullets do need to be challenged but coherently, which this post achieves.

    The only thing missing for me is the role Developers on the team play in automation. I think they have a large part to play here too, I rarely write check automation myself, but advise on what is effective to automate, plus what tooling would be useful to assist testing. I guess my other truth would be that it’s a team game over being a specific role.

    1. Mr.Slavchev Mr.Slavchev says:

      Thanks for the comment, Ash. And you bring a great topic – ” it’s a team game over being a specific role”. Also, your point about the role of development in automated checking is interesting topic, I will think about it.
      Thanks.

  3. Viktor, my hat’s off to you. This is another great blog post. I love the way how you tackle the topic and set the limits of what automation can do. You answer most of my questions I asked a few weeks ago here: https://testpappy.wordpress.com/2016/04/15/test-automation-am-i-the-only-one/
    And you answer them from the view of a tool-smith understanding the capabilities and boundaries of his tools. Well done. I’ll definitely share this with my colleagues.

    1. Mr.Slavchev Mr.Slavchev says:

      Thank you, Patrick. This is quite a compliment from a person with your experience, I really appreciate it.
      Yes, indeed we think in the same direction and I am happy more and more testers are getting “awakened” from the “automation excitement” and start think for it in a more reasonable way.
      Thanks for the feedback. 🙂

  4. Stéphane Colson says:

    Thanks, I like to read this these days, totally agree. Could you please elaborate on “If you think your existing testers can do it part-time, while doing their day-to-day tasks, my answer is … yeah, right”?
    I think I agree that you really need to have someone dedicated to this, in order to be a real expert of automation, but what would be your points about this? My managers think that the all dev team writing their own automated checks (end2end checks) is a good idea.
    Thanks, and I will share the post 😉

  5. Mr.Slavchev Mr.Slavchev says:

    Hi, Stephane! Thanks for the positive feedback.
    My opinion is that it is almost as big of an issue to leave the developers to develop the automated checks alone, as it is if you leave it to QA team that has ongoing tasks to do. In case with devs writing tests, it could be even hard for them to come with the right testing critical thinking.
    That’s why, I guess devs might add a lot in developing the framework – after all they are the pro’s in writing code, so fromm code maintenance and code architecture perspective they can do it better. But when it comes to writing the checks, it is time for us to jump in, or at least provide information which are the “interesting” paths we wish to apply automated checks for.
    Is that answer helpful for your question?
    Thanks for commenting. 🙂

    1. Stéphane Colson says:

      That’s why I’d prefer a dev, dedicated to this task, which no more develop features/bug fix, but only checking code with the help of QA to chose test cases to automate….then this dev will have the critical thinking/QA mindset. At least, I hope so…and it probably depends on the team.
      Thanks for the answer and the feedback 😉

  6. Great article, I completely agree on so many fronts!! I have some discrepencies when you look at cost, due to the fact that cost associated with automation is extremely complex for every individual circumstance.

    For example; Reduces cost, in that if you introduce automation in an agile environment and provide regression testing earlier on in the cycle, or the earlier it is caught the more money you save.
    Also, what’s the cost of not automating? How many manual, man hours, benefits, training, etc.
    Humans are humans, they add the human element, but whats the cost of letting a failed product into production due to a small mundane human error?
    In healthcare specifically, hospitals are able to reduce their labor budget in IT heavily, I am talking about 40x reduction in labor budget to run bi-monthly releases by their EMR provider. That is a very real cost that is mitigated by automation in an industry that thrives on razor-thin budgets.

    1. Mr.Slavchev Mr.Slavchev says:

      Hello, Chris. Thanks for the comment and the positive feedback. Below are my answers to some of your questions.
      At first, I hope you understood from the article, that I am not trying to say “do not automate”, but “Automate in a meaningful way, answering the question “Why am I doing it?””.
      “…due to the fact that cost associated with automation is extremely complex for every individual circumstance” I’d say it’s not just automation, it’s everything, every project is unique and has different complexity, that’s why I doubt bold statements like “reduces cost” – how does it reduce it, compared to what other project, how was that assessed, where is the proof for that? And people seem to get such statements for granted.
      For example; Reduces cost, in that if you introduce automation in an agile environment and provide regression testing earlier on in the cycle, or the earlier it is caught the more money you save.
      I don’t see how this is automation exclusive only, in CDT community we share the believe that testing should be performed as early as there is something testable,no matter if it is version of the product, documentation file, design spec or anything else.
      Again, you are not answering the questions I talk about – cost of automation, hours involved and when subtracted, how does that ends with cost reduction, I would pretty much like to see that explained, other wise it still seems to me like “magic solution”.
      Humans are humans, they add the human element, but whats the cost of letting a failed product into production due to a small mundane human error?
      How does automation ignore that “small mundane human error” since it is written and supported by human? Do you think human coders never make mistakes, that bugs don’t exist in automation code?
      In healthcare specifically, hospitals are able to reduce their labor budget in IT heavily, I am talking about 40x reduction in labor budget to run bi-monthly releases by their EMR provider. That is a very real cost that is mitigated by automation in an industry that thrives on razor-thin budgets.
      That’s a great statistic. I would like to know where is that result coming from, how was the research performed, compared to what other project, “tested manually” was that measured and how. How are we sure that the two project are comparable in first place so we could compare? Can you provide url to the research that you quote, I would like to take a look at it, definitely.
      Thanks for the comment, Chris! 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

I am speaking at

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Goodreads Read books

Follow me on Twitter