Test automation – the bitter truth

Reading Time: 7 minutes

Recently, I came across this awesome blog post by Mike Talks where he tells the story of his personal experience with test automation and all the false expectations that he and his team had about it, and it is really an awesome post, if you hadn’t read it, go do it now.

And I really got inspired by it, I recently write a little post concerning automation and I figured out I had much more to say. I was really busy doing the “Software testing is not….” series, so I didn’t really feel like going off topic. But now I feel the time came to tell each other some bitter truth about “test automation”. It is bitter, because people don’t like it, they don’t like talking about it, they don’t like promoting automation like this, but, yet it’s the truth.

It all starts with love.

I have to say I love automation, I really do. And the reason why I write this post is, because I love testing and automation and I want to give the best perspective how they can supplement each other. Yes, it is related to saying some uncomfortable truths about it, but it has to be done.

So, I said I love test automation, not because I believe it is the one and only solution to all my testing problems, yet I am sure when I was new I had that delusion about it, probably. The reason I like it, is because I like to code. I like writing cryptic “spells” and see them turn into magic and I really don’t care if it’s a source file, an interpreter or a console, I just love it.

I see a lot of people being excited about automation, but for a different reason – because they see it as the ultimate solution to all testing problems, the silver bullet, the philosopher’s stone or you name it, it seems to them that human testing might be  replaced with automated scripts and everything will be great, we will ship the product in no time and the world will be a better place.

I have to object this. And trust me, this is not a hate post against automation, in fact it is post aiming to justify why automation is useful, but useful in the right way, with the right expectations, not the way that we want it to be. So, in order to achieve this, I will have to write for some bitter truths, about automation.

Bitter truth # 1: It’s not test automation, it is tool assisted testing.

James Bach and Michael Bolton did an awesome job, making the difference between “testing” and “checking” , so I think it will be useless for me to repeat what they already said. If you are  not familiar with their work, I strongly recommend you do check “Testing and checking refined” and their white paper on automation in testing – A Context-Driven Approach to Automation in Testing.

In a nutshell, the testing that we perform as humans is not easy to translate in machine language, we can not expect from a machine to act like a human. So, they make the difference between “checking” – shallow form of testing that could be translated into set of instructions and  expected conditions and “testing” – describing the process of human testing with all its complexity and depth. Therefore, the term “test automation” carries a false meaning – we can not automate testing, we can automate checking and it is more correct to say “automated checking”. Yet, I prefer another  term they used, for the reason it sounds more elegant and provides more information – “tool assisted testing”. Because that’s what we do, when we automate something – we use tools to assist our testing, that’s it, we don’t want to replace it, or get rid of it. We want to use tools to enable us, to do more, rather than to replace the human effort in testing.

Bitter truth # 2: Automation doesn’t decrease cost of testing.

I am really interested how does that equation work?. People claim automation reduce cost of testing. Let’s see.

• You hire an additional person to deal with building the framework and write the tests. (If you think your existing testers can do it part-time, while doing their day-to-day tasks, my answer is … yeah, right ) so that’s additional cost
• You probably will spend additional money for the tool licence, or otherwise you will use open source tools which means that guy will have to spend additional man hours in order to make that framework work for your specific testing needs.
• You will have to pay that guy to write the actual tests, not only the framework, so that’s additional cost.
• The code that you write isn’t “pixie dust”, it’s not perfect, it turns into one additional code base that has to be taken care of, just as every other production code – maintenance, refactoring, debugging, adding new “features” to it, keep it updated. Guess what, it all will cost you money.
• And of course, let’s not forget about all the moments of “oh” and “ah” and “what was the guy writing this fucking thinking” that you will run into, related to the framework that you use and its own little bugs and specifics, that will also cost you additional money and time investment.

I think that’s enough. The main reason that people give for the “low-cost” of automation is – it pays back in long-term. Now that’s awesome. And it would be true, if we accept that once written a check is a constant that never changes, that it works perfectly and our application never changes until the end of the world. Well, that would be nice, but there’s two problems:

1. We all know from our testing experience that things written in code don’t actually work perfect, out of the box. In fact it takes a lot of effort to make them work.
2. If your code base never changes, if it stays the same, you don’t add anything, you don’t change anything, you don’t redesign anything, it is pretty possible that you are going out of business. We are working in a fast evolving environment and we need to be flexible to changes.

So, again. I don’t get it, how having all of the above leads to cost reduction.

Bitter truth # 3: Testing code isn’t perfect.

I see that very often at conferences and in talks among colleagues. It seems that test automation scripts are some kind of miracle of the nature that never breaks, never has bugs, once you write it and it’s a fairy tale. Well, unfortunately it is not.

As stated above already, it is code, normal code just like any other, it has “its needs” – to be maintained, to be refactored and guess what … to be tested :O Why is nobody talking about testing the automated checks, I mean are these guys testers, did they left their brains in the fridge or something?! It’s code, it has bugs, it has to be tested, so we make sure it checks what we want it to check, wake up! Not only that, I bet you, that doing your automation, you will be dealing with bugs introduced by your checks, much more than bugs introduced by the code of the app that you test.

Bitter truth # 4: Automation in testing is more reliable… it’s also dumb as fuck

One of the many reasons why testers praise automation is – it is more reliable than human testing. It can insure us that every time we execute exactly the same check, with exactly the same start conditions, which is good, at least it sounds good. The truth is, it’s simply dumb and I mean machine-like dumb. It’s an instruction, it simply does what it is told, nothing more, nothing less. Yes, it does make the same thing every time over and over, until you don’t want it do that thing any more. Then it turns into huge pain in the ass, because someone will have to spend the time to go and update the check and make sure it checks the new condition correctly. In other words, automated checks are not very flexible, when it comes to changes and changes are something we see pretty often in our  industry.

Bitter truth # 5: Automated checks don’t eliminate human error

For reasons that I stated above I claim that automated checks can not eliminate human errors. Yes, they eliminate natural variety that human actions have, but that doesn’t mean we don’t introduce many more different new human errors while writing the code. The conclusion is simple, while the code is produced by a human being, it is possible it has errors in it, it’s just life.

Not only that, but having our checks automated, we introduce the possibility for machine error. Yes, our code has all the small bugs and weird behaviors that Java, C#, Python, Php or any other language have. The framework that we use also might have these, the interaction that it has with the infrastructure it runs on, might also introduce errors. So, we must be aware of that.

Bitter truth # 6: Automated checks are not human testing automatically

I see and hear that pretty often, everyone saying – “we know we can’t automate everything” and yet they continue to talk as if they could. Not only that, they talk like if they could mimic exactly the same process that a human is doing. And it is not possible. There is no human activity that is cognitively driven, that includes analysis, experience and experimentation to work together in order to achieve a goal, that could be automated, not now, not with the current technology. Yes, AI and robotics are constantly moving forward, if in some beautiful day that happens, I would love to see it. Until then, human testing can not be automated, at least not the one I understand as high quality human testing, what we can automate, though, is shallow testing and act like it will do the job, just right.

Conclusion

Again, this is not a hate post. It is informative post, informative of the risks that automation has. Clever testers know these risks and base their testing on that knowledge. And yet there’s a lot of people running excited from conference to conference, explaining how automation is the silver bullet in software testing and it will do magic for your testing. This is wrong.

Tool assisted testing is useful, it is very useful. And it is fun to do, but we have to use it in the right way – as a tool, as an extension to our testing abilities, not like if it’s their replacement. We should know why and when it works, how it works and what traps it might have for us. In other words, when we want to use it, we should know why and if we are using it in the right way.

And most important, human and machine testing, simply don’t mix. One can assist the other, but they are not interchangeable.

Hope you liked the post. If you did, I would appreciate your shares and retweets. If you didn’t, I would love to see your opinion in the comments. Thanks for reading 🙂 Good luck.