Outdated testing concepts # 3.

Link to Outdated concept #1 – Anyone can test.

Link to Outdated concept #2 – The guardian of quality.

This week’s outdated concept in testing will be kind of celebrity in its area, because it created so much confusion, so much excitement at the same time and is so often misused and misunderstood, that it’s probably the best candidate for review in outdated testing concepts and should take its honorable place – automation in testing.

rusty gears
Source: http://i.istockimg.com/file_thumbview_approve/11202223/3/stock-photo-11202223-rusty-gears.jpg

Few words before we start…

The definition that I use about automation is in the general meaning – automating an action and it’s possible uses in the testing process, and by this I want to explicitly state that I mean automating an action in testing and not automating the process of testing. I deliberately try not to use the term “checking” and dive into discussion about testing and checking, just because I don’t feel ready to give my opinion on that. Other wise, the post “Testing and checking refined” and the work Michael Bolton and James Bach have put into it, is amazing, I share many of their thoughts and conclusions and I would strongly recommend anyone to read it.

Outdated concept #3: The cult to automation.

I believe every one of us, testers, did at least in some period from his career been part of the cult to automation as I said earlier, automation created a lot of excitement, as well as a lot of confusion. The main reason behind both is the same, it was and still is presented as the ultimate solution to problems in software testing, it’s “The Cure” of everything. When I was a newbie (in fact I still am, I just pretend I’m a smart ass 😀 ) I was presented with this bright vision of automation and how cool it is, and all the possibilities it could give you and so on and it looked so cool and awesome, I was sure that’s the thing I want to do. Of course we would fall for that, we are not dumb, we all want to perform our job faster, more efficiently, with fewer errors on our side and automation was pretending to provide that. BUT this believe back then, was based on some false assumptions, we need to take a look at:

  • “Automation is cool, because it saves time” – from the perspective of “I have to fill in that form with all that data and submit it” or “have the script do that for me”, yes, it looks like it saves time. But there are so many other aspects missed here – building the proper infrastructure to have effective automated scripts is a time-consuming activity and it is a full development process, having all the development phases in it, including bugs. We can spend way more time debugging our scripts, rather than just perform the tests manually.
    The maintenance of these tests is as well time-consuming activity, consider all the changes that will occur in your application, for example, a simple hiding of an input field in the form might cause all your tests to break and you will have to spend time in order to adapt them.
    There’s different types of testing, requiring different type of infrastructure and we can’t always reuse all our tests, which is another time investment.
  • “Automation is cool, because we can automate user interaction with the system under test (SUT)”.
    No, we can’t. We can make the script act on it, but not interact with it. The term interaction itself suggests the process of mutual action, of communication. Human can communicate, process and evaluate the information that is distributed from the SUT, automated script can not, it can just act.
  • “Automation is cool, because we can automate all the tests.”
    … and rainbows and ponies, that poop Skittles candies and other mythic creatures … This obviously is the “elephant in the room”, we all have seen it, sometimes. Everyone imagines the utopia where all of us will be able to automate all the tests that we will have to do and we will just go and “click” in our IDE and it will run on its own, will “execute” all the tests, will make screen shots for us, report a bug if a script fails, attach logs, screen shot, it will probably do the dishes and the laundry, and eventually breakfast for us in the morning … joking of course.  This is impossible, for too many reasons some of which are – there’s no such thing as “all tests”, we all know exhaustive testing is impossible, therefore all tests is a hollow statement, we can’t even sit and create all the scripts that will be executed. Another reason is, not all types of testing are good candidates for automation. The non-functional tests or as the CDT community prefers to call them – para-functional types of testing are a great example – usability, accessibility,  installability, maintainability etc.
  • Automation is cool, because testing, in a nutshell is performing series of predefined actions.
    Saying this is like saying “software development is just writing code”. It’s just the tip of the iceberg, there’s so much more that testing includes in order to provide a high quality service – observation, exploration, critical thinking, evaluation, experimentation, application of different heuristics, adapting to specific conditions and context … and many, many more actions, that could not be predicted and predefined.

The list can go on and on, if you prefer you can continue it on your own. In time, the automation concept has become that demi-god that is supposed to solve issues automagically. Mike Talks (@TestSheepNZ) amazed me with his article on MacGuffin effect in testing, so inspired by him I will make another analogy.

In ancient Greek tragedy, there was the term – “deus ex machina” or “god from the machine”. In ancient tragedy it was a character, playing a deity that was brought on the stage with a crane(that’s the machine part), in order to solve unsolvable mystery or a problem that seemed to came to a dead-end. Enough with the history lesson, in case you don’t see the resemblance I will be more explicit, I believe that the common understanding for automation is the same – it should solve all problems, detect all errors, automate all actions effortlessly. Which is impossible and this is where confusion comes from. As in the ancient tragedy in automation also, there’s a human pulling the crane or the automated script. And it’s all up to human intellect to use efficiently the tool to automate action or fall into false beliefs.

So, is automation evil? Is it the enemy?

Of course not. This article isn’t about how useless automation is. It’s about how to think of it in order to make it useful, rather than expect surreal results of its use.

One more thing, sometimes I see some rebellious comments among testers, that go to the extreme of – automation is useless, it is wrong, nothing should be automated, and the most absurd – “we will be replaced by machines”. That last claim alone, shows that the person who said it, declares his agreement, that testing performed by a human is completely irreplaceable by an automated tool/script, which as I stated above is false. And actually, it’s proven long ago by respected contributors to the testing craft such as Cem Kaner and Gerald M. Weinberg, so I am literally just paraphrasing them.

So, how is automation useful?

  • At first we should consider to stop the whole manual vs. automation dispute. This was never true in first place, there is no such thing as manual only, or automation only (having “manual” in the meaning of “human performed testing”).
  • We should threat automation as a tool, not as a solution. It’s a tool to help us gain information about the product we are testing.
  • As a tool, we should know what is it good for as well as what are its limitations. There’s some purposes for which automation is useless, let’s not try to hunt rabbits with a bazooka.
  • Automation tools can only automate actions. They cannot automate observation, analytical thinking, problem solving, nor any strategy for discovering bugs.
  • Last, but not least and here I am quoting by memory Michael Bolton’s tweet – we shouldn’t think of automation as freeing us from doing something, but as enabling us to do something. So, the option of letting the thing run and go to grab a beer and watch the game isn’t an option, sorry.

That’s it for this part. I know it’s a controversial topic and many more opinions are about to come on it, but I believe talking about it will help us to clear things out. Of course, I should give the credit to the materials I used – many of the conclusions I make in this blog post are not my own, but influenced by James Bach, Michael Bolton, Cem Kaner, Gerald Weinberg’s works and many other blog posts and comments I’ve read.

Thanks for your time, I hope it was interesting and useful for you, too and, of course, don’t forget – your opinion matters and I will be happy to read about it. 😉