Loved this post by Paul Takken on Agile as observed in the nature and the lesson we can learn from observation of some natural processes. Great parallel and great article: Making Agile Even More Awesome. By Nature.
This week’s outdated concept in testing will be kind of celebrity in its area, because it created so much confusion, so much excitement at the same time and is so often misused and misunderstood, that it’s probably the best candidate for review in outdated testing concepts and should take its honorable place – automation in testing.
Few words before we start…
The definition that I use about automation is in the general meaning – automating an action and it’s possible uses in the testing process, and by this I want to explicitly state that I mean automating an action in testing and not automating the process of testing. I deliberately try not to use the term “checking” and dive into discussion about testing and checking, just because I don’t feel ready to give my opinion on that. Other wise, the post “Testing and checking refined” and the work Michael Bolton and James Bach have put into it, is amazing, I share many of their thoughts and conclusions and I would strongly recommend anyone to read it.
Outdated concept #3: The cult to automation.
I believe every one of us, testers, did at least in some period from his career been part of the cult to automation as I said earlier, automation created a lot of excitement, as well as a lot of confusion. The main reason behind both is the same, it was and still is presented as the ultimate solution to problems in software testing, it’s “The Cure” of everything. When I was a newbie (in fact I still am, I just pretend I’m a smart ass 😀 ) I was presented with this bright vision of automation and how cool it is, and all the possibilities it could give you and so on and it looked so cool and awesome, I was sure that’s the thing I want to do. Of course we would fall for that, we are not dumb, we all want to perform our job faster, more efficiently, with fewer errors on our side and automation was pretending to provide that. BUT this believe back then, was based on some false assumptions, we need to take a look at:
“Automation is cool, because it saves time” – from the perspective of “I have to fill in that form with all that data and submit it” or “have the script do that for me”, yes, it looks like it saves time. But there are so many other aspects missed here – building the proper infrastructure to have effective automated scripts is a time-consuming activity and it is a full development process, having all the development phases in it, including bugs. We can spend way more time debugging our scripts, rather than just perform the tests manually.
The maintenance of these tests is as well time-consuming activity, consider all the changes that will occur in your application, for example, a simple hiding of an input field in the form might cause all your tests to break and you will have to spend time in order to adapt them.
There’s different types of testing, requiring different type of infrastructure and we can’t always reuse all our tests, which is another time investment.
“Automation is cool, because we can automate user interaction with the system under test (SUT)”. No, we can’t. We can make the script act on it, but not interact withit. The term interaction itself suggests the process of mutual action, of communication. Human can communicate, process and evaluate the information that is distributed from the SUT, automated script can not, it can just act.
“Automation is cool, because we can automate all the tests.” … and rainbows and ponies, that poop Skittles candies and other mythic creatures … This obviously is the “elephant in the room”, we all have seen it, sometimes. Everyone imagines the utopia where all of us will be able to automate all the tests that we will have to do and we will just go and “click” in our IDE and it will run on its own, will “execute” all the tests, will make screen shots for us, report a bug if a script fails, attach logs, screen shot, it will probably do the dishes and the laundry, and eventually breakfast for us in the morning … joking of course. This is impossible, for too many reasons some of which are – there’s no such thing as “all tests”, we all know exhaustive testing is impossible, therefore all tests is a hollow statement, we can’t even sit and create all the scripts that will be executed. Another reason is, not all types of testing are good candidates for automation. The non-functional tests or as the CDT community prefers to call them – para-functional types of testing are a great example – usability, accessibility, installability, maintainability etc.
Automation is cool, because testing, in a nutshell is performing series of predefined actions. Saying this is like saying “software development is just writing code”. It’s just the tip of the iceberg, there’s so much more that testing includes in order to provide a high quality service – observation, exploration, critical thinking, evaluation, experimentation, application of different heuristics, adapting to specific conditions and context … and many, many more actions, that could not be predicted and predefined.
The list can go on and on, if you prefer you can continue it on your own. In time, the automation concept has become that demi-god that is supposed to solve issues automagically. Mike Talks (@TestSheepNZ) amazed me with his article on MacGuffin effect in testing, so inspired by him I will make another analogy.
In ancient Greek tragedy, there was the term – “deus ex machina” or “god from the machine”. In ancient tragedy it was a character, playing a deity that was brought on the stage with a crane(that’s the machine part), in order to solve unsolvable mystery or a problem that seemed to came to a dead-end. Enough with the history lesson, in case you don’t see the resemblance I will be more explicit, I believe that the common understanding for automation is the same – it should solve all problems, detect all errors, automate all actions effortlessly. Which is impossible and this is where confusion comes from. As in the ancient tragedy in automation also, there’s a human pulling the crane or the automated script. And it’s all up to human intellect to use efficiently the tool to automate action or fall into false beliefs.
So, is automation evil? Is it the enemy?
Of course not. This article isn’t about how useless automation is. It’s about how to think of it in order to make it useful, rather than expect surreal results of its use.
One more thing, sometimes I see some rebellious comments among testers, that go to the extreme of – automation is useless, it is wrong, nothing should be automated, and the most absurd – “we will be replaced by machines”. That last claim alone, shows that the person who said it, declares his agreement, that testing performed by a human is completely irreplaceable by an automated tool/script, which as I stated above is false. And actually, it’s proven long ago by respected contributors to the testing craft such as Cem Kaner and Gerald M. Weinberg, so I am literally just paraphrasing them.
So, how is automation useful?
At first we should consider to stop the whole manual vs. automation dispute. This was never true in first place, there is no such thing as manual only, or automation only (having “manual” in the meaning of “human performed testing”).
We should threat automation as a tool, not as a solution. It’s a tool to help us gain information about the product we are testing.
As a tool, we should know what is it good for as well as what are its limitations. There’s some purposes for which automation is useless, let’s not try to hunt rabbits with a bazooka.
Automation tools can only automate actions. They cannot automate observation, analytical thinking, problem solving, nor any strategy for discovering bugs.
Last, but not least and here I am quoting by memory Michael Bolton’s tweet – we shouldn’t think of automation as freeing us from doing something, but as enabling us to do something. So, the option of letting the thing run and go to grab a beer and watch the game isn’t an option, sorry.
That’s it for this part. I know it’s a controversial topic and many more opinions are about to come on it, but I believe talking about it will help us to clear things out. Of course, I should give the credit to the materials I used – many of the conclusions I make in this blog post are not my own, but influenced by James Bach, Michael Bolton, Cem Kaner, Gerald Weinberg’s works and many other blog posts and comments I’ve read.
Thanks for your time, I hope it was interesting and useful for you, too and, of course, don’t forget – your opinion matters and I will be happy to read about it. 😉
Hello everyone and thanks for the positive feedback from the previous part of this weekly summary, I am really happy the format suits you and really hope it’s useful for you.
So, the posts I picked for this week are:
This interview with Jerry Weinberg was as awesome as it possibly could, as in everything Jerry Weinberg always provokes thinking and analysis over ones way of testing and general understanding over testings philosophy. I strongly recommend it and one really inspirational quote from that talk, which was really knocking out was – “First of all, management must avoid building or encouraging a blaming culture. Blame kills learning.” INTERVIEW WITH JERRY WEINBERG
James Bach, one more time threw the testing community into hard thinking with a provoking post on redefining testing terms, in this post he gave a live demonstration of his Socratic approach in asking students to explain and define testing terms. The post itself is interesting from the perspective if the reader himself could answer all the questions in a satisfying way. The end is left open, so the readers could give their input and the discussion went on and on and on … I think it’s even more interesting by the post itself due to the teacher – student interaction. Re-Inventing Testing: What is Integration Testing? (Part 1)
The new Testing circus issue was out last week, some really interesting articles caught my attention right there, one by Rajesh Mathur called “The Painter who had a Certificate” – pretty interesting analogy between testing and painting, really creative. The other was the loony sounding “10 Million USD Research Project to Guarantee Bug-Free Software”, which sounds almost as ambitious as finding the eternal engine or the ultimate painkiller, but it is interesting read, after all. Testing circus December.
Few more great posts about Docker and all its diverse uses, I admit I have no experience with Raspberry Pi and IOT stuff, but I bet this first article is going to be interesting for people who have passion on poking hardware and building stuff on their own. Building a Rasperry Pi based Smart Gateway for IoT
The next one is mostly a practical tutorial on how to use Docker in order to improve encapsulation and module control for our binaries, deamons and processes, which is basically the idea behind running containerized applications. Dockerizing Commands
Another wonderful post by Jeff Nyman, I like his way of writing for he tends to go deeper than just a tutorial and has a really interesting manner to connect things from different domains. Here he made a really profound analysis on what BDD is and what possible traps it might hide. The BDD Lure and Trap
Recently I watched this awesome talk by Simon Stewart which is kind of old, but still awesome, since I am quite green yet in the area of test automation or at least the meaningful and useful test automation. In it he used to talk about test isolation and how important it is. In a few words test isolation is the ability of your tests to run in an isolated environment, but also to run separately, non-depending on each other. So, there’s many ways to do so, may be if you’re interested you should watch the talk, but one of them is to randomize test execution. So, I decided to incorporate this using TestNG, since we are using it in our code and it’s kinda interesting to have this in my tool set, I didn’t find it a lot on the web.
How to randomize test execution using Java and TestNG.
Here are some code examples that I made just for the demo, I repeat for the demo, so don’t mind the moronish names that I’ve chosen for my classes. And it’s fairly simple example – we have a test class with some code, in our case it’s just outputting “in method …”. That’s just to keep track on which test is executed when, what’s the sequence of our test execution. So here’s the code of our test class:
System.out.println("Here in test 1.");
System.out.println("Here in test 2.");
System.out.println("Here in test 3.");
System.out.println("Here in test 4.");
System.out.println("Here in test 5.");
As I said, simply moronish code example. So if we run this as testNG test we get the following output in the console:
And it does that every time we run them. So what we need is to randomize test execution in our class.
Why randomize test execution? How is this important?
Well, if you have a test suite of 30-40 test or more and you run them every time in the same order, try applying this solution and you will find out why. Turns out we as coders assume that we have many things for granted, for example – a specific state of the data in the DB or a specific screen that we land or start from. Turns out, this is not like that and unknowingly we’ve incorporated some dependency on our tests, which is not good if we want to use them in continuous integration or in some other environment. So, the biggest plus from having a good test isolation is – we don’t want to fix the damn thing every fucking time we change the environment it runs at.
System.out.println(String.format("Randomizing order of tests with seed: %s",seed));
And in order to randomize test execution of certain class you have to add the following annotation to your test class:
So, when your run the tests now, the output looks like this.
Randomizing order of tests with seed:217769194550514
Which is a bit better. The cool part of randomizing test class like this is the nanoseconds feed of the random, as you see it is logged on the console and if you find anything wrong in a specific sequence you could always copy the feed from console and hard code it in our random feed, in order to re-create the exact same scenario. Which is fucking awesome.
Thank you all for reading and hope that was interesting and helpful for all of you. If you liked it, please share it with your friends and/or share your thoughts in the comments.