If there’s a contest for the buzzword of the century, although I don’t think testing was around for a century, yet, I am almost 100% sure it will be test automation, or automated testing. If you read or listen to test talks it’s “automation this”, “automation that”, “automation everything”.
It’s definitely over used and over abused and I bet large number of the people giving advises on it have no fucking clue what they are talking about. Or at least the statements produced by them show absolute lack of understanding about the purpose of automation. Then I said, wait a minute …
It was discussed plenty of times that the terms “test automation” and “automation/ed testing” are misnomers in many ways, that we can’t actually “automate” testing, that we only can automate subsets of it, yada, yada, yada I won’t repeat this, as I am getting really bored having the same conversation over and over again. The thing that came and struck me as a lightning was that actually – there is no “testing automation”, there is “testing” and “automation”. Seems that our thinking hadn’t evolved enough yet, so we can combine both in an efficient way. Let me demonstrate.
Two different mindsets.
I got this observation somewhere around preparations about Testbash Munich and it started to grow and evolve and I needed time to give it shape in my mind.
Whenever you as a tester are at a “test automation” conference or you as automation expert are at testing conference – don’t you feel that weird, feeling that people are talking in a different language?
Because that’s how I feel – every time an automation person starts speaking about “flaky tests”, I am like “c’mon, man, there’s no flaky tests, there’s just bad test design! When are you going to learn it? There was plenty of discussions about it, where were you?”
I bet test automators are also being puzzled every time they listen to talks or conversations going deeper in teamwork and critical thinking, bias, heuristics and are probably thinking – “what the fuck are these guys talking about? Don’t they know how many tools and technologies there is? It’s all code, why all the drama?”.
The bottom line that I am trying to get to here, is also the answer to the question – why can’t we create automation that really helps testing and the answer is – because we have initial misunderstanding of the goals of testing and automation and no possible transparency of what problems the other “camp” is trying to solve.
From my point of view, testing and automation are simply talking in different languages and aligned towards different goals, which is the reason why there’s initial misunderstanding in the communication between both.
The testing mindset.
In order to figure out how we fail to communicate, let’s first see what are our goals. These are just my observations, probably biased in a way, so don’t treat them as a recipe.
What problems testers are trying to solve:
- In short-term – provide information about existing problems and assist with their resolution.
- In long-term – provide information about evaluation of quality and support making an informed decision.
- Perform exploration in order to understand the product or system under test
- Perform experiments(this is what “tests” are, in fact) with the product in order to gain profound understanding and information about it, its state, behavior existing problems etc.
- Trying to conduct experiments in an incremental manner so that every next session gives us better information, new information, or such of different aspect – functional, parafunctional, performance, security, etc.
- Evaluate risks and their impact on quality.
- Build understanding about products supposed working and quality criteria and use them as oracles.
- Trying to challenge the stability of the product by any means possible.
Where does testing mindset fail to assist automation?
Normally the testing mindset fails to assist automation in a few different ways.
“The testing resistance”
In a way our efforts in testing turned into the “testing resistance” as we are able to almost always critique automation and point out what it is incapable of doing, but not very often to demonstrate the strong benefits of automation in testing. I also realize my guilt in this one, because I also did it in “Bitter truths about automation”, and said nothing to support automation’s usefulness, but I will try to fix it.
I believe the “manual/automation” stupidity has to do a lot here too and all the bullshit around “being replaced by machines” and other ways people try to look clever, by speaking sci-fi stuff. May be it’s about time to get a realistic view and try to build a solution, instead of just naming problems.
Failing to address good automation aspects of testing
Another way testing fails to help automation, is by trying to squeeze all the complexity in testing and convert it into bytes.
Either because of the “replace the manual morons with scripts” fallacy or because we are aware of testings real value and complexity and want to demonstrate it, we tend to give almost impossible tasks to code to our automation engineers.
The thing we should realize here in order to improve is – testing and automation are good for specific problems, which often don’t mix. The question is not one or another, it’s “how both can co-exist in symbiosis”.
The automation mindset
This mindset is way different from the testing one. People involved in automation have much more a development oriented thinking rather than a testing one. I don’t say it’s bad, it’s simply different. And we should appreciate it, because some automation developers are brilliant coders.
What problems does automation try to solve?
When used in the right manner…
- Functional correctness
- Precision checks
- Low level checks
- Stability
- Providing feedback in fast and consistent manner
- Asserting definitive conditions
- Isolate variable conditions from tests
- Provide consistent execution every time, over and over again
- Solving all types of programmatic problems such as – code sanity, bugs in testing code, complexity, re-usability, refactoring, etc.
Now having both, we can see that automation mind-set is much more technically and algorithmically oriented, while the testing mindset is mostly oriented towards unpredictability and critical approach. While testers try to experiment in an incremental way every time, to push harder in order to gain new knowledge, automators are trying to do the opposite – make sure tests are executed in the same environment, under the same conditions, every time. This is where a lot of us fail to realize testing and automation work in a different way and are not irreplaceable.
Where automation mindset fails to help testing?
The tools cult
We see that automation is very technical, very tool oriented, but this becomes problematic whenever automation engineers start treating tools as a solution or blame them for their own test design mistakes. In fact, many talks on automation dig so deeper on the “hows” of doing it, that they totally forget the “whys”. The result of this, is automation starts to feel totally detached from testing and again, may be “automation vs manual” and “replacing” one with the other talks have to be blamed for this.
But let’s not fool ourselves, the meaning and purpose of automation, and I am really happy I saw Bas Dijkstra got to the same conclusion in a brilliant article, few days ago – the main purpose of automation is to support testing.
No matter how robust is your framework, if you followed all the OOP principles, if it’s easily extendable and has loose coupling and strong cohesion, if your tests run within nanoseconds and manage test data in a perfect manner – all of this, is absolutely useless if your automation doesn’t uncover problems. If it doesn’t challenge the system aggressively enough to expose bugs. It might be a perfect piece of software, although perfection is utopic, rather than something achievable, it can be really masterpiece of code, but it won’t serve it’s purpose unless it provides meaningful, reviewable information about existing problems in the product.
How is this possible? We get to problem 2…
The belief of “what we do as testing is enough”
Because of the cult to tools, the will to replace testing, rather than support it and the focus on how to perform automation, most automation solutions are merely a demonstration, rather than experiments. Go to an automation conference, read an automation blog post – have you ever seen someone ask themselves “Am I testing the right thing right now?” or “Is this the best experiment I can produce with the tool I am currently using?” or “what is the quality of my tests”? I bet the answer is “no”. Very often automators get so busy mimicking human behavior, expanding menus, interacting with elements, transitioning between pages, that they forget the main focus that their code should have – to be a high quality test of certain condition and make sure the condition itself is worth testing.
Instead, I see everyone doing automation treating test design just like some fossil from the past that we figured out long ago. And it’s wrong, conducting meaningful experiments is the main goal for both – testing and automation.
The solution – build the missing link
Well, this is hardly a solution, I am talking to myself here, after all, but what I see as a solution here is the realization of few important things.
Testing and automation don’t mix.
Let’s cut the bullshit with “replacing” and “if”, and “when” and “manual” and try to do something productive, shall we?
As we see, goals of testing and automation try to solve different problems, if we treat them as mutually replaceable, we won’t gain any progress.
There’s no “universal soldiers”
In my humble opinion, there’s no one person with complex enough understanding about testing and complex understanding of automation and there’s nothing wrong with that. No need to try to ask testers to learn to code and write automation or ask automators to learn critical thinking and overcomplicate their code. Just stick with what you are happy doing and keep yourself open for discussion on how to be useful to testing the product or SUT.
Try to make your problems and goals transparent to others.
As testers it will be useful if we make an effort to explain where automation is needed and provide examples of where automation makes perfect sense and is helpful. This, of course, with the clear mind that, it does not and should not “take the effort from us”.
For automators, I believe it will be useful if effort is made to explain the technological limitations that our tools and approaches have, the eventual pitfalls of using one or another the time effort and the complexity involved.
Also, a “tea-spoon” of healthy skepticism will be useful, when writing automated scripts. One of the core components of the testing mentality is doubting in your own ability to perform good testing. Therefore, whenever writing automated scripts we should challenge their qualities in first place and not settle with simple confirmatory testing.
How I will try to solve the problem for myself?
As I said, I have “a debt” to provide information on how to use automation in a way productive to testing, so I decided to start a series of short posts, targeting answering practicle questions about automation that are obvious, but yet not often spoken of. I still haven’t come up with a heading, but I will.
Am I the right person to write about it? Who cares?! I have a blog, I will write for whatever I want, if people like it – they will show it, if they don’t – I will stop. 😀
I would love to read and discuss with you about your solutions of building the missing link. Any shares and retweets are very welcome. Thanks for reading. 🙂
Mr.Slavchev
Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.
Follow Me:
Great points made in the article. I will for sure share your article to start discussions. It also triggered a lot of things I would like to get changed. 🙂
I am looking forward to the follow up articles and will join discussions about the article on Twitter.
Dirk
Thank you, Dirk!
I am delighted that my thoughts in some way are being able to provoke action in people willing and being able to initiate change in their companies. I would love to hear how that changed your approach.
Good luck! 🙂
Brilliant and thought-provoking.
Thank you, keep it up!
Hi Viktor, just a heads up:
The twitter link on this page goes to “http://www.twitter.com/Mr_Slavchev” but I think it should rather go to “https://twitter.com/TheTestingTroll”
Cheers
Matthew (@CapeTownTester)
Hello, Matt!
I guess you mean the one in the signature. Yes, I fixed it. Thanks.
Thanks for the feedback, as well.
Good luck. 🙂
It’s not about what we want, it’s about what we have.
I agree that testing and automation should be 2 different jobs but the trouble is more and more companies blurr the lines between the two. More often in the detriment of testing (manual testing as they put it) considering it as an afterthought or something not important enough. Then there are scrum teams that must have automated testers(everyone needs them, right?) that end up doing 90% exploratory testing of some sorts due to many reasons that do not make the object of this article (nor should they).
I have participated in lots of discussions where automation is seen as the normal step in evolution from manual testing. I have considered this to be true also but I am now in doubt. Is it? Do you afford to be “just” a tester if the paychecks are so clearly uneven?
Hello, TestingDad!
Thanks for sharing your opinion on that topic. I believe we align in our views.
Yes, it is what we have, but what you call “blurring of lines”, I see in my experience as a very stubborn and forceful merging of skill sets, that simply are too different.
Yes, the industry aims to create cross-functional specialists, and yet what I see, again based on my experience, is automation engineers with limited or no testing expertise, propagating fallacies about testing, and on the other hand – testers who are forced to code, utterly unsatisfied of what they do, because that is what the “industry desires”. This is by no means a way to develop intrinsically motivated experts and it will reflect to the quality of testing, if it isn’t already.
Will I afford to remain “just” a tester? I don’t consider this a dilemma. All the talk of transitioning from “manual” to “automated” testing (two terms that were useful only to confuse people) is lying on the fundamental misunderstanding that testing is – easy, procedural set of actions that testers execute in a script-like manner, therefore it’s much more appealing to automate it. Everyone who tested for more than 6 months is aware this is far from the truth. This is normally an opinion supported by non-testers – devs, PMs, POs, management, etc.
The problem is not that automation has proven more efficient than human testing, it is that human testing gave up the fight without trying. The reason our colleagues think automation is the natural evolution of “manual” testing is because of our failure to demonstrate the value of real testing expertise and this is something I will never get tired to repeat – visualizing the value of testing is a primary goal to stand your ground as a tester. If you are able to demonstrate your value and expertise, tools that simply mimic actions an expert testers could perform will be no match for you or your paycheck.
Hello! Thanks for the article! Can you please talk more about “No universal soldiers” Topic in further article? I’ve seen some people majoring in one or another thing within different time periods, and can not say, that it was necessarily a bad thing…
Hello, Daria!
Sure thing I can, but let me know what are your questions here? Why is the “universal soldier” a threat for you? Did you felt a push from a company to become universal soldier? What are your experiences with it?
May be your thoughts will help me give more relevant advice or will help somebody else in the same situation as you.
Thanks.
Viktor
Hello Viktor!
I can not say that i definitely see “Universal soldier” as a threat and it’s interesting for me to find out how did you get to your conclusion on this.
I have following experience so far:
There were times, when i was focused on manual tests, but also enjoyed possibility to get coding experience step by step (since i was allowed to spend some percentage of my time on this topic). I find it nice to have possibility to switch between topics (and knowing the system pretty well, by the moment i’ve started working on test automation, was also beneficial). At that point i would doubt your point.
In current project my “major” is test automation. And since manual tests are done in exploratory manner, there is no test documentation i can use as a base for test automation. I am the only test engineer (meaning that other people, who execute tests don’t have experience with test documentation etc) in the team, and now i really miss having either people, who took care of test design or having more time myself for that.
Hello, Daria!
“I can not say that i definitely see “Universal soldier” as a threat and it’s interesting for me to find out how did you get to your conclusion on this.”
It’s not a conclusion, just a guess if you see it as such.
The part about the universal soldier means that I don’t think we can grow brilliant automation mindset along with brilliant testing mindset at the same person, at the same time.
Considering that (also in the article):
1. Automation and testing are not interchangeable and not doing the same thing, in a different manner. You may be see I avoid the term “test automation”, because we are not automating the testing, we are automating subset of the test execution and it’ s a fairly small one.
2. Testing and automation mindsets have different characteristics and they sometimes conflict.
Having this in mind I don’t see it possible to be expert automator and expert tester at the same time, there will definitely be a dominating side, which will be more compelling for you. This also comes with set of bias and weaknesses.
We can still approach these as roles – at one point you have your automator hat on, in the next you have your explorer hat on.
Of course, that doesn’t mean you can’t do both, career-wise. Of course you can, I hate when people ask me if I am manual or automation tester. To me testing is interdisciplinary and automation and coding are part of it.
What is useful to be aware of, is the boundaries of both mindsets and that we are prone to errors when we are in one role or another.
I am also in the situation you’re at – I started with coding and had basic skills, I moved to testing, was given “manual testing” to deal with. At some point I had both exploratory testing and automation in my tool belt. Do I consider myself expert in both – absolutely not. The more important – I see myself performing worse in critical thinking when I write automation. For example, whenever I write API tests I tend to forget aggressive cases that expose risks and problems. During exploratory sessions, I find problems that I didn’t even think of while I was coding and written tests for, just like part of my brain had an outage. 🙂 So, I go back and add automated checks for these.
Conclusion that I derive from this is – when we speak about creating cross-functional teams and people who test and write code, at the same time – there will be such, that want to do it naturally, just like you, me and many others. There will also be people who will prefer to stick with one of both mindsets and that’s totally OK, that doesn’t mean they are counter-progressive, not willing to change or whatever. Forcing them to adopt the other mindset can actually have destructive, rather then constructive effect. Instead, it makes sense to teach testers to visualize the automation needs of testing and for automators to discuss the gaps that their tests have and what changes can be made to provide better support to testing.
Does that help to clarify what I meant? Do you find some part of the article above confusing and not stating that clearly enough?
Thanks.
Hello Viktor!
Thank you very much for detailed response! It clarified what you mean really well.
I wouldn’t say, that article was either not clear, nor confusing. I just saw there comparing of two different roles and automatically considered those roles belonging to two different specialists. So i just missed angel that would cover people who are interested/involved in both topics.
Thanks again!:)
Happy to help! 🙂
All that is written above is great – but you haven’t answered the “Missing Link” issue – as most companies don’t either…
To Me – The “Missing Link” is having a minimal UI to allow testers to easily incorporate the benefits of automation within their daily chores – thus becoming what I call – “Super Testers”.
We have already invested in automation infrastructure and tons of functions – why not make these easily available to assist the more tedious & repeatable tasks of a Tester either when running test cases, or when exploring the system?
I have found its just a matter of simple UI which allows selection of an automated function or a file describing a set of these, plus a field stating how many cycles you wish for it to run.
Hello, Kobi!
Thanks for sharing your thoughts.
“All that is written above is great – but you haven’t answered the “Missing Link” issue – as most companies don’t either…”
I object this statement and I don’t see argumentation for it. Have you read “The solution – build the missing link” part of the article? If yes, what in it makes you think I failed to answer the missing link quesiton? I agree it is not exhaustive solution, it might have some gaps, but it is some start.
“becoming what I call – “Super Testers”” Terms like super testers gives me the creeps, I’ve seen the effect on the community from such buzz words and my advice is to be careful when using it. There’s nothing super in using tools, it’s practically just software literacy.
I don’t think the UI barely touches the solution as it is again operating in the area of execuation and aplicaiton and not in the area of strategy, goals, design and proper allignment of testing and automation effort.
Hi Viktor,
Beautifully worded and succinct in many ways. As an automator and a tester I’ve fought daily for the last few years for the separation of the two (also we need to get the name/word automator more widely accepted, QA automator sounds far better than automation tester). I abhore the oft muddied waters that I end up working in, and quite frankly we don’t need it. This most definitely the symptom of the ‘just a tester’ mentality in many software teams, being seen as the infinite number of monkeys trying to break their software, which hasn’t been helped by the various programming fads in recent years, of test in dev and trying to make every tester a developer and every developer a tester, a good analogy would be your universal soldier.
It’s not impossible for an automator to be a damned good tester, or for a tester to be a damned good automator, hell I would consider myself a healthy bridge across the void between them, but my automation tasks are complex programs and my testing tends be built around simple tests with simple steps, and a big part of that is because; if you want more than a sharp yes/no answer from your automation tasks you need to build yourself a complex reporting feature. We have automation tools out there, but they are all built on the assumption that you want a simple yes/no answer, and worse than that a lot of them are visual recording devices that fall over if an anomalous dialogue box or screensaver pops up, or if you hit any kind of resource race, causing a response to take too long. This is why we require some kind of intelligence (at least we hope so) behind the wheel.
While machine learning is becoming a bigger thing, and automators would definitely benefit from a spell working with AI and even learning from pathfinding and obstacle detection from rovers and video games (I have always insisted on hiring testers with a passing aquaintance with video games and the various bugs that slip through) and this might provide the missing link between the two separate worlds of testing and automation, but as the industry is at the moment we simply do not have either the knowledge or resources to do it while testing is seen as a necessary evil in most teams.
I think the first step is for testers and automators to start pushing their importance, celebrate your wins, and I mean in a big way, push the point when you find a game breaker, and we need clear off the bullsh*t buzzword accumulation that test teams have become, here in the UK I have seen the offered salary of testers rise quickly, but I feel this is more because of an increase in demand rather than an increase in the perceived value of testers, I could still easily earn more as a developer than as an automator or a tester, that said I love doing what I do, I find it infinitely more rewarding than programming.
I’m not *just* a tester, I’m a *tester*!
This has been a bit of a disjointed ramble in the woods, I’m afraid, so a tl;dr would be: Both testers and automators need to celebrate their skill and we need to move away from the mentality that testing is just something we need to have. We are effectively quality consultants for our respective development teams and we shouldn’t forget that acutally, no, this job cannot be done by just anyone, not if you want it done properly.
Ady
Thanks for that opinion, Ady!
I share that view point completely and I often write about it – the bad reputation of testing is mainly, because of the the bad “advocating” we do for it.
Beste Mr.Slavchev,
I published many articles about testing on ITpedia. See https://www.itpedia.nl/tag/testen/ . But this insight is not there yet. Can I share it with the ITpedia visitors? Best Read, Wim
Hello, Wim!
Sorry for the delay.
I don’t mind if you share or translate any of my articles as long as:
– they contain link to the original article in my blog
– they state clearly the origin and the author
– you don’t make any changes in the content
Best regards,
Viktor