If there’s a contest for the buzzword of the century, although I don’t think testing was around for a century, yet, I am almost 100% sure it will be test automation, or automated testing. If you read or listen to test talks it’s “automation this”, “automation that”, “automation everything”.
It’s definitely over used and over abused and I bet large number of the people giving advises on it have no fucking clue what they are talking about. Or at least the statements produced by them show absolute lack of understanding about the purpose of automation. Then I said, wait a minute …
It was discussed plenty of times that the terms “test automation” and “automation/ed testing” are misnomers in many ways, that we can’t actually “automate” testing, that we only can automate subsets of it, yada, yada, yada I won’t repeat this, as I am getting really bored having the same conversation over and over again. The thing that came and struck me as a lightning was that actually – there is no “testing automation”, there is “testing” and “automation”. Seems that our thinking hadn’t evolved enough yet, so we can combine both in an efficient way. Let me demonstrate.
Two different mindsets.
I got this observation somewhere around preparations about Testbash Munich and it started to grow and evolve and I needed time to give it shape in my mind.
Whenever you as a tester are at a “test automation” conference or you as automation expert are at testing conference – don’t you feel that weird, feeling that people are talking in a different language?
Because that’s how I feel – every time an automation person starts speaking about “flaky tests”, I am like “c’mon, man, there’s no flaky tests, there’s just bad test design! When are you going to learn it? There was plenty of discussions about it, where were you?”
I bet test automators are also being puzzled every time they listen to talks or conversations going deeper in teamwork and critical thinking, bias, heuristics and are probably thinking – “what the fuck are these guys talking about? Don’t they know how many tools and technologies there is? It’s all code, why all the drama?”.
The bottom line that I am trying to get to here, is also the answer to the question – why can’t we create automation that really helps testing and the answer is – because we have initial misunderstanding of the goals of testing and automation and no possible transparency of what problems the other “camp” is trying to solve.
From my point of view, testing and automation are simply talking in different languages and aligned towards different goals, which is the reason why there’s initial misunderstanding in the communication between both.
The testing mindset.
In order to figure out how we fail to communicate, let’s first see what are our goals. These are just my observations, probably biased in a way, so don’t treat them as a recipe.
What problems testers are trying to solve:
- In short-term – provide information about existing problems and assist with their resolution.
- In long-term – provide information about evaluation of quality and support making an informed decision.
- Perform exploration in order to understand the product or system under test
- Perform experiments(this is what “tests” are, in fact) with the product in order to gain profound understanding and information about it, its state, behavior existing problems etc.
- Trying to conduct experiments in an incremental manner so that every next session gives us better information, new information, or such of different aspect – functional, parafunctional, performance, security, etc.
- Evaluate risks and their impact on quality.
- Build understanding about products supposed working and quality criteria and use them as oracles.
- Trying to challenge the stability of the product by any means possible.
Where does testing mindset fail to assist automation?
Normally the testing mindset fails to assist automation in a few different ways.
“The testing resistance”
In a way our efforts in testing turned into the “testing resistance” as we are able to almost always critique automation and point out what it is incapable of doing, but not very often to demonstrate the strong benefits of automation in testing. I also realize my guilt in this one, because I also did it in “Bitter truths about automation”, and said nothing to support automation’s usefulness, but I will try to fix it.
I believe the “manual/automation” stupidity has to do a lot here too and all the bullshit around “being replaced by machines” and other ways people try to look clever, by speaking sci-fi stuff. May be it’s about time to get a realistic view and try to build a solution, instead of just naming problems.
Failing to address good automation aspects of testing
Another way testing fails to help automation, is by trying to squeeze all the complexity in testing and convert it into bytes.
Either because of the “replace the manual morons with scripts” fallacy or because we are aware of testings real value and complexity and want to demonstrate it, we tend to give almost impossible tasks to code to our automation engineers.
The thing we should realize here in order to improve is – testing and automation are good for specific problems, which often don’t mix. The question is not one or another, it’s “how both can co-exist in symbiosis”.
The automation mindset
This mindset is way different from the testing one. People involved in automation have much more a development oriented thinking rather than a testing one. I don’t say it’s bad, it’s simply different. And we should appreciate it, because some automation developers are brilliant coders.
What problems does automation try to solve?
When used in the right manner…
- Functional correctness
- Precision checks
- Low level checks
- Providing feedback in fast and consistent manner
- Asserting definitive conditions
- Isolate variable conditions from tests
- Provide consistent execution every time, over and over again
- Solving all types of programmatic problems such as – code sanity, bugs in testing code, complexity, re-usability, refactoring, etc.
Now having both, we can see that automation mind-set is much more technically and algorithmically oriented, while the testing mindset is mostly oriented towards unpredictability and critical approach. While testers try to experiment in an incremental way every time, to push harder in order to gain new knowledge, automators are trying to do the opposite – make sure tests are executed in the same environment, under the same conditions, every time. This is where a lot of us fail to realize testing and automation work in a different way and are not irreplaceable.
Where automation mindset fails to help testing?
The tools cult
We see that automation is very technical, very tool oriented, but this becomes problematic whenever automation engineers start treating tools as a solution or blame them for their own test design mistakes. In fact, many talks on automation dig so deeper on the “hows” of doing it, that they totally forget the “whys”. The result of this, is automation starts to feel totally detached from testing and again, may be “automation vs manual” and “replacing” one with the other talks have to be blamed for this.
But let’s not fool ourselves, the meaning and purpose of automation, and I am really happy I saw Bas Dijkstra got to the same conclusion in a brilliant article, few days ago – the main purpose of automation is to support testing.
No matter how robust is your framework, if you followed all the OOP principles, if it’s easily extendable and has loose coupling and strong cohesion, if your tests run within nanoseconds and manage test data in a perfect manner – all of this, is absolutely useless if your automation doesn’t uncover problems. If it doesn’t challenge the system aggressively enough to expose bugs. It might be a perfect piece of software, although perfection is utopic, rather than something achievable, it can be really masterpiece of code, but it won’t serve it’s purpose unless it provides meaningful, reviewable information about existing problems in the product.
How is this possible? We get to problem 2…
The belief of “what we do as testing is enough”
Because of the cult to tools, the will to replace testing, rather than support it and the focus on how to perform automation, most automation solutions are merely a demonstration, rather than experiments. Go to an automation conference, read an automation blog post – have you ever seen someone ask themselves “Am I testing the right thing right now?” or “Is this the best experiment I can produce with the tool I am currently using?” or “what is the quality of my tests”? I bet the answer is “no”. Very often automators get so busy mimicking human behavior, expanding menus, interacting with elements, transitioning between pages, that they forget the main focus that their code should have – to be a high quality test of certain condition and make sure the condition itself is worth testing.
Instead, I see everyone doing automation treating test design just like some fossil from the past that we figured out long ago. And it’s wrong, conducting meaningful experiments is the main goal for both – testing and automation.
The solution – build the missing link
Well, this is hardly a solution, I am talking to myself here, after all, but what I see as a solution here is the realization of few important things.
Testing and automation don’t mix.
Let’s cut the bullshit with “replacing” and “if”, and “when” and “manual” and try to do something productive, shall we?
As we see, goals of testing and automation try to solve different problems, if we treat them as mutually replaceable, we won’t gain any progress.
There’s no “universal soldiers”
In my humble opinion, there’s no one person with complex enough understanding about testing and complex understanding of automation and there’s nothing wrong with that. No need to try to ask testers to learn to code and write automation or ask automators to learn critical thinking and overcomplicate their code. Just stick with what you are happy doing and keep yourself open for discussion on how to be useful to testing the product or SUT.
Try to make your problems and goals transparent to others.
As testers it will be useful if we make an effort to explain where automation is needed and provide examples of where automation makes perfect sense and is helpful. This, of course, with the clear mind that, it does not and should not “take the effort from us”.
For automators, I believe it will be useful if effort is made to explain the technological limitations that our tools and approaches have, the eventual pitfalls of using one or another the time effort and the complexity involved.
Also, a “tea-spoon” of healthy skepticism will be useful, when writing automated scripts. One of the core components of the testing mentality is doubting in your own ability to perform good testing. Therefore, whenever writing automated scripts we should challenge their qualities in first place and not settle with simple confirmatory testing.
How I will try to solve the problem for myself?
As I said, I have “a debt” to provide information on how to use automation in a way productive to testing, so I decided to start a series of short posts, targeting answering practicle questions about automation that are obvious, but yet not often spoken of. I still haven’t come up with a heading, but I will.
Am I the right person to write about it? Who cares?! I have a blog, I will write for whatever I want, if people like it – they will show it, if they don’t – I will stop. 😀
I would love to read and discuss with you about your solutions of building the missing link. Any shares and retweets are very welcome. Thanks for reading. 🙂