In this part of hindsight lessons and as a natural follow-up of the previous part I will continue with the question what tests to automate, what are worth being automated or what are the possible candidates to be automated.
The underlying idea of these posts is to be honest, based on experience and bullshit-free. I can easily draw thousands to read my blog If I just repeat all the nonsense that are out there, but I will not, I have my principles. This makes me say…
What tests to automate is the wrong question to ask
Anyone with expertise in testing for more than 6 months knows that we are not, can not and will not “automate testing”. What we do, in fact, is to automate fraction of testing execution and it is a tiny one compared to the spectrum of activities we perform during testing. I think I spent enough posts to explain that automation is not human testing turned into code and that these are two separate and very distinct types of testing, aiming for specific goals, that are not interchangeable, but complimentary to each other. (If you missed them see: Bitter truths about automation, The non-manual un-automated tester)
Having that said, it will be misguiding to speak of “what tests to automate” as it will imply, that we are automating what we are already performing as human testers.
Urban legends
There’s a good amount of common and mostly wrong advises what is automatable. Here’s some of them with my explanation why they are dangerous:
- “Automate the boring part of testing”
There is no such thing as “the boring part of testing”. If you are being bored by testing, why are you working in testing? You are simply lacking intrinsic motivation and that is something no tool or framework will solve for you.
Testing is quest for discovery – if you are not discovering anything new of value, because you iterate over the same routines, then change them. You can enrich your testing by researching and adding these concepts into your testing:- Risk based testing
- Test framing
- Session based testing
- Testing tours
- Testing heuristics
- “Automate the repeatable work”
This again, has an underlying problem – why are you repeating tests? Is this because you want to check that the result of an action is consistent after n iterations or because you have no fucking idea what you are doing?
This advice is a generalization of a problem that has different dimensions. Repetition is easy to automate, but it’s the information that we get out of it, that should be leading our automation effort. - “Automate your regression”
This again is in the repetition domain and unfortunately according to many testers and devs, regression testing means run tests over and over – automatically or repeating testing procedures, even worse – repeating them blindly without considering context. Problem is a regression bug is not necessarily a bug that we already know about and have a test about, regression bugs are not bugs who “rose from their graves”. Therefore, repeating tests to find regression bugs is like throwing bunches in the dark to hit a mosquito.
We should be aware of the limitations of repetitive checks – they will check what we already know, they will rarely signal for unexpected issue, but this will be mostly based on luck.
The right question is – how can I help testing?
If we move away from the urban legends, the right purpose to automate a test is to help yourself or your fellow testers with a job that is difficult to test by a human. So, how do you know what is bugging your colleagues? Well, go ask them. It’s all about collaboration. In the missing link between testing and automation article, I was talking that we don’t necessarily have to turn everyone into an exploratory/automation tester, but it will definitely help if we make testers and automators to talk to each other and share how they can be useful to each other. There’s tons of small automatable tasks that automators can improve and create tools for, to make their fellow testers’ lives easier:
- Creation of test data
- Automated environment setup
- Automated deploys
- Improving testability of the product
- Forcing prerequisite conditions in the product
As you see, all of these barely has to do something with performing testing itself, but it brings a lot of value to testing.
What about tests?
I said automation is useful for testing right, so how?
To make sure we are on the same page I’d like to focus on couple of variables that your checks have, that will either make them good or bad candidate for automation, and again by automation I mean automating simplistic actions in the system, if you want to automate end to end and scenarios and flows, you are in trouble, already.
Variables:
- Information gained
The most important aspect of a test – the information that the check will provide and mostly, information that it will provide when it fails. Is it the right thing to focus our check for or we are just performing demonstration of a functionality. I like the concept that a check should be like a trip-wire trap, we put in a key spot and it should make a lot of noise when triggered. - Investment of time money and effort
This one is straight-forward – is the test worth the effort? How much time is it going to take. This variable isn’t as important by itself as it is important compared to information gained. That’s why it’s plain weird for me that people spend time creating frameworks without having anything running that brings any benefit to them. Frameworks are an awesome thing to have, but they are part of the natural evolution of the code, if you try to build them upfront, you will mostly fail, but I prefer to expand this in a separate topic. - Short term value
Short term value is the one that we are getting during and right after creation of an automated check. This is where the huge difference between turning scripts into code vs. serving to help testing and exploring comes into the game. If we simply turn script/test cases into code, we convert one type of procedure to another type of procedure. If we explore, we gain tons of short-term useful information that help us create more mindful tests and improve our test code. For example, while exploring the application I am gaining tons of useful information about problems with docs, bugs, missing features, inconsistencies with design and user story. - Long term value
One question – did you see your tests failing? If not why, how are you sure they are working? There’s a huge difference between aggressive tests that do anything possible do expose a bug, and such that are modest demonstration, rather than a test. Long term value is provided by tests that will fail over and over and over, unless they make the developer fix their code. So, important question is – will that check fail? - Stability
Last, but not one of less importance is the stability of our check and its capability to provide consistent reliable feedback. Again, it is not as important on itself, although it is very important, as it is in relation to information gained and investment. If we have a check that we invested a ton of hours into and it’s not stable, the information gain is unreliable.
Visualization of these
How does a bad test look represented with these variables?
Bad test
Good example for a bad test are the ones I called a modest demonstration – such tests normally take a lot of time to develop, because they represent long scenarios, switching screens etc. They have some short-term value, but long-term it’s almost none, as they are not aggressive enough to fail. Information gained is almost none, but their stability is rocking, because they will never fail, they are not given the chance to fail.
Flaky test
Flaky tests are absolutely the worst, related to these variables. Investment in such checks is enormous as we are constantly trying to make the fucker work and it yet fails, they have 0 stability and they are totally unreliable, very small long-term value, because the results are totally random.
it’s important to say that “flaky test” is another misnomer and a failed attempt to redirect responsibility from the design to the tool. Flaky test is indication for terrible test design. How to deal with it? Simply delete the mother fucker, again – value and information gained over number of checks, it doesn’t provide information, it’s useless. You will also like to take a look at your testing strategy and find out where the “flaky logic” came into existence.
Good test
Good tests are mainly recognizable by the good proportion of a lot of information gained, for small amount of time and effort investment. When your framework grows and matures, you will get rid of code repetition and useless coupling, and this will allow you to write tests fast and get them running quickly. Anyway, this is an effort that gets better in time, that’s why good tests pay much better in long-term, rather than in short.
As you see I didn’t give any specific tests or scenarios or situations that are good to automate, because that would be silly. Good automation, just as testing is highly dependable on context, resources, expertise, coding skills and testing skills. I will be selling snake oil if I try to make you believe that specific scenario is the best candidate for automation. That’s why I decided to try to provide tools for you to do the evaluation on your own. Hope it helps.
Thanks for reading. 🙂
1 thought on “Hindsight lessons about automation: What tests to automate.”