The first topic of the series will be how to start automating. As I gave a promise to myself, I will put some effort in clarifying, mainly to myself and whoever wants to read, knowledge of automation as a specifically targeted activity towards the goals of automation and in support of testing.
So, this series of blog posts aim to be:
- Short, about 5 mins reads, although that’s very hard for me.
- Practical – no bullshit, just empirical, productive advice
- Based on personal experience and bad decisions. I think that helps s lot.
Last summer, I read a book called “Thinking fast and slow” by Daniel Kahneman, it’s a psychology book and a really good one, if you have a chance definitely read it. I learned a new word from the book and it was “hindsight” it was used in a bias called “hindsight bias”. Hindsight bias occurs normally in a negative aspect whenever something bad happens, there’s always that one person saying: “I knew it, I knew it would happen.” Turns out, he or she didn’t knew, it’s just a bias, but more on psychology – read Kahneman’s book.
So, what does this have to do with testing?
Once I had a discussion with another tester about a talk I gave at a conference called “Test automation – the bitter truth”. He asked what was it all about. I answered it was about people’s expectations about automation and how we should approach it with realistically, not treat it as it was a magic. His answer was: “Well isn’t this obvious?” Well, turns out it’s not.
Another example – Maaret Pyhäjärvi made an observation, while we talked at a conference few months ago – very often the things we consider common sense are not so common for new comers in our craft and this applies to automation, too.
So, turns out, there’s some layer of knowledge we all think we know and share, but we still fail miserably to apply it. That’s why I decided to start with basic stuff.
Formal definition of hindsight.
Recognition of the realities, possibilities of events, situations,decisions etc.. after they have occurred
Or in other words – we all fucked up doing automation, we had to fuck up over and over again to learn a lesson. It will be useful if we share this lessons, so we don’t have to be the one saying “I should have known”.
Anyway – there’s even better definition of “hindsight” in urban dictionary:
After you screw something up, you magically and conveniently gain a vast amount of knowledge on what you SHOULD have done to avoid screwing up.
… and sounds very familiar to me unless I read the 3rd definition:
The universe rubbing your face in your own fuck up.
Isn’t that lovely, isn’t that “the story of my life” in everyone’s testing? 😀
So, the inspiration for this blog post series is – what valuable lessons have we learned and can we transfer, after we got our faces rubbed in our own fuck up? 🙂
Last before I dive into concrete stuff, I assume few things:
- You’ve read this article first, otherwise you kind of miss the context.
- You are aware there’s no such thing as “manual” testing.
- You are aware automation supports testing, but can’t replace it.
- You figured out the right reasons to do automation or the right reasons to do testing, using automation as a support.
Where to start automating?
[Note: This particular topic might be a bit boring for experienced automators, but I hope you understand why I write about it.]
Although it might seem like a lame question, a basic one to people with experience in automation, I really want to clear it out. The answer is really simple – programming.
In order to think of dealing with automation, you should have clear idea in your head about the following:
- Automation is programming
- Automation code is a program
- You are a coder
- Professional automation is written in code
- Automation has bugs, always!
- Most of the time you will be dealing with bugs in your code, rather than bugs in production code.
I know there are tons of tools that offer alternative to coded testing via any sort of record and playback, support frameworks, visual testing, etc.
Unfortunately, when somebody speaks of automation developer or software developer in test, nobody means “record and replay” developer.
Something that is probably obvious for anyone in testing and automation, but totally not transparent to new people is – automation community is sort of layered. Whenever you speak of automation, people will expect you to mean “coded automation”. If you want to just use record and replay tools, you will be treated as second class citizen in the automation community.
The reason for this is that in time record and replay tools didn’t prove reliable in the long-term. They are really easy to use for short and fast tests, but they are brittle and hard to support when project scales.
Another reason why record and replay tools are not desirable step of your automation career is because they inherently support the false claim that we can automate human activities by simply recording and repeating them and may be that’s why they failed (Yoda citation intended).
Get your testing hat off and put your developer hat on.
As you see in the article so far, I am avoiding the use of terms “test automation” and “automated testing” and I will continue to avoid them, on purpose. The reason why automation is in trouble, I think, is mainly because of these terms. What I prefer speaking of is “programmatic testing”. I offered this term once, but it didn’t really sunk, that’s why I decided to forget about it. What I mean is – being an automation developer you are much more developer than being a tester. What you do, writing automation, in fact is performing testing by programmatic approaches. Now, I think when we speak of “programmatic testing” or “performing testing by using programmatic approach”, something slightly changes in the way we think about the activity compared with the term “automated testing”.
When we speak of “programmatic testing” we kind of realize the limits that we are facing by using coded approach in testing, because every programming framework, language, API or library has its limitation, and that’s totally OK.
Of course, using programming languages we can do almost everything, but considering the time limits we face, the quality criteria, the limitations of tools and frameworks, the variability of behaviors and the vastness of conditions that we have to reproduce, performing our tests, trying to achieve “almost everything” isn’t so much of a good idea, anymore.
That’s why I brought the idea about the two mind sets in the previous article – there’s simply difference in the approach we use being a human exploratory tester and being a programmatic tester.
So, I conclude, people willing to do automation are dealing with it, because they like solving testing problems in programmatic way.
If you are trying automation, because you think it will replace testing, you probably don’t have good understanding about testing.
If you are into automation simply, because of a better salary, you probably lack the proper motivation to do it.
Again, success in automation, according to my humble opinion is – to do it because you love it and because you like solving testing problems programatically.
What language to start with?
This is a big and complex topic and the more people you ask about it, the more biased answers you’ll get.
Simply put, there’s automation framework for almost any programming language that’s out there. If you have experience in a language and you like it, go for it. The main thing in programming and the main reason why people have “religious fights” over programming languages is – you stick with the language/framework you like and find comfortable using. That’s why people fight over languages, because they fail to realize they simply like the language, there’s no one ultimate supreme feature that their favorite language has over others.
If you don’t have any idea where to start and don’t know any programming language, my advice is start with Java. Statistically, demand looks higher for Java automators, rather than any other kind. The good thing with programming languages is, once acquired the basics, you can easily learn another, as it’s simply differences in syntax, strict or loose type and compiled vs interpreted languages.
Also, if you want to do automation, you will need basic knowledge in:
- HTML5, CSS and JS – simply because (sigh) sooner or later you will be digging in the UI, so better be prepared for it. Good news is, HTML and Css are fairly easy to learn, if you don’t want to make too complex stuff. Not the case with JS, though, but I won’t spoil your pleasure. 😀
- Databases – not only because of the interview questions, but database knowledge is needed when you have to deal with data management for your tests.
- Linux + shell scripting – not that it’ll be expected of you to have an admin skill set, but being a developer you are expected to be able to set up your own environment. Mostly it will run/have a server on a Linux machine. So, it’s way better if you can deal with it, before it turns into a huge pain in the ass.
- Web services and API knowledge – almost in any case, you will be required to use, integrate with a web service or at least, know how it is used by the application. It’s good idea to be aware what a service is, how it is used by the application, what possible methods there is in an API, what file formats could be used and how we can use or produce them, with a programming language.
I know this is probably too basic, but I humbly hope it might help some poor soul just stepping in our craft to get a little bit of a clue what they are dealing with from a meaningful perspective and not from all the bullshit out there.
Stay with me for the next chapter where I’ll focus on what types of automation are there. Until then – comments, shares and productive opinions are highly appreciated.
Thanks for reading. 🙂
6 thoughts on “Hindsight lessons about automation: Start automating”
True, to a degree regarding programming as a starting point for automation. It’s a form of programming, we build “Testware”. Software that drives the “execution” of a specific scenario using a set of data. But once you come to that realization you need to take a step back and think about what are all the things you need to do before you write a line of code. That’s really where we need to start regarding automation; answer the questions of Who, What, When, Where, Why and How. In my experience (and opinion) first answer the question of Why, second What, third How, fourth Who, fifth When and last Where. It would take a while to explain all of this, and I don’t think I’ve got the time and space here to do this. But think about those questions before you get a tool and write a line of code. You will save yourself a lot of headaches and the mistakes that cause them.
Thanks for this addition, Jim! I couldn’t agree more. I do consider covering on most of these lately, but considering the “why” earlier was a good reminder for me.
Thanks for sharing your thoughts.
As for Katalon, UFT and the other tools you mentioned, they have place of course. They are tools – we use any kind of tools – automation is a tool, heuristics are tools, oracles are tools. Problem with complex tools is we sometimes get so excited with them that we focus on the tools and not on the task we aim to achieve. You will see this in automation a lot.
As for the developer role – I don’t know if you find some sort of negative meaning in the role developer. But here are some questions to ask yourself:
– Do you write code?
– Do you enchance that code?
– Do you support it by removing defects in it?
– Did you do it in order to help resolve a problem?
If answer to all these questions is “yes”, then yes, you are a developer. And there’s nothing wrong with that. Because what is a developer – isn’t that a person that writes code in order to provide solution to a problem? Then what do we do as automation developers? We write code to provide solution to a testing problem?
See, automation in testing is a weird hybrid, interdisciplinarian craft. I think Angie Joens said something like: “Test automation is to have the tester mindset combined with developer skill set”, but I do think it’s more beneficial if we treat it like a dev activity with a testing goal, rather then a testing activity with programmatic approach.
I think, based on my experience, writing code and testing products, we are capable of numbing some of our testers senses when we develop, that why we should approach automaiton with some healthy scepticism.
I’m with Viktor on this front and believe that if you do indeed create and maintain any kind of code, you are welcome to call yourself a developer. You’re highlighting, at least I think so, a problem I see regularly in the industry, and that’s this ‘us and them’ mentality. Automators end up being developers, and I feel to large extent so do testers, at least those who take an interest in the code behind the bug. I have become so entwined in some projects that I actually worked as an ad hoc developer for them, well, I knew the codebase, so why not? It also helps get rid of the animosity between testers and developers.
Record and play are great ways to become aquainted with the system, certainly, and I’d warrant that the majority of automators started with Selenium IDE or something similar, the problem with them is that they have very large and very obvious limitations, whether it’s being stuck to a single browser, the inability to run several scripts at once or the lack of reporting, most serious automators move on from record and play very quickly. It does what it does, and while it does it well it is still just a record and play automation tool.
I’ve worked with tools like QTP or UFT before and again, they have their limitations, but they still ultimately acheive a goal, and sometimes the goal is to automate a single task, which it does well. I personally do not like record and play, or the numerous visual based test suites, mostly because I started my career as a developer, but I also find them inflexible and unreliable for anything beyond a handful of steps.
In answer to Paul’s post. Yes, you are a “developer”. You are an “Automation Developer”, you are programming the tool to do its work. You are building “Testware” to drive the execution of test/checks against the “Software” Under Test (SUT). So yes, you are a hybrid. A combination of Tester and Developer. Automation is its own form of software development, and as such there are certain processes and rules we need to follow in order to make sure our testware is maintainable, reusable and robust.
Even in standard application development there are different types of “Developers”. There are people who focus on the UI, or internals, or communication, or database, etc. They each have a niche they fill. We, as Automation Developers, fill a specific niche.