In case I never stated it clearly enough on the pages of my blog, because I am normally saying this when I speak at conferences – I absolutely disagree with the existence and the usefulness of the term “manual testing”. I truly believe testing has never been manual, nor it will ever become. I am ready to go further – I will challenge anyone defending the usefulness of the term manual testing to share why does he or she thinks manual testing actually helps describing testing better. Here are my thoughts.
What’s the problem with manual testing?
The first problem I see with the term is that testing was never manual. Testing was never an activity which included executing action in repeatable, excluding thinking, simplistic manner just like someone working on a production line where same actions are performed over and over and over again. Testing is referred to as “manual testing” mainly from tool vendors who want to make it look dumb, so they can sell you tools to make it easier (both claims are lies), people who are not involved in testing or people who don’t really care what they talk.
The other problem that I see related to this is people trying to “fit” in the description “manual testing” and trying to perform testing in a unified unnecessarily formal way so they fit into the label they were given. I think I wrote something about it in the article The non-manual, unautomated tester, not too long ago.
Why “manual testing” is invalid term?
Testing can never be simply manual. The reason for this is – it includes way too much thinking and brain work. If I have to paraphrase a famous saying – if 100 “monkeys” tap on keyboards and click around your product for 100 years, they won’t produce high quality testing service, no matter what low-cost, outsourcing companies claim.
Testing includes a lot of thinking and brain work, such as, but not limited to:
- Modeling – the ability to “imagine” what the product is like and how its inner workings are arranged
- Critical thinking – the ability to construct theories about the product and challenge them to prove them wrong.
- Social knowledge – software is not simply code, it’s something serving to human beings as a bridge, so it has to work in specific social context.
- Risk mitigation – thinking about what can fuck up, before it’s too late.
- Cognitive psychology knowledge – knowing your own thinking patterns and how they can fool you, knowledge of bias.
- Metacognition – or thinking about your thinking, playing the spectator role on your own approach strategy or thinking, so you evaluate the efficiency of your own methods.
- Logic – being able to derive conclusions from clues using deductive, inductive and abductive reasoning.
- Epistemology – explaining yourself – how to you know what you know, how are you sure a statement about software is true, how do you know software has to work in a certain way?
- Creative thinking – the ability to produce new ways and strategies to expose problems
Now, don’t worry, you mostly use these without even knowing they are called like this or that they exist. Terms are put there just to help explain what we do.
Note: If you want a deeper dive in these, please refer to the “What software testing is…” series on my blog. I know they are not complete, but they will be, soon.
What I really want to emphasize here is sort of black swan conclusion – the fact that you never seen black swan doesn’t mean there are no black swans, therefore the fact that the mass of testing resources don’t speak about thinking in testing, doesn’t necessarily mean there’s no thinking in testing or that we can label it “manual testing”.
So, testing is an activity related to thinking and cognition, if you want to refer to a body part to describe it, call it “brainial testing”. Sounds ridiculous, isn’t it? Now revisit the term “manual testing” and tell me how does that make any sense, ffs!?
Why people like the term “manual testing”?
If I have to be brutally honest, I will cite this article from Scott Berkun:
People love easy, binary models for things and take pride in basing our primitive notions on the pretense of science.
And this is why the manual/automation bullshit lived for so long, for the only reason that testers prefer to have the simple, but wrong binary division, instead of diving into thinking and come up with a better answer.
What people really mean when saying “manual testing”?
According to my observations, when most testers speak of “manual testing” they normally mean “testing performed in a non-programmatic way” or in other words – testing in which we don’t write code to perform the testing, but do it ourselves. Anyway that doesn’t turn testing into manual activity, so I’d suggest to repair this “defect” and call it “human testing”, rather than manual. I think it gives a little bit more clarity to the term and what is involved in it.
Calling “manual testing” exploratory, doesn’t really help.
This is something that I mostly see among people involved in coded testing e.g. automators, but I see it spreading.
Whenever someone is showing the automation pyramid (which I will address in my Hindsight automation lessons, soon) they normally show that “cloud” on the top and refer to it as “exploratory testing”. When I go to conferences and see companies present, they again speak of testing being “executed” with this percent of automation and this percent of exploratory.
Well, I guess the community finally got it when we react in an allergic-like manner to the term “manual testing”, so they probably decided to call it exploratory, so it sound “cooler”. Well, this doesn’t really help, when they mean the same old characteristics they meant for “manual testing”. On the opposite, this in fact distorts the meaning of the term exploratory testing, giving it attributes it was never meant to have.
As a conclusion…
What I very often see is that the term “manual testing” is normally used as sort of self-explanatory theorem to derive conclusions (mostly wrong) about testing. Just like, testing is manual therefore it’s easy or testing is manual, therefore we can make it more efficient with automation. This all show a shallow understanding about testing and the software domain in general.
What “manual testing” really turned into is a synonym of “very low quality, low expertise human checking”. And I am ready to bet that no sane tester really wants to be labeled this way.
So my call for action is – stop calling testing “manual”, it is not manual, never was and it won’t become.
To those who disagree with me – my challenge is open – in the comments, give me your best argument why the term “manual testing” is useful on describing the testing role?
Aside from that, any comments, shares and retweets are highly appreciated. Thanks for reading! 😉
Mr.Slavchev
Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.
Follow Me:
I think I’m keeping this one for future sharing.
Thank you
Thanks, Amit!
How do you feel about the use of manual to denote executed/tested by people using exploratory techniques themselves versus the use of a computer to complete the test itself?
e.g. the use of selenium to test a website would be automated and the use of exploratory testing to denote manual? Or is this fitting into your bucket of too binary? If so, what would you use to describe these two scenarios?
Hello, Global App Testing!
It does not fit in my bucket of too binary, but it does fit in the bucket of “why not name things with their real names?”. 🙂
When people speak of automation, they normally mean coded testing, Jim Hazen, in a comment on another blog post on this blog, called it “testware”, meaning software written to test software. In time this proven to be the industry standard for automation. But I’d rather describe it as “coded testing” or “programmatic testing”.
If we want to mean exploratory, human tests, why not call them exploratory, human tests? Why would we have to “denote” to this using “manual”. Don’t you feel referring to testing as “manual” is kind of oversimplifying testing to just test execution. Trying to denote to something brings some contextual knowledge that has to be transferred, other wise listeners will assume the meaning themselves, which in the case of “manual testing” is, unfortunately, clicking around following a script.
Also, I don’t like to dive into the manual/automation dichotomy, the challenge here was – try to explain what does “manual” bring to the meaning of testing, how manual explains what you do better?
That’s exactly how also I see it, thanks for that! We as tester don’t test “manuals”, we test software in very different and thoughtful ways.
Manual testing simply means that it is not automated and cannot be run by machines on demand. Once a human tester fully understands the problem space, the testing requirements can be codified to make it easy to perform on demand as regression tests by a machine. Humans are bad at doing the same thing over and over, machines are good at it. Automated testing is the final form of really good human testing.
Hello, Abhi!
Thanks for your comment, because I think you share the most popular view point, which exactly I disagree with in many ways. Here are some of them:
“Manual testing simply means that it is not automated and cannot be run by machines on demand.”
Yes, I wrote that above, but what I was asking for is not definition of manual testing, but rather how the manual part helps explain the nature of testing better, in that particular case?
“Once a human tester fully understands the problem space, the testing requirements can be codified to make it easy to perform on demand…”
The problem is we are not only testing against requirements, sometimes we even don’t have requirements, or requirements are outdated, what do we do then? Surrender?
We also perform tests against (often) unwritten parafunctional requirements – for instance – does our client has to state it explicitly that it’s not OK to wait for a site to load for 1 min? No, we know this as a domain standard. Same for reliability, usability, security and so on.
Requirements are just one of the testing oracles, as we call them, which we use to identify correct behavior from wrong.
What you focus on here is test execution and testing is not simply executing tests, we have modeling included, critical thinking, design, evaluation, review, learning phase and many more.
“…can be codified to make it easy to perform on demand as regression tests by a machine”
The ultimate goal of a test is not to become a regression test, regression tests make sense whenever we are looking for regression bugs and they have a limited capacity of checking a condition we already anticipate – meaning, they know only what we told them to expect, they can’t find unanticipated problems.
“Humans are bad at doing the same thing over and over, machines are good at it.”
Why would you consider that “to test” means “to repeat tests”? I guess that’s where many people get confused – testing is not simply executing steps of a script, it includes a lot of creative work. If you are not aware of this, review some of my posts where you can probably find some additional info why this isn’t so. I can not “squeeze” it into a single comment.
“Automated testing is the final form of really good human testing.”
The purpose of human testing is not “to ascend” to an automated test as it’s final form, human testing and automation are good for different purposes and have different goals.
In human testing, we try to perform our testing in diverse way, keeping context and our experience in the equation, trying to look for unanticipated risks, hidden complexity and do this fast and in favor of information about problems concerning quality.
In automation, we try to execute tests in a robust way, consistently, we have constant execution time, constant expected conditions (oracles) and limited discoveries – limited mainly, because we will only find what we asserted there.
Automation and testing are simply targeting different problems, that’s why a productive way would be to combine them, using their most efficient assets, rather than trying to replace one with the other or evolve one into the other. They simply don’t mix.
May be you can find better explanation in this article of mine:
https://mrslavchev.com/2017/11/23/the-missing-link-between-testing-and-automation/
Thanks for reading!
It helped me to name and differ it “testing vs checking”:
http://www.satisfice.com/blog/archives/856 (original from 2009 http://www.developsense.com/blog/2009/08/testing-vs-checking/)
Testing needs intelligence, checking is repeatable and dump.
Quote:
– – –
Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.
Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
– – –
What you don’t like is to name human checking as testing. Humans can do checks and test, but these are too different things. Machines just can check.
Perfect
Great article! I found one article (can’t remember by whom) which coined automation perfectly for me – “Checking” instead of “Testing”.
To me, automation is only good to assist in regression, to ensure that the core functionalities of the existing features are not broken by a new feature. Most of the time, I find it quicker and more effective doing the tests “manually” rather than spending too long to write and maintain automated tests. And most of the bugs/defects could only be found when you test something “manually” as you go on and put on different hats and perspectives. Automated tests will only say yes or no upon what your assertion is.
I must say that my confidence was shattered when management told me that I’m just a common manual tester that can be found easily on the market. And this perception of “automation is the real deal” that is embraced rapidly here at least is one thing that really demotivates me.
Thank you, Hanif!
I don’t think automation is good for regression testing, only. It’s good for multiple activities, but it has certain limitations. Human testing also has limitations.
I think the problem persists mainly because of lack of information for both – the better capabilities of human testing, and the fact that it is not “manual”, but creative activity, and speaking about the limitations of automation. Thinks are normally put in different light – automation being perfect and human testing, called incorrectly “manual”, being bad or outdated.
I am really sorry you had that negative experience, unfortunately, decisions about testing and testers are often being made by non-testers, who mostly don’t care about expertise.
Good luck and thanks for reading.
Your premise is based on the false assumption that “manual” means not requiring thought or brain power, and I wonder where in the world you got that idea. If someone makes the distinction between writing a letter by hand vs with a word processor, would you assume that the one by hand did not include any thinking or reflection? If a plane is flown manually, as opposed to by autopilot, does it mean the (human) pilot is not thinking?
In short: no, I don’t see your argument at all. Manual is an absolutely correct antonym to automated, where the latter is defined as “runs with minimal human intervention” (see the dictionary definition for automation). Manual = run by humans.
I do agree with this part:
“What “manual testing” really turned into is a synonym of “very low quality, low expertise human checking”. And I am ready to bet that no sane tester really wants to be labeled this way.”
But then the solution to people making wrong assumptions about manual testing is not to change the term but to correct those assumptions, as in writing an article about what constitutes good (well thought-out, structured, etc) manual testing.
Hello, Mariana!
Thanks for bringing your view point to the discussion. It would have helped if the second part wasn’t in full contradiction with the first part, though. Anyway, let’s focus on the part we disagree.
It’s is not an assumption, it’s an observation. I very much like your “generic definitions” about “manual” and “automated”, but I am afraid they don’t work for testing. In fact, this is the meaning that a huge part of the CDT community is after, whenever we speak of testing we mean the analytic and critical thinking activity, not the way it is executed, because it doesn’t really matter.
What we can observe in testing, on the other hand is a way that the manual/automation dichotomy twisted the meaning and the reality. If it wasn’t so, how would you explain the existence of job offers that look for “manual testers” and their role will be to “manage and execute manual test scripts”? Why is it so, that whenever someone is discussing performing testing, the “manual way” always has a connotation of old/ineffective/low performance archaic method? Why is it so, that whenever someone speaks of automating something they claim purpose of automation is to automate “tedious, repetitive tasks”? Why do people refer to testing as “tedious repetitive tasks”? Why do people claim that testing is automate-able? Do you still think I am making an assumption or is it the reality in testing that I am trying to represent? It might not conform with our bias, but it’s true.
Yes, you are right – whenever a pilot takes manual control, he doesn’t perform “mindless” piloting, but is he referred to as “manual pilot”, then? Every profession has the option to use high-efficiency automated tools to optimize work, but did you ever hear of “manual surgeon”, or a “manual driver”? What is so different in testing, so we have to make this distinction?
It also useful to mention, “manual” is normally used to distinguish human, unassisted action from automated once, we rarely see it out of the couple, what we should point out, at least in my experience and according to what I see or hear is that “manual testing” is always used as a pejorative.
Another reason why I don’t think it’s a useful term is that it brings all the attention on the way tests are executed, and not on the way how test ideas are designed.
“But then the solution to people making wrong assumptions about manual testing is not to change the term but to correct those assumptions, as in writing an article about what constitutes good (well thought-out, structured, etc) manual testing.”
Great idea, if you write some, please let me know, I would love to take a look at it.
Thanks 🙂
Mariana, I find it a bit difficult to follow your argument here.
One one hand, you insist that “manual testing” does “require brain power”, but on the other, you acknowledge the fact that in the testing sphere, in way too many places, this terms means “very low quality, low expertise human checking” – putting two meanings on a term does create confusion, and as much as all of us would like to have testing with good reputation even without coding – this is not the case.
The term “manual testing” is pure poison – most of the time, it is used for the sole purpose of belittling a task (or a human).
Hiding behind a dictionary definition won’t change this reality – the term carries a lot of negative thinking.
In addition, it is meaningless. Are you testing manuals? are you manually pushing bits around? Testing is almost always tool-assisted (even if this tool is a browser), and writing code is simply another tool.
For the past two days I was comparing several DB entries created by different systems. I did not write software to help with that – but I did use excel and wrote some simple formatting rules that highlighted the differences so that I’d be able to better spot them. would you say I was “manually” testing? Would it be different if I wrote a comparison script that would spew out the differences for my inspection? or if I wrote a complex plsql query to do the same thing?
Claiming that I was “manually testing” tells me only one thing – what I was not doing. I was not writing code in a conventional IDE. It tells me absolutely nothing about what I was actually doing. Did I use sophisticated tools to get my goal in the most efficient way? was I wasting my time doing repetitive boring stuff? was I shallow testing? was I creating intricate models of the system under test?
When we focus on what we don’t do instead of what we do do, we are poisoning the conversation.
I like to make to difference between testing and checking.
I got this from here: http://www.satisfice.com/blog/archives/856
Testing is always manual and needs brainpower.
Checking can be manual as well as automated. Maybe some people are labeling manual checking as manual testing?
My own definition: Testing creates what checking needs.
Creating the checks is test(design). Executing is checking.
Quote:
Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.
Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
Hello, Sebastian!
Thank you kindly for your feedback and the provided opinion.
Yes, correct! And I also state this in the post – “What “manual testing” really turned into is a synonym of “very low quality, low expertise human checking”. And I am ready to bet that no sane tester really wants to be labeled this way.”
Problem appears when somebody treats all testing as checking. I also disagree with this:
“Testing is always manual and needs brainpower.”
Testing is intellectual process of purposely creating variable conditions with specific goal – exposing problems that might affect quality and end user satisfaction. The result of this process are the experiments that we can perform in order to confirm or refute our hypothesis. We do this with or without instruments.
The problem with the term “manual testing” is the focus on the instrument and not on the intellectual process involved.
Hi Victor,
i think your last sentence is the most important one. The intellectual process is often forgotten and testing only treated as checking.
What i meant with ‘manual’ was this intellectual process, which no machine can do (yet?). But machines can support us humans here.
Thats what the qoute is telling.
Sorry for being so vague here.
I think we are on the same side. Thanks to you, i got now a better way to express this.
Actually i got a challenge about that topic.
Does anybody has an idea how tell this differences to some who is not familiar with it?
Hey, Sebastian!
Yes, we are on the same page! 🙂
I think there is, there’s plenty of people out there who can give a good distincion between expert, intelligence involved testing and mere manual following of instructions. I think in order to make a difference, talking has to come second. We need to first demonstrate what we are doing in all its beauty and complexity, but without asking people to cope with it. What they really care for are resutls. So, we need to show well formed, expert level results, once we have ’em others will come asking how we did it, and then we run into explanations.
Seems to me, if we just “preach” listeners will often get bored and tune out. One of the reasons testing is considered low hanging fruit by outsiders, is that besides bugs or test cases, we can’t show anything valuable as a deliverable of the process of testing.
For examples, devs have the code, designers have the designs, PO have the business logic, but when it comes to testing we only show cases and bugs.
In order to expose the expert part of testing we should deliver expert resustls of our job and I personally guarantee this makes difference and you will be asked to explain how you do this.
Hi, Very good article. In my opinion, we need to upgrade ourselves. Automation Testing is also a mandatory technology for every manual test engineer but also we have to be more focus on our manual testing work and concepts.
I think Future technology will bring more automation in the industry but we can not deny the fact that manual testing will always require for the successful project.
Read more: https://softwaretestingboard.com/q2a/2541/
Hello, Ellen!
I am afraid you got it all wrong, The resource you quote is nothing new as far as I am concerned, it’s a just perpetration of already known and in all honesty false thoughts about testing.
I wouldn’t use the term “manual testing” to start with and that’s what this all article is about – “manual” puts too much effort on the way we are executing tests and none on the way we are thinking about testing or designing these tests – this is a dangerous, terrible, harmful oversimplification. To a non-tester – “manual testing” seems like low quallified, low quality, low reward job that somebody just does by repeating test steps over and over and comparing expected to actual conditions. There are people who do that, but far, far away from the concept of expert testing, we have.
Having this in mind – knowing automation is not an upgrade it’s a separate “track” of skills that you can develop, because it has nothing to do with how you perform testing, matter of fact, some automators are the worst testers you’d ever see, but it’s a different way of approaching testing execution, specifically.
The one bit upon which I agree with you, is that we need to refine our thinking from manual OR automated testing to human testin AND automation, because both can’t replace or replicate each other, but they suplement eachother.
Hope that helps.
Thank you for this great article….had to share it on Linkedin.
I’ve just taken over from an Automation Tester but I have picked up a lot of issues which were missed by the BA. When I’m testing I don’t just look at the functionality of the system. I use my experience to make everyone in my team to ‘think’ about the quality of the requirements. Obviously representing a customer and working hand in hand with the developers. Hence I call myself a Test Analyst.
Hi good to see your endless efforts in creating such a wonderful piece of article. I am also interested in Software testing but I also think manual testing in today’s digital era has no existence. Most of the companies around the world make use of Automation testing tools as they are easy and effective for Testing purposes. Automation in technology has also increased rapidly.
Hello, Diksha Kakade!
No need to thank me for anything, because as far as I am concerned, you haven’t read a single word of what I wrote. If you are interested in having a discussion under my posts , I’d prefer having it on the content I wrote and not your random thoughts.
Thanks!