In the previous part of hindsight lessons about exploration, I decided to share some knowledge about heuristics. In this part we will speak about one very specific heuristic and that’s the testing oracle heuristic.
What is a testing oracle?
In the area of technology, we can observe the word oracle in couple of contexts, aside from the company of course.
If you are interested in the broader meaning of the term oracle in computing, you can look at the definition of the oracle machine, related to the work of Alan Turing. It’s somehow related to the topic, because it is an entity capable of solving a problem.
In the domain of testing we use the concept of testing oracle – which has pretty much the same use – it is an entity, it might vary what kind, that provides us with a solution for a problem, or to validate, whether or not information that we have, concerning testing is correct.
Why oracle?
In the ancient times, oracle was a wise person, or an elder that was a go to advisor whenever important decision was to be made. “Do we have to start war on the north?”, or “Does or ruler’s daughter has to marry X to ensure peace?”, etc. It was all questions concerning critical problem and normally oracle’s words were taken very seriously.
This parallel is important, because in testing we call a testing oracle anything that can provide us with vital information, related to deciding or validating our decisions – normally related to quality, risk or possible problems.
What can be a testing oracle?
The answer to this question is simple, yet pretty deep – practically anything that holds important information about the product. If you are interested in learning more about oracles, I advise you to look at Michael Bolton’s series “Oracles inside out”. So, oracle can be (summed up, from Michael’s articles):
- A mental model or a feeling (Experience) – this might seem a bit abstract, but if you ever tested software, if you ever found a problem in it, you know that it all starts with a feeling. Something’s telling you – this is wrong, I need to investigate it. Anyway, that’s not good enough to file a bug report, so we investigate further.
- Artefacts (Reference) – any sort of written knowledge or document, design, draft or whatever communication is also a source of knowledge we can use as an oracle.
- Discussing with important knowledge holders (Conference) – one of the most important and sometimes underestimated sources of knowledge about the software or the project are the people that were involved in it, the end users, our colleagues, execs and anyone that was involved. I had my behind covered so many times, whenever I faced situation with limited or missing documentation, only due to the fact I had the initiative of asking important participants in the development for some clarity and information.
- Consistency with known products (Inference) – applying our knowledge of comparable products to the one we test now and search for:
- Inconsistencies – with community standards for the applications we are testing for example – all mobile calculators would have specific feature, but ours has it a bit different
- Consistencies – with problems we experienced in the same class of applications – ex. every time I’m testing file upload I am looking for problems with file size limitation validations, just because my previous experience (one oracle) with other similar apps(another oracle), let me know this is something that often has bugs.
Why are oracles so important?
Oracles are important concept, that I find being omitted in many materials and this is a problem, because they are corner stone in understanding software testing. Being a professional tester, means living at the edge of uncertainty – changing deadlines, changing requirements, questionable quality – oracles provide us with small bits of information that we can refer to as true.
Even further – they can provide us not only with a comparable way of validating questionable product behaviour, they can also teach us how to construct experiments that can uncover such problematic behaviour.
Unfortunately, although every tester is using oracles, the term oracle and the concept and theory behind that term is not very common among testers, which is a problem.
Where will you find oracles in the wild?
As I said earlier, oracles are a concept you use like it or not, in your testing, you might not know you are using it or call it an oracle.
So where are we using oracles:
- Test cases and test scripts – I am not a huge fan of these, yet people use them. So, if you are familiar about the concept of writing a test case, you know there is this part called “expected condition” – this is a classic example of an oracle, as it is telling anyone what the correct behaviour of the application is.
Same applies for expected in bug reports and automated checks. - Documentation – docs or specs are used as oracle, which is not bad, the problem is since they are written and maintained by people, they suffer the same problems as code, meaning they get outdated, have problems, vague wording, too technical, too shallow, too detailed, etc. The problems come when we, as testers, treat docs as the only valuable source of knowledge, where experimentation and exploration should be our primary guide.
Oracles in depth
I am not the first to write about oracles, nor am I saying anything new, in fact, I am just putting together whatever I believe worked for me and passing it on.
So, if you are interested in the testing oracles concept, there are few resources you might be interested in.
Here’s what oracles I got from these articles and what I applied in my daily routine.
- FEW HICCUPPS – this is irreplaceable oracle if you want to use it to validate whether you observe correct behaviour, and this is exactly what it was meant for – to be used as a check list. Anytime you find yourself in a situation where you are not sure if this really is a bug, you can refer to FEW HICCUPPS and try to brainstorm – can I validate it with a standard, do I understand the purpose, is this consistent with the product itself or a claim or product’s history, etc.
This sort of question driven reasoning is why I claim oracles can work backwards – not just as external entity to validate the truth about the product, but also to provoke us into coming up with experiments that can reveal mismatches between expectations and reality. - Regression oracle – if you think of regression testing as one possible oracle, in my understanding, you are being relieved of a lot of its burden, which is mainly, the expectation it’s irreplaceable, omni-potent requirement for efficient testing. Many people believe regression testing means to execute a suite of tests every time, or all your tests, or all your automated checks. I can spend an article to explain how many wrong things there is with all above, but because I cherish your time, please take my honest word for it. Instead, try treating regression as one kind of oracle, and being an oracle and respectfully, a heuristic it has limited scope of efficiency.
- Constraints as an oracle – I believe this is the most obvious, easy to use, and at the same time most omitted oracle. Every time you test something, you are given a list of constraints – max char length, number of allowed special chars, file size – all of these are clues for you. As a good tester, the first thing that you should think of is – what if I break the rules, what if I don’t write the number of symbols you asked me to? What will happen? Will there be error, will there by an exception, did the dev considered this scenario?
If you ever are one these people saying – “I just don’t know where to start testing?” – here’s a good action plan for you – find all constraints the product has, list them, try to break them in at least 3 different manners, take notes of your results, let the information you gather carry you into deeper testing. There’s a good start for your testing and it’s there, in front your eyes. - Self-verifying data as an oracle – I am using this one intensively in automated checks mainly in naming my tests and I try to teach devs to use it, too.
By common sense, or at least to me, the name of your automated check should give out the result that you are checking. This is also good game if you want to make sure your checks are focusing on a single action and the result of it or are getting distracted between different responsibilities.
Example for this: if my check is called testingValidAdminUserFlow and it fails, I have no clear idea where it failed, what’s wrong with it, etc. Which means I will have to investigate and that is increase in operational time.
On the other hand, if you make your checks small and definitive using the self-verifying data as an oracle, you can easily determine what the problem is. For example, if the check is called loginAsAdmin200Test and it fails, you will clearly know that logging in as administrator didn’t return status code 200.
Oracles are heuristics
As a few concluding words, like it or not, you are using oracles and believe me, that’s good for you. The more you know about them, the more you know.
The one thing you shouldn’t ever forget is that oracles are also heuristics and they suffer from the same intrinsic problems as heuristics – they are fallible and they rarely work as a single resource of truth. Your test documentation might be outdated, or confusing, or missing, or using fuzzy words. The knowledge holders could forget, or misguide you or have significant bias, the reference app that you rely to can have bugs, etc.
The bottom line I am trying to get to – please treat oracles as heuristics, keeping in mind their fallible nature and don’t stick to just one oracle as a provider of universal truth. Instead what we’re trying to do is to have multiple oracles provide data and see what of this data gives us confidence in making a decision or a conclusion, of course not without running it through our analytical, critically thinking brain.
Thanks for reading! 🙂
Mr.Slavchev
Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.
Follow Me:
I don’t understand something. Sorry if it is extremely stupid but I am still learning and it seems I miss something very important. You say: “Test cases and test scripts – I am not a huge fan of these, yet people use them.”
So you don’t write test cases? At all? Then how do you document the tests that you performed? I always thought that after we perform some exploration of the site, even though we don’t follow test cases written beforehand, we document the testing in the form of a test case with the steps we followed. And everything is saved in a bug tracking system. And then a statistic chart can be generated which shows how many test were performed, how many bugs….etc… Or I am totally missing something… I will ask this next time in the lessons too. I got a little confused.
Hello, Katerina! It is a totally reasonable question. If I am writing test cases – No, or at least not in the sense of test case as a “A set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test conditions.”(as per the ISTQB syllabus, since many people find it reliable – I don’t). Anyway, in my experience I noticed some testers use the term “test case” assigning it a broader meaning – for example – “You have tested the VAT field with input BG123456789, but have you considered the (test) case where there’s a letter in the end, for example in Irish VAT numbers”. In that regard, the test case here, is not a specific procedure that is supposed to be followed or executed, but a different variation if an idea about testing – that’s the reason we introduced the term “test ideas” as I believe this brings more clarity.
As for writing tests beforehand – you can do both ways – there’s nothing stopping you, as long as you don’t follow scripts step by step – this by definition is not exploration, but human checking.
How do I document testing then? There’s various ways of documenting exploratory testing and writing a procedural script is only one of them, sad thing is many testers don’t know any other.
How do we document testing matters also who’s reading that documentation – is it just used in a way to memorise our testing for an unknown future moment or it might serve as a deliverable to somebody we have to report to?
Assuming I am only writing documentation for myself with the probability for somebody else to read it, I normally write either test notes or test reports. This means that either during testing or after it, I sum up all the essential data related to it – important business cases, test data, integrations that are not obvious, problems that were deferred, but are still there, suspicious things, etc. I plan on adding a part about test documentation and reporting to the series, so stay posted.
“And then a statistic chart can be generated which shows how many test were performed, how many bugs….etc…”
This will be a very useless metric, though – counting how many tests or bugs you have has no relation to risk, nor value, it’s a cheap way to scratch “obsessive check-marking” itches or in other words – to believe you are done, when you actually aren’t. Here’s an example – what does it matter how many tests you have if you haven’t tested the core functionality all clients are going to use? Or is it reasonable to believe our product is OK, if we only have one bug open, but that bug has the potential to crash the system or produce tons of incorrect data? Counting bugs and tests is only useful to people who don’t understand testing for purposes that don’t help testing in any manner.
Thank you very much for the detailed explanation. It expained everything I was wondering about. I am very happy that you describe in all the articles here all the details even though they are very obvious for the experienced testers. But not obvious to us under-beginners.