As a second part of the hindsight lessons about exploration, I’d like to focus on a topic I don’t see discussed very often, but in the same time I find it is a vital point in widening your horizons to better exploratory approach.
Very often, when I ask my students “How would you test an application?”, I get answer such as “I’d compare the product to the documentation or specification”. My logical follow-up question to that is: what if you don’t have a specification or the product is not built, yet?
In this part we will think on exploring artifacts that testing can benefit from and see how we can build testing strategy with or without docs or even find issues with them. In other words, we will find out how to “think outside the dox” (pun totally intended).
A game of shadows
When Plato, the famous ancient philosopher, was explaining the concept of ideas or forms, he was using the following metaphor. Imagine the world as a giant cave with a big fire in the middle. All humans were chained with their backs facing the fire, only seeing the walls of the cave. Plato claimed, that everything we saw in our everyday lives were just the shadows of the real objects, projected on the walls of the cave. he also claimed there was only one person who managed to “leave the cave” and that was his teacher – Socrates. Now that’s enough philosophy.
As testers, if we blindly follow whatever is written in the documentation or specification, we are risking to harm our critical abilities, in other words, we will be only seeing the shadows projected on the wall. Testing is the ability to explore, discover, experiment, question and model, doubt even your own beliefs and hypothesis in order to provide fast, meaningful and accurate evaluation of risk and quality.
In order to have holistic understanding about the product and its risks, we should treat documentation just like any other testable asset of our product. And it needs to be reviewed with our critical glasses on and tested in the same manner that we test our product.
Testing the documentation
You will not be asked very often to test the documentation, and there’s couple of good reasons for that, in fact they are not good, but they are handy and true:
- Docs are considered the 10 commandments of a software project, you doubt them, you doubt the very foundation of the business decisions that are made. That’s not an area of comfort to let somebody with a professional drive to question and dig deep into problems, I guess.
- Testers, are still, unfortunately, looked at as less technical or often non-technical members, who “don’t quite get it” to question the documentation.
- Another good reason is, the doc is so out of date, that nobody really cares about it.
Why is it a good idea to test docs and specs?
Aside from all of the above, which I hope you understand is common folklore and urban legends, testing docs, specs and any artifacts has as much value as testing the product itself, especially when it comes to testing in a phase too early to test the actual product.
If you were successful to build good credibility, people will be happy to give you specification for review. Just like any other part of deliverable – code, bug report, testing report, etc. the doc or spec is a work in progress, it could be hardly considered perfect and nobody expects it to be. It takes time and iterations to go over it and get something really valuable.
I guess the most important reason to test early and dig your nose in the docs is the propagation of mistakes and complexity. If there’s a mistake in the docs, it will be a mistake to implement it, it will be mistake to test it. Same with complexity.
Another good reason to test the docs is it enables dialogue with your stake holders or representatives of the business, if you are lucky and you are not too annoying, you will be able to build a good relationship and understanding of it, that will help you later on.
Developers very often use the term “code smells” for patterns that they find harmful for good coding. In the same manner, based on my experience, I can come up with some doc smells that should invoke a warning in your brain, whenever you see them in your documents or technical specification.
So, here are few warnings that I find valuable addressing:
Now, this might vary, there’s different type of artifacts, some of them represent a user story, some of them represent a technical specification, sometimes it’s a user guidance, but bear that in mind – nobody would read a lengthy document. Unless they work in a library or in a scientific discipline. The reason for this is purely psychological, it’s nothing to do with processes or methodologies, it’s simply that people are being carefull with the amount of data they can devour.
A possible advice could be, try to keep your technical docs and user stories short, concise and understandable. Other wise the risk to run into the next problems is immanent, and it is …
I will probably get slightly trivial, but I will use this quote from Einstein:
We are having terrible problem with complexity. Everyone explains what sort of complex environment they work at, or what complex process they had or what complex software they build. And guess what? Complexity sucks! Wanna know why? Because nobody understands it, unless they invest every single piece of brain cells in order to understand it, which simply doesn’t happen.
So, having complexity in your documentation might provoke misunderstanding in taking important decisions. It also might invoke lazy reading in anyone involved in the chain of implementation – architects, developers, testers. If it is complex, the risk to misunderstand, therefore build the wrong thing is enormous. The biggest problem with complexity is it has this avalanche effect, if there’s something misunderstood in the docs or spec, there will be wrong decisions taken, wrong, complex and misunderstood implementation on the dev side, by the time this reaches testing it will be a huge ball of complexity and mistakes. In this case testers either take the easy way to “wave it” and also test it wrong, or start picking in it and normally raises a discussion that requires a redesign. None of this is pleasant, to be honest.
I use the term “fuzzy words” in my lectures a lot, so my student perhaps hate them, but I like this term and it relates very well to any type of document, from bug report, to test notes, to technical specs.
Fuzzy word is a word or more likely a term that doesn’t have very specific meaning, but is tightly bond with some very narrow context.
In technology, whenever we write documentation, we loove fuzzy words, especially whenever we generate documentation for members of our teams.
Some such fuzzy words are: “should work as it is expected” just like there’s a thick book with a definition how everything in this world should be expected to work or “should work according to users expectations”, like we only have one user, with one expectation… or my personal favorite – “should be consistent” with whatever. I guess you see, that in order to understand what these mean, you should be immersed in the context, but imagine a new team member, or a non-technical person included in the project, or a technical person from a different domain. These won’t mean anything to them.
A good solution for these – ask for a review from a fresh pair of eyes. Try to find someone who doesn’t share your context and ask them for advice if what you wrote is transparent enough.
Tribal knowledge is the type of knowledge we share whenever we work together for a long time, it is that vital part of our communication that helps us understand each other almost without saying a word. Now, there’s a huge problem with tribal knowledge – it is very implicit and it doesn’t belong to documentation. In practice tribal knowledge is not not knowledge, it’s hiding knowledge.
Normally, what we refer to as “tribal knowledge” is that “everyone knows this, because it’s common sense” or “anyone would know how it works, because it was in the system forever”. I guess I don’t have to explain how wrong is that.
A good way to try to heal this is by trying to make your documentation as explicit as possible.
Well, I guess that’s my short list of interesting topics for testing documentation. I am open to hear and reading your advices in testing docs in the comments.
Thanks for reading, hope you had a good time! 😉
2 thoughts on “Hindsight lessons about exploration: Testing documentation – “think outside the dox””
I was going to comment after I read all of your articles on the exploratory testing but I just want to say that I learned a great deal so far. I am one of your current students. The thing that I am most shoked was the last title Tribal knowledge. Wow! I didn’t know this has a name but I have always thought this thing is the problem of all fields of education and learning. And because of this tribal knowledge a lot of concepts are not explained the right way and all the new generations understand all the “well known” information less and less and the knowledge just disappears. (And legends are born.) And it’s great that I just learned a new term and what this is called as a whole! Thank you very much! There are a lot other suprising interesting new things in addition to everything you hve told us but I won’t spam more. I continue…