Hindsight lessons about automation: Models of automation

Reading Time: 6 minutes

This chapter of hindsight lessons about automation is dedicated to models of automation. When we say “a model” or “to model” we mean to (random definition from Google, luckily a correct one):

a three-dimensional representation of a person or thing or of a proposed structure, typically on a smaller scale than the original

When we speak about automation modeling we will mean – representation of the structure of automated checks that we perform and their distribution across different layers. And yes, we will talk about the pyramid.

Models of automation: The testing pyramid

“The testing pyramid” by Martin Fowler is something that you will almost never miss at a conference about automation or testing.(In fact originally it was introduced by Mike Cohn in a book called “Succeeding with Agile”, attributed by Fowler, as well.) You will see various different representations of it, you will see it in a shape of a cone, a funnel, having clouds on the top, etc.

I mean, I’ve been to couple of conferences lately and I can say there’s “literally” (yes, literally, literally) no event that passes without somebody showing the pyramid and, of course, their own interpretation of it.

My humble hindsight lessons about the pyramid.

The testing pyramid triggers one of the “holy wars” in testing and I am taking my fair share of risk talking about it. The obvious things we can say about the pyramid, is that it is everywhere, it is simplistic and everyone is getting the idea, BUT because it is simplistic, it allows a lot of subjective interpretations.

I don’t like referring to it as “The testing pyramid”, it is not “The” it is “a pyramid” (triangle, if we want to be dicks about it, but we are not 🙂 ) and most importantly it is “a model” about automated checking, automation in testing or whatever suits your taste. Meaning – it is one of the many models that are out there – I can create a hamburger model and there it is, one more model.

Models vary with contexts, it is weird to rely on the same model to serve different purposes.

The good parts

The good parts of the pyramid are that it is easy to show and grasp.

It is awesome for showing that there is more than the UI layer where we can automate. Amen!

It is also very good to show approach towards automation to anyone unaware of the technical side of testing and development – PMs, POs, managers, etc.

The bad parts

Ambiguity – because the pyramid is simple, it looks like everyone gets it. And everyone gets it in a different manner, which leads to the “holy war”. Which is a problem – if something doesn’t serve to ease conversation, but to trigger misinterpretation, it’s a problem.

Numbers – one of the interpretations is that layers mean number of tests and that going closer to the base you have to write more tests, while focusing on small number of tests in the UI layer. There is a hidden bias here and it is that a quality of a test is constant. And this is a bias I see more often in coders, rather than in testers (I wrote an article about the two separate mindsets of a tester and a coder here – the missing link). A written test is not a good test, it’s simply a test. Writing a high quality test is journey on its own, any tester knows this, therefore number of tests means absolutely nothing related to the quality of the tests. So, rather than focusing on numbers of tests, it’s a better idea to think about the value that these tests provide, or the potential risks that they can uncover.

James Bach focused on that same topic and presented his “Round Earth” heuristic, I encourage you to take a look at his slides.

Some hidden dimensions of the pyramid

Although Martin Fowler made a hint about these in his model (above) there is few hidden dimensions of the pyramid, that we often forget about. Some of them are pretty straight forward, all of them are mirror like (inverted pyramids), but at least to me, they give a lot of clarity why sometimes people do stuff as crazy as focusing only on UI checks.

Inverted pyramid of check complexity and time investment

This is a pretty obvious one and it’s one of the main ideas behind the pyramid – lower level tests are cheap fast and easy to write, higher the stack we go, the more troubles we face.

Logically, the more we move upwards the harder it gets to write tests that are deterministic, which means we invest more time and effort for tests that provide less reliable information. And this is normal, because going up the stack we are moving from code only (unit), code to external code(integration), code to API, until we hit code to client. In the UI we are trying to automate an interface, that is not meant to be operated by code. I will focus  more on interfaces in the next chapter.

So, reasonable question arises – why, considering all the above, somebody will be that crazy to write tests for UI and UI only. And the answer is:

Inverted pyramid of failure visibility.

A valid reason to automate in the UI is that it gives us the richest understanding how that failure affects our clients. Try to think about it. You have a unit test, it fails – expected result is 4, you get null. Great we found a bug, but how does that affect our client?  Will it freeze, will it throw error, will it result blank page or a stack trace? We have no idea, we can only guess. The reason why it makes sense to invest in UI and API tests is that closer to the client you get, the better understanding you get about the depth and real impact of the problem. Of course, this comes at a significant price.

I focused a lot on the pyramid, but a bit earlier I said I don’t consider it a “testing pyramid”, but an “automation pyramid”. Why is this?

It was never a pyramid

It is in fact an iceberg. I know probably somebody, somewhere already used the iceberg metaphor, so I won’t act like I invented the wheel or the hot water, and will just stick to the specifics.

Enjoy this ugly diagram, created by me 😀

The reason why I refer to it as an iceberg is that very often I see discussions only focused on the visible parts – the checks, the layers, the implementation, the tools, the practices, etc.

Almost never or a limited amount of times I see somebody to try to relate automation and testing and ask themselves – “why am I doing this?”, “Is it because it brings value and helps achieve the testing purpose, or am I doing it, because it’s cool and my salary will get higher?”

I almost never see automation developers to focus on the strategy, on approaches, on test framing – trying to connect each test that you write the purpose and the testing mission and answer the question – “how does my test help to achieve it?”.

What we should realize, is that if our pyramid doesn’t have a base lying on a solid strategy, clear approach, clear reason why are we doing it, we will probably generate something that is a good demonstration what our product and our testing framework can do, but will not produce sufficient testing.

If our automation model has that solid base, it might turn it’s not a pyramid at all, but something else, something that serves our context, our purpose and our strategy.

So, should I paint myself in the colours of war and wait the “holy war of testing”  to happen in the comments or you are willing to share your view 🙂 . I leave that up to you.

Thanks for reading! Any shares in social networks are highly appreciated 😉