The CI build is not a Christmas tree

Reading Time: 6 minutes

 Short story about the Evergreen CI build

Let me tell you a story, it’s a Christmas story. Someone will probably say “but Christmas is gone, it was like 4 months ago”, but you what’s not gone? Stupidity that people do, calling it a work process. Have you heard developers (and may be testers) saying something like: “We should fix the CI build, so all tests pass”? Or something like “My CI build failed again, how do I deal with these flaky, unstable tests?”. Sounds familiar? Well, seems to me, developers believe that the purpose of the build is to be always green, I like to call it – the Evergreen CI build (that’s why it’s a Christmas story), or even shorter, in order to remember easily, I call the Evergreench. Now, the Evergreench is a monster, in fact, it’s a monster that wants to steal your quality and kill your project. It’s a Christmas horror story.

The purpose of the CI build

If you are a curious kid like me (although I’m definitely not a kid, anymore) you probably opened a book about CI/CD when this was a booming new trend in development, in order to find out, what is it all about.

The purpose of CI/CD builds, as per the books and the community, is to provide faster feedback or shorter information loop. But faster feedback about what?

In testing, we believe that the purpose of it is information – providing vital information about compromised quality, inevitable risks and how to mitigate them. There are many ways we can obtain this information (exploration, modeling, questioning, etc. check Michael Bolton’s definition) and testing is only one way to do it. Checking (the process of confirmation a belief or information that we already possess) being manual or machine is another such methodology, for obtaining more information.

As a conclusion, the purpose of a CI/CD build is not just have a flashy new development technology to brag about when we are drinking beers with the colleagues or go to conferences, but to run automated checks quickly, so we can receive fast feedback about existing problems that might threaten the quality of our project, delay it’s completion, risk the integrity of the product or simply slow and put obstacles in front of human testers. I very deliberately put those words in bold – the purpose of the CI/CD build is to fail, because failure means we found a problem.

Why is failing build a good thing?

To have the build failing due to failing automated checks might mean couple of different things even sometimes combined.

• If you are lucky, it will mean you found a problem, that otherwise go to you client, or to other testers that will have to document it and retest it later.
• It might mean your checks are testing incorrectly and failing for reasons different than finding a problem. This might seem annoying, but again it is a good thing – this shows you, there’s a bug in your testware (or the code that’s performing checks), therefore you can fix it and have a meaningful, useful check again.
• In general a failing check will be an open invitation for in depth investigation of a possible suspect issue. And this is a good thing again, because it drives you to think critically.

Stepping in their shoes

This sounds all good, but what’s the reason for people looking for evergreen builds?

In my humble experience – on one side it’s the builder’s syndrome – people involved in coding a software product are interested in building it and shipping it as fast as possible. Everything that stays between them and a merged feature is a problem, an obstacle, annoyance, etc. What we as testers and people involved in testing and quality advocacy should remind them constantly is that – deploying can’t be pursued at any cost, if we deploy a compromised feature, we are all in trouble, not just the testers. If we delay release due to the enormous effort of testing, part of which might be handled by automated checks, we are again all in trouble. And even further – just writing code, throwing it to a wall of tester and waiting for them to spit back bugs at you, is not a cool thing anymore, it’s outdated, and the cool kids don’t do it anymore.

In the modern world of development, every single person involved in the creation if a product, is bearing their fair share of testing.

You think you are immune to the “builders’ syndrome”? Watch yourself writing automated checks, man! You will be running them once and committing them in source control in a blink of an eye, causing them to fail hundred times in future, while you will be claiming they are “flaky”. No, they are not flaky, that’s simply bugs in your code.

The “test this for me” problem

Related to the builder’s syndrome, I find very often another problem when the mindset of “here test this for me”. Very often, being a tester, you will face the following problems with software development teams:

1. No time will be left for your testing or it will not even be accounted for when estimates are done.
2. Devs will report tasks “done” while they are in fact queueing up on your account not even started for testing, which will either put you on the spot for blame games or you will be under serious time pressure.
3. You will get an angry, passive aggressive comments when you’ve found new bugs.

Considering all the above, I believe a found a regularity – seems to me, that many professionals in IT even managers and testers don’t want from testing to be an emblem of critical thinking and in depth analysis, they just want it to “pass”.  Many individuals see testers as some external entity “certifying” their work by confirming it’s OK. Which is a terrible idea and if you are a tester, you should put everything you have in rebelling against it. That’s the reason they want green builds, instead of bug reports.

What this means is – they will happily pass you a product or a feature they didn’t even open to see if it loads, because it’s “supposedly working”, but they will put all the blame on you, if you fail to find the problems of their negligence. And this is a game you definitely don’t want to be playing all day long.

How to counter that?

It might seem like this is only “their” problem, whoever the fuck “they” are, and I bet that’s what you think, but the truth is it’s our problem.

First, it’s our problem, because when you are a part of a team, it’s a problem of the team, there’s no “us” and “them” in teams, we fail, it’s a failure for everyone.

Second, it’s our problem, because we testers still support or stay silent to falsehoods like one claiming “testing is quality control” that happens in the end of a production line and it should confirm the product works. This is simply bullshit, people that believe it are bullshit, as well. You can easily see why there’s so many wrong expectations for green builds, if you layout the purpose of your craft like that. Your work, as an experienced testers is to tell truth and bullshit apart and fight against the latter.

It helps a lot, to show your teammates that testing is something that requires time, because it’s a valuable in depth activity.

How to make your team understand testing is not confirmatory activity? Well, I have a solution for that, but the thing is not a lot of individuals will have the guts to test it. Put the responsibility on their shoulders – tell them – “If you are in a hurry and you can’t wait for me to test it, go ahead, deploy it, I trust you! But I won’t put my name under it, because this will be a lie”. I can almost certainly guarantee you, that will change their perspective very fast, if not within one or two sprints, when they “reap” the product of their high self-esteem. Anyway, that might not be necessary, as most of the times their conscience will do the job, instead.

What I consider vital for a role of a tester as a part of a software team is mentoring and supporting teammates in activities related to testing.

Some such good lessons might be:

• Offer your help and support in their testing activities such as unit and integration tests – although the code is theirs, the tests are theirs, you can provide your unique perspective on test design, length, pre-conditions and handling of data seeding if needed.
• Make them aware of the most important principles of testing that we follow, like for example, that testing everything is impossible and that bugs are part of the life cycle of the software. You might have to make couple of trainings with them, to demonstrate these in action.
• Make them aware that testing starts as soon as ideas are born and not at the end of development, if we want to have a working product, we should create it with the respective testing in mind. This way, they will know they should ask for testing as soon as possible, at the earlier possible moment, not at the last possible one “at the end of the production line”.

Instead of a conclusion

Instead of a conclusion, I’d like to give you a homework. I bet most of you encountered the situation I described earlier or been part of a team in which testing is considered secondary to development. My task for you is – next time some of the devs says something like: “We must fix the tests, because they failed”, work with them, speak with them, ask them to pair with you. Eventually, you will find out it wasn’t the check that failed, but the system and the check was in fact successful in finding a problem that would instead creep into prod.

Learn the important lesson to demonstrate the wins of our craft, when the things we do are successful and save time and effort and nerves. Be a mentor instead of a simple employee. This is experience in action.