Randomize test execution in TestNG to improve test isolation.

Recently I watched this awesome talk by Simon Stewart which is kind of old, but still awesome, since I am quite green yet in the area of test automation or at least the meaningful and useful test automation. In it he used to talk about test isolation and how important it is. In a few words test isolation is the ability of your tests to run in an isolated environment, but also to run separately, non-depending on each other. So, there’s many ways to do so, may be if you’re interested you should watch the talk, but one of them is to randomize test execution. So, I decided to incorporate this using TestNG, since we are using it in our code and it’s kinda interesting to have this in my tool set, I didn’t find it a lot on the web.

How to randomize test execution using Java and TestNG.

Here are some code examples that I made just for the demo, I repeat for the demo, so don’t mind the moronish names that I’ve chosen for my classes. And it’s fairly simple example – we have a test class with some code, in our case it’s just outputting “in method …”. That’s just to keep track on which test is executed when, what’s the sequence of our test execution. So here’s the code of our test class:

import org.testng.annotations.Test;

public class TestingRandomizer {

	@Test
	public void Test1(){
		System.out.println("Here in test 1.");
	}
	
	@Test
	public void Test2(){
		System.out.println("Here in test 2.");
	}
	
	@Test
	public void Test3(){
		System.out.println("Here in test 3.");
	}
	
	@Test
	public void Test4(){
		System.out.println("Here in test 4.");
	}
	
	@Test
	public void Test5(){
		System.out.println("Here in test 5.");
	}

As I said, simply moronish code example. So if we run this as testNG test we get the following output in the console:

Here in test 1.
Here in test 2.
Here in test 3.
Here in test 4.
Here in test 5.

And it does that every time we run them. So what we need is to randomize test execution in our class.

Why randomize test execution? How is this important?

Well, if you have a test suite of 30-40 test or more and you run them every time in the same order, try applying this solution and you will find out why. Turns out we as coders assume that we have many things for granted, for example – a specific state of the data in the DB or a specific screen that we land or start from. Turns out, this is not like that and unknowingly we’ve incorporated some dependency on our tests, which is not good if we want to use them in continuous integration or in some other environment. So, the biggest plus from having a good test isolation is – we don’t want to fix the damn thing every fucking time we change the environment it runs at.

So, I’ve searched around and I found this Stackoverflow question, which had my solution.

In a separate package let’s say it’s called utils you may add the following class:

package com.utils;

import java.util.Collections;
import java.util.List;
import java.util.Random;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;

public class TestOrderRandomizer implements IMethodInterceptor {
    @Override
    public List intercept(List methods, ITestContext context) {
        long seed = System.nanoTime();
        System.out.println(String.format("Randomizing order of tests with seed: %s", seed));
        Collections.shuffle(methods, new Random(seed));
        return methods;
    }
}

And in order to randomize test execution of certain class you have to add the following annotation to your test class:

@Listeners(TestOrderRandomizer.class)

So, when your run the tests now, the output looks like this.

Randomizing order of tests with seed: 217769194550514
Here in test 5.
Here in test 1.
Here in test 3.
Here in test 2.
Here in test 4.

Which is a bit better. The cool part of randomizing test class like this is the nanoseconds feed of the random, as you see it is logged on the console and if you find anything wrong in a specific sequence you could always copy the feed from console and hard code it in our random feed, in order to re-create the exact same scenario. Which is fucking awesome.

Thank you all for reading and hope that was interesting and helpful for all of you. If you liked it, please share it with your  friends and/or share your thoughts in the comments.

Please follow and like us:

Mr.Slavchev

Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.

More Posts - Website

Follow Me: