Randomize test execution in TestNG to improve test isolation.

Reading Time: 3 minutes
TestNG randomize test execution
Photo by http://testingfreak.com/wp-content/uploads/2015/01/testng1.png

Recently I watched this awesome talk by Simon Stewart which is kind of old, but still awesome, since I am quite green yet in the area of test automation or at least the meaningful and useful test automation. In it he used to talk about test isolation and how important it is. In a few words test isolation is the ability of your tests to run in an isolated environment, but also to run separately, non-depending on each other. So, there’s many ways to do so, may be if you’re interested you should watch the talk, but one of them is to randomize test execution. So, I decided to incorporate this using TestNG, since we are using it in our code and it’s kinda interesting to have this in my tool set, I didn’t find it a lot on the web.

How to randomize test execution using Java and TestNG.

Here are some code examples that I made just for the demo, I repeat for the demo, so don’t mind the moronish names that I’ve chosen for my classes. And it’s fairly simple example – we have a test class with some code, in our case it’s just outputting “in method …”. That’s just to keep track on which test is executed when, what’s the sequence of our test execution. So here’s the code of our test class:

import org.testng.annotations.Test;

public class TestingRandomizer {

	public void Test1(){
		System.out.println("Here in test 1.");
	public void Test2(){
		System.out.println("Here in test 2.");
	public void Test3(){
		System.out.println("Here in test 3.");
	public void Test4(){
		System.out.println("Here in test 4.");
	public void Test5(){
		System.out.println("Here in test 5.");

As I said, simply moronish code example. So if we run this as testNG test we get the following output in the console:

Here in test 1.
Here in test 2.
Here in test 3.
Here in test 4.
Here in test 5.

And it does that every time we run them. So what we need is to randomize test execution in our class.

See also  100 posts in the blog – reflections, thoughts, and insights on blogging as a tester.

Why randomize test execution? How is this important?

Well, if you have a test suite of 30-40 test or more and you run them every time in the same order, try applying this solution and you will find out why. Turns out we as coders assume that we have many things for granted, for example – a specific state of the data in the DB or a specific screen that we land or start from. Turns out, this is not like that and unknowingly we’ve incorporated some dependency on our tests, which is not good if we want to use them in continuous integration or in some other environment. So, the biggest plus from having a good test isolation is – we don’t want to fix the damn thing every fucking time we change the environment it runs at.

So, I’ve searched around and I found this Stackoverflow question, which had my solution.

In a separate package let’s say it’s called utils you may add the following class:

package com.utils;

import java.util.Collections;
import java.util.List;
import java.util.Random;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;

public class TestOrderRandomizer implements IMethodInterceptor {
    public List intercept(List methods, ITestContext context) {
        long seed = System.nanoTime();
        System.out.println(String.format("Randomizing order of tests with seed: %s", seed));
        Collections.shuffle(methods, new Random(seed));
        return methods;

And in order to randomize test execution of certain class you have to add the following annotation to your test class:


So, when your run the tests now, the output looks like this.

Randomizing order of tests with seed: 217769194550514
Here in test 5.
Here in test 1.
Here in test 3.
Here in test 2.
Here in test 4.

Which is a bit better. The cool part of randomizing test class like this is the nanoseconds feed of the random, as you see it is logged on the console and if you find anything wrong in a specific sequence you could always copy the feed from console and hard code it in our random feed, in order to re-create the exact same scenario. Which is fucking awesome.

See also  Testing is a mental process.

Thank you all for reading and hope that was interesting and helpful for all of you. If you liked it, please share it with your  friends and/or share your thoughts in the comments.

Please follow and like us:


Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.

More Posts - Website

Follow Me:

6 thoughts on “Randomize test execution in TestNG to improve test isolation.”

  1. Thanks for the good write-up. I had to include the Listeners annotation in my test class well:

    import org.testng.annotations.Listeners;

  2. Hi, these methodInterceptor sorting and shuffling examples are nice, but do you have an idea on how to make TestNG execute the following sequence of test methods: method1, method1, method2, method1, method1, method1, method2 , method1 ?
    I tried to create a new list of IMethodInstance and then just add method1 and method2 elements of original list (the one passed into intercept call) to the new one in my special order, and return it – but this way TestNG executes each test method just once. Probably because after 1st execution the methodInstance is already “executed”, so all other links to THE SAME methodInstance that are in the list are just ignored.
    So, Is there any way to ADD a NEW methodInstance with the specified test name to the list?

    1. Hello, Alexey!
      No, I don’t know how. Probably you can, if you write a different implementation, but it’s not within the domain of “randomized test execution”.
      In fact, the need to execute same tests more than once, is only useful if you want to tests their brittleness, e.g. if 10 consecutive runs will yield same results.
      Running same tests in a specific sequence, seems like a pretty weird design to me. Do you mind sharing what case are you actually trying to solve?

  3. Hello Mr.Slavchev,

    Thank you for the nice article.

    Instead of randomizing the seed within TestOrderRandomizer, is it possible to feed the seed as a input parameter from my test class for randomization?

    Best regards,

    1. Hello, Deepakraj!
      As you can see I am not the original author of the solution, and it’s been some time since I worked on this one, but I will try to help.
      Short answer seems like No. This is listener class and as you can see we invoke the class as an anotation, not really having any control as passing params to it, etc. And it doesn’t really makes sense, given we want random execution.
      Yet, a cool thing you can do during debuggin, if you want to run your tests with a specific run, that say causes issues in CI, you can easily do so, if you just copy that seed number that’s is logged in the consol with test results and pass it to the line:
      Collections.shuffle(methods, new Random(seed));
      instead of the second parameter new Random(seed).
      Let me know if that answered you question.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.