Randomize test execution in TestNG to improve test isolation.

Reading Time: 3 minutes
TestNG randomize test execution
Photo by http://testingfreak.com/wp-content/uploads/2015/01/testng1.png

Recently I watched this awesome talk by Simon Stewart which is kind of old, but still awesome, since I am quite green yet in the area of test automation or at least the meaningful and useful test automation. In it he used to talk about test isolation and how important it is. In a few words test isolation is the ability of your tests to run in an isolated environment, but also to run separately, non-depending on each other. So, there’s many ways to do so, may be if you’re interested you should watch the talk, but one of them is to randomize test execution. So, I decided to incorporate this using TestNG, since we are using it in our code and it’s kinda interesting to have this in my tool set, I didn’t find it a lot on the web.

How to randomize test execution using Java and TestNG.

Here are some code examples that I made just for the demo, I repeat for the demo, so don’t mind the moronish names that I’ve chosen for my classes. And it’s fairly simple example – we have a test class with some code, in our case it’s just outputting “in method …”. That’s just to keep track on which test is executed when, what’s the sequence of our test execution. So here’s the code of our test class:

As I said, simply moronish code example. So if we run this as testNG test we get the following output in the console:

And it does that every time we run them. So what we need is to randomize test execution in our class.

Why randomize test execution? How is this important?

Well, if you have a test suite of 30-40 test or more and you run them every time in the same order, try applying this solution and you will find out why. Turns out we as coders assume that we have many things for granted, for example – a specific state of the data in the DB or a specific screen that we land or start from. Turns out, this is not like that and unknowingly we’ve incorporated some dependency on our tests, which is not good if we want to use them in continuous integration or in some other environment. So, the biggest plus from having a good test isolation is – we don’t want to fix the damn thing every fucking time we change the environment it runs at.

See also  Hindsight lessons about automation: Test isolation principle

So, I’ve searched around and I found this Stackoverflow question, which had my solution.

In a separate package let’s say it’s called utils you may add the following class:

And in order to randomize test execution of certain class you have to add the following annotation to your test class:

So, when your run the tests now, the output looks like this.

Which is a bit better. The cool part of randomizing test class like this is the nanoseconds feed of the random, as you see it is logged on the console and if you find anything wrong in a specific sequence you could always copy the feed from console and hard code it in our random feed, in order to re-create the exact same scenario. Which is fucking awesome.

See also  Some kick ass blog posts from last week #10

Thank you all for reading and hope that was interesting and helpful for all of you. If you liked it, please share it with your  friends and/or share your thoughts in the comments.


Senior software engineer in testing. The views I express here are mine, they don't represent any position held by any of my employers. Experience in mobile, automation, usability and exploratory testing. Rebel-driven tester, interested in the scientific part of testing and the thinking involved. Testing troll for life. Retired gamer and a beer lover. Martial arts practitioner.

More Posts - Website

Follow Me:

4 thoughts on “Randomize test execution in TestNG to improve test isolation.”

  1. Thanks for the good write-up. I had to include the Listeners annotation in my test class well:

    import org.testng.annotations.Listeners;

  2. Hi, these methodInterceptor sorting and shuffling examples are nice, but do you have an idea on how to make TestNG execute the following sequence of test methods: method1, method1, method2, method1, method1, method1, method2 , method1 ?
    I tried to create a new list of IMethodInstance and then just add method1 and method2 elements of original list (the one passed into intercept call) to the new one in my special order, and return it – but this way TestNG executes each test method just once. Probably because after 1st execution the methodInstance is already “executed”, so all other links to THE SAME methodInstance that are in the list are just ignored.
    So, Is there any way to ADD a NEW methodInstance with the specified test name to the list?

    1. Hello, Alexey!
      No, I don’t know how. Probably you can, if you write a different implementation, but it’s not within the domain of “randomized test execution”.
      In fact, the need to execute same tests more than once, is only useful if you want to tests their brittleness, e.g. if 10 consecutive runs will yield same results.
      Running same tests in a specific sequence, seems like a pretty weird design to me. Do you mind sharing what case are you actually trying to solve?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.