TestNG - Notes and Support

Detailed info:

http://testng.org/doc/documentation-main.html



Presention that makes you understand TestNG better and differentiates from Junit:

source: http://kaczanowscy.pl/tomek/sites/default/files/testng_vs_junit.txt.slidy_.html#(24)

 Why TestNG?

My (biased) observations People are moving from JUnit to TestNG once they start to write integration and end-to-end tests (e.g. with Selenium). Apparently, on unit tests level you can put up with JUnit deficiencies, but once you go outside the unit tests realm, JUnit stands in your way.

	From what I have noticed, people who work with JUnit usually test their code with only one set of parameters. I suspect this is because parametrized tests are unusable with JUnit. And this is bad, because such tests are really useful.
	If you really, really have to use JUnit (do you?) then at least us JUnit Params!

Parametrized tests
Parametrized tests
@DataProvider
private static final Object[][] getMoney(){
    return new Object[][] {
        {new Money(4, "USD"), new Money(3, "USD"), 7},
        {new Money(1, "EUR"), new Money(4, "EUR"), 5},
        {new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}

@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
        int expectedResult) {
    Money result = a.add(b);
    assertEquals(result.getAmount(), expectedResult);
}
Parametrized tests
@DataProvider
private static final Object[][] getMoney(){
    return new Object[][] {
        {new Money(4, "USD"), new Money(3, "USD"), 7},
        {new Money(1, "EUR"), new Money(4, "EUR"), 5},
        {new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}

@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
        int expectedResult) {
    Money result = a.add(b);
    assertEquals(result.getAmount(), expectedResult);
}

@DataProvider
private static final Object[][] getInvalidAmount(){
    return new Integer[][] {{-12387}, {-5}, {-1}};
}

@Test(dataProvider = "getInvalidAmount", expectedExceptions = IllegalArgumentException.class)
public void shouldThrowIAEForInvalidAmount(int invalidAmount) {
    Money money = new Money(invalidAmount, VALID_CURRENCY);
}

@DataProvider
private static final Object[][] getInvalidCurrency(){
    return new String[][] {{null}, {""}};
}

@Test(dataProvider = "getInvalidCurrency", expectedExceptions = IllegalArgumentException.class)
public void shouldThrowIAEForInvalidCurrency(String invalidCurrency) {
    Money money = new Money(VALID_AMOUNT, invalidCurrency);
}
Test Fixture Setting
@Test
public class TestFixtureTest {

    private Server server;
    private Client client;

    @BeforeSuite(timeOut = 1000)
    public void setUpServer() {
        server = new Server();
        server.start();
    }

    @BeforeMethod
    public void setUpClient() { client = new Client(); }

    // some test methods here

    @AfterSuite
    public void shutDownServer() { server.stop(); }
}
Logical dependencies gives you a much more realistic error information - you learn that 1 tests has failed and 99 has been skipped, which is much easier to fix than the information about 100 failed tests (OMG! what’s going on!? All tests failed…. aaarrhghg!).
Dependencies between tests
@Test
public void serverStartedOk() {}

@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}
Dependencies between tests
@Test(groups = { "init" })
public void serverStartedOk() {}

@Test(groups = { "init" })
public void initEnvironment() {}

@Test(dependsOnGroups = { "init.* })
public void method1() {}
Listeners, Reporters
OK       FirstTest           fifthTestMethod            1944
OK       FirstTest           firstTestMethod            3799
OK       FirstTest           fourthTestMethod           1920
OK       FirstTest           secondTestMethod           4891
OK       FirstTest           thirdTestMethod            2963
OK       SecondTest          methodA                    3525
OK       SecondTest          methodB                    1390
FAILED   SecondTest          methodC                     117
SKIPPED  SecondTest          methodD                      23
OK       SecondTest          methodE                    4552
There is more of this
Who is the boss?

	JUnit	TestNG
releases since 2009 (not counting betas)	3 - from 4.8.2 to 4.10 (see JUnit announcements)	13 - from 5.12 to 6.4 (see TestNG Changelog)
mailing list	JUnit mailing list, members: 8511 emails since January 2012: ~25	TestNG mailing list, members: 2858, emails since January 2012: ~560, threads: 115

While better support for integration and end-to-end tests is definitely true, not many people know that TestNG offers more than JUnit also in case of unit tests.

Concurrency

No need to create own Thread objects! Use annotations and that is it!

An example of running same test method simultaneously (20 times, by 3 threads):

@Test(threadPoolSize = 3, invocationCount = 20)

public void concurrencyTest() {

    System.out.print(" " + Thread.currentThread().getId());

}

Result:

13 12 13 11 12 13 11 12 13 12 11 13 12 11 11 13 12 11 12 13

Parametrized tests

In general, it is a good practice, to test your code with different sets of parameters:

§  expected values: sqrt(4), sqrt(9),

§  boundary values: sqrt(0),

§  strange/unexpected values: sqrt(-1), sqrt(3)

A good testing framework should make it simple, and avoid copy&paste coding.

Alas, the default JUnit implementation of the parametrized tests is rather unusable… In short, JUnit expects all test methods to be parameter-less. Because of this constraint, parametrized tests with JUnit looks awkward, because parameters are passed by constructor and kept as fields!

You can see some examples here, here or here.

Parametrized tests are very simple with TestNG (no constructor awkwardness).

You can have as many data providers in one class as you wish. You can reuse them (call them from other classes), and you can make them "lazy", so each set of parameters is created when required.

Once you migrate to TestNG you will notice that you start writing a lot of parametrized tests. Because they are so very useful!

Ok, so this is how it looks with TestNG. There is a test method (shouldAddSameCurrencies()) which has some "testing logic" and a data provider method (getMoney()) which provides data. That is all.

Parametrized tests

You also get a very detailed information about the test results (in this case thanks to reasonable implementation of Money class toString() method), e.g.:

BTW. not sure if you noticed, but parametrized tests are perfect for verifying if all your "utils" methods work - you know, all this date/strings/numbers handling/transforming methods.

TestNG does not care how many parametrized tests you have in one class. See below for an example ofMoneyTest which verifies both the constructor and add() method using data providers.

TestNG offers well-thought @Before… and @After… annotations (on suite, class, group and method level).

An example below shows that a server (something heavy) is created only once before a test suite, and client (something small and cheap to create) is created before each test method.

Parameters passing

Want to:

§  configure your tests so they e.g. invoke GUI methods on different servers/ports?

§  run tests using mysql database instead of h2?

§  run GUI tests with Opera, Firefox, Chrome (or even IE 6)?

It is pretty simple to pass parameters to test methods (example copied directly from TestNG documentation):

@Parameters({ "datasource", "jdbcDriver" })

@BeforeMethod

public void beforeTest(String ds, String driver) {

    m_dataSource = ...;

    m_jdbcDriver = driver;

}

Parameters can come for example from configured Maven Surefire Plugin

Groups of tests

§  each test class (and method) can belong to multiple groups,

§  it is pretty easy to run only selected group:

·         e.g. you can run only tests from groups "smoke", or only "gui" tests,

·         it is possible to define dependencies between methods/groups (see next slide).

§  all done with annotations (e.g. @Test(groups = "smoke"))

This is also supported by tools - e.g. it is pretty simple to instruct Maven to do it (see here).

Dependencies between tests

Rather bad practice with unit tests,

but very important for integration and end-to-end tests, e.g.:

§  fail fast:

·         run Selenium tests only if application was deployed properly,

·         run full system tests only if smoke tests passed,

§  logical dependencies between tests:

·         execute shouldDeleteUserFromDatabase test only if shouldAddUserToDatabaseworked

Fail fast means that the feedback will be much quicker in case of failed tests.

Example copied from TestNG documentation.

Method method1 will be skipped (will not run) if serverStartedOk fails.

Example copied from TestNG documentation.

Method method1 will be skipped (will not run) if any method of group init fails.

It is pretty easy to write your own listeners (which act in runtime) and reporters (which are invoked after the execution).

Example output of listener which prints test result, class and method name, and execution time of each test method:

This feature is not very useful with unit-tests but can be very handy to report progress/results of integration and end-to-end tests.

There are few more interesting features of TestNG (e.g. factories) but lets skip it.

(chart from Google trends)

Well, as you can see TestNG is not really closing the gap. It rather seems to find its niche and sit there. Well, it is time to change it! :)

But who is the boss now?

All data gathered on 4th April 2012.

One could say, this all means only that JUnit development has been finished, all documentation has been written, all questions answered and easy to find by googling, all issues solved, … well, actually no. It rather looks, hm… a little bit abandonned.

Documentation

§  TestNG offers much better documentation

·         even though there are numerous articles about different versions and features of JUnit you will find it hard to find a comprehensive resource on the net which explains it thoroughly.

·         TestNG documentation is in very good shape, and always up-to date (seriously, it is!)

Have no fear

Yes, with TestNG you still can:

§  run tests with Maven, Ant, Gradle, …

§  execute tests with your IDE: Eclipse, IntelliJ IDEA, …

§  use Jenkins, Sonar, Cobertura, Mockito, FEST Fluent Assertions, Hamcrest, …

§  enjoy Spring support for testing,

§  etc., etc.

How to migrate from JUnit to TestNG?

§  There is Eclipse converter which does a decent job of changing imports etc. so you do not have to do it by hand

§  TestNG plugin for IntelliJ IDEA also provides a "Convert JUnit test to TestNG" feature

§  There might be an issue with objects creation, because TestNG does not create test classes before every test

·         This means you need to enclose creation of objects within some method.

§  Another - minor - issue is with assertions, because TestNG uses different order of arguments (first actual, then expected) than JUnit (first expected, then actual)

·         If you use matcher libraries, you do not have to change anything

The End

Great things are done by a series of small things brought together.

— Vincent Van Gogh

It is many small things which sum up and give you a much better experience.

TestNG offers support for all kind of tests. It is great, and still getting better with each new release.

…is TestNG a silver bullet? No, but close to it. :)

P.S. Want to know more about TestNG? Read the book.

List of Annotations:

@Beforesuite

You can set driver paths (System.setProperty(...)) in@BeforeSuite method. 

@BeforeMethod

Instantiate browser in @BeforeMethod.