Detailed info:
http://testng.org/doc/documentation-main.html
Presention that makes you understand TestNG better and differentiates from Junit:
source: http://kaczanowscy.pl/tomek/sites/default/files/testng_vs_junit.txt.slidy_.html#(24)
Why TestNG?
|
My (biased) observations
People
are moving from JUnit to TestNG once they start to write integration and
end-to-end tests (e.g. with Selenium). Apparently, on unit tests level you
can put up with JUnit deficiencies, but once you go outside the unit tests
realm, JUnit stands in your way.
|
|
From what I have noticed, people
who work with JUnit usually test their code with only one set of parameters.
I suspect this is because parametrized tests are unusable with JUnit. And
this is bad, because such tests are really useful.
|
|
If you really, really have to use
JUnit (do you?) then at least us JUnit Params!
|
Parametrized
tests
Parametrized
tests
@DataProvider
private static final Object[][] getMoney(){
return new Object[][] {
{new Money(4, "USD"), new Money(3, "USD"), 7},
{new Money(1, "EUR"), new Money(4, "EUR"), 5},
{new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}
@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
int expectedResult) {
Money result = a.add(b);
assertEquals(result.getAmount(), expectedResult);
}
Parametrized
tests
@DataProvider
private static final Object[][] getMoney(){
return new Object[][] {
{new Money(4, "USD"), new Money(3, "USD"), 7},
{new Money(1, "EUR"), new Money(4, "EUR"), 5},
{new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}
@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
int expectedResult) {
Money result = a.add(b);
assertEquals(result.getAmount(), expectedResult);
}
@DataProvider
private static final Object[][] getInvalidAmount(){
return new Integer[][] {{-12387}, {-5}, {-1}};
}
@Test(dataProvider = "getInvalidAmount", expectedExceptions = IllegalArgumentException.class)
public void shouldThrowIAEForInvalidAmount(int invalidAmount) {
Money money = new Money(invalidAmount, VALID_CURRENCY);
}
@DataProvider
private static final Object[][] getInvalidCurrency(){
return new String[][] {{null}, {""}};
}
@Test(dataProvider = "getInvalidCurrency", expectedExceptions = IllegalArgumentException.class)
public void shouldThrowIAEForInvalidCurrency(String invalidCurrency) {
Money money = new Money(VALID_AMOUNT, invalidCurrency);
}
Test
Fixture Setting
@Test
public class TestFixtureTest {
private Server server;
private Client client;
@BeforeSuite(timeOut = 1000)
public void setUpServer() {
server = new Server();
server.start();
}
@BeforeMethod
public void setUpClient() { client = new Client(); }
// some test methods here
@AfterSuite
public void shutDownServer() { server.stop(); }
}
Logical dependencies gives you a much more realistic error information - you learn that
1 tests has failed and 99 has been skipped, which is much easier to fix than
the information about 100 failed tests (OMG!
what’s going on!? All tests failed…. aaarrhghg!).
Dependencies
between tests
@Test
public void serverStartedOk() {}
@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}
Dependencies
between tests
@Test(groups = { "init" })
public void serverStartedOk() {}
@Test(groups = { "init" })
public void initEnvironment() {}
@Test(dependsOnGroups = { "init.* })
public void method1() {}
Listeners,
Reporters
OK FirstTest fifthTestMethod 1944
OK FirstTest firstTestMethod 3799
OK FirstTest fourthTestMethod 1920
OK FirstTest secondTestMethod 4891
OK FirstTest thirdTestMethod 2963
OK SecondTest methodA 3525
OK SecondTest methodB 1390
FAILED SecondTest methodC 117
SKIPPED SecondTest methodD 23
OK SecondTest methodE 4552
There
is more of this
Who is
the boss?
|
JUnit
|
TestNG
|
releases since 2009 (not counting betas)
|
|
|
mailing list
|
|
|
While better support for
integration and end-to-end tests is definitely true, not many people know that TestNG offers more than
JUnit also in case of unit tests.
No need to create own Thread objects!
Use annotations and that is it!
An example of running same
test method simultaneously (20 times, by 3 threads):
@Test(threadPoolSize
= 3, invocationCount = 20)
public
void concurrencyTest() {
System.out.print(" " +
Thread.currentThread().getId());
}
Result:
13
12 13 11 12 13 11 12 13 12 11 13 12 11 11 13 12 11 12 13
In general, it is a good
practice, to test your code with different sets of parameters:
§ expected
values: sqrt(4), sqrt(9),
§ boundary
values: sqrt(0),
§ strange/unexpected
values: sqrt(-1), sqrt(3)
A good testing framework
should make it simple, and avoid copy&paste coding.
Alas, the default JUnit
implementation of the parametrized tests is rather unusable… In short, JUnit
expects all test methods to be parameter-less. Because of this constraint, parametrized
tests with JUnit looks awkward, because parameters are passed by constructor
and kept as fields!
Parametrized tests are very simple
with TestNG (no constructor awkwardness).
You can have as many data providers
in one class as you wish. You can reuse them (call them from other classes),
and you can make them "lazy", so each set of parameters is created
when required.
Once you migrate to TestNG you will
notice that you start writing a lot of parametrized tests. Because they are so
very useful!
Ok, so this is how it looks with
TestNG. There is a test method (shouldAddSameCurrencies()) which has
some "testing logic" and a data provider method (getMoney()) which provides
data. That is all.
You also get a very
detailed information about the test results (in this case thanks to reasonable
implementation of Money class toString() method), e.g.:
BTW. not sure if you
noticed, but parametrized tests are perfect for
verifying if all your "utils" methods work - you know, all this
date/strings/numbers handling/transforming methods.
TestNG does not care how many
parametrized tests you have in one class. See below for an example ofMoneyTest which verifies both the constructor and add() method using data providers.
TestNG offers well-thought @Before… and @After… annotations (on suite, class, group and method
level).
An example below shows that a server
(something heavy) is created only once before a test suite, and client
(something small and cheap to create) is created before each test method.
Want to:
§ configure
your tests so they e.g. invoke GUI methods on different servers/ports?
§ run tests
using mysql database
instead of h2?
§ run GUI
tests with Opera, Firefox, Chrome (or even IE 6)?
It is pretty simple to pass
parameters to test methods (example copied directly from TestNG documentation):
@Parameters({
"datasource", "jdbcDriver" })
@BeforeMethod
public
void beforeTest(String ds, String driver) {
m_dataSource = ...;
m_jdbcDriver = driver;
}
Parameters can come for
example from configured Maven Surefire Plugin
§ each test
class (and method) can belong to multiple groups,
§ it is
pretty easy to run only selected group:
·
e.g. you can run only tests from groups "smoke", or only "gui" tests,
·
it is possible to define dependencies between methods/groups (see
next slide).
§ all done
with annotations (e.g. @Test(groups =
"smoke"))
This is also supported by
tools - e.g. it is pretty simple to instruct Maven to do it (see here).
Dependencies between tests
Rather bad practice with unit tests,
but very important for integration and end-to-end tests, e.g.:
§ fail fast:
·
run Selenium tests only if application was deployed properly,
·
run full system tests only if smoke tests passed,
§ logical dependencies between
tests:
·
execute shouldDeleteUserFromDatabase test only if shouldAddUserToDatabaseworked
Fail
fast means
that the feedback will be much quicker in case of failed tests.
Example copied from TestNG
documentation.
Method method1 will be skipped (will not run) if serverStartedOk fails.
Example copied from TestNG
documentation.
Method method1 will be skipped (will not run) if any method of
group init fails.
It is pretty easy to write your own
listeners (which act in runtime) and reporters (which are invoked after the
execution).
Example output of listener which
prints test result, class and method name, and execution time of each test
method:
This feature is not very useful with
unit-tests but can be very handy to report progress/results of integration and
end-to-end tests.
There are few more interesting
features of TestNG (e.g. factories) but lets skip it.
Well, as you can see TestNG is not
really closing the gap. It rather seems to find its niche and sit there. Well,
it is time to change it! :)
All data gathered on 4th
April 2012.
One could say, this all
means only that JUnit development has been finished, all documentation has been
written, all questions answered and easy to find by googling, all issues
solved, … well, actually no. It rather looks, hm… a little bit abandonned.
§ TestNG
offers much better documentation
·
even though there are numerous articles about different versions
and features of JUnit you will find it hard to find a comprehensive resource on
the net which explains it thoroughly.
Yes, with TestNG you still
can:
§ etc.,
etc.
How to migrate from JUnit to TestNG?
§ There is Eclipse converter which
does a decent job of changing imports etc. so you do not have to do it by hand
§ There
might be an issue with objects creation, because TestNG does not create test
classes before every test
·
This means you need to enclose creation of objects within some
method.
§ Another -
minor - issue is with assertions, because TestNG uses different order of
arguments (first actual, then expected) than JUnit (first expected, then actual)
·
If you use matcher libraries, you do not have to change anything
Great
things are done by a series of small things brought together.
— Vincent Van Gogh
It is many small things
which sum up and give you a much better experience.
TestNG offers support for
all kind of tests. It is great, and still getting better with each new release.
…is TestNG a silver bullet? No, but
close to it. :)
P.S. Want to know more
about TestNG? Read the book.
List of Annotations:
@Beforesuite
You can set driver paths (System.setProperty(...)) in@BeforeSuite method.
@BeforeMethod
Instantiate browser in @BeforeMethod.
