« The Case Against XP: evidence please | Main | That is the sound of inevitability: Linux on the desktop »

More on Ant's JUnit task

Commenting on the criticism of the JUnit task. Chuck Hinson pointed out that he sets the printsummary attribute. to "no". I do know this does reduce the verbiage, but highlights another problem with task - it swallows the JUnit diagnostic output for a failure.

Here's some failing code using the jar I mentioned:

  test:
.F.....................
Time: 4.026
There was 1 failure:
1) testHome(net.amadan.beats.BeatsTest)
junit.framework.AssertionFailedError: you can see me
at net.amadan.beats.BeatsTest.testHome(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
FAILURES!!!
Tests run: 21, Failures: 1, Errors: 0

Here's the failure case using the JUnit task with printsummary set to "no":

  junit-test:
TEST net.amadan.beats.BeatsTest FAILED
BUILD FAILED

This output is useless. What failed?

Now again maybe there's a flag for this. But the point is that the task is by default swallowing JUnit's useful output, and replacing it with something that is not. This is no better than eating an exception trace, in fact that's exactly what's going on. This also happens with the verbose mode. The irony is that for all its verbiage, the task doesn't actually tell you what you need to know.

Later: Got a suggestion from Mike Clark, who can see traces by tuning the formatter:

  <formatter type="plain" usefile="false" />

Obvious ;) The traces are there now, thanks Mike (should I put a FAQ entry together for using the task like this?). I'm still getting a summary for each Test file though -why can't it default to JUnit output (arrghhh!). Think I'll shutup now and stick to my java hack, which is simple enough for my tiny brain!


August 16, 2003 10:29 AM

Comments

Margaret Green
(August 16, 2003 02:41 PM #)

Hi Bill.

Have you tried using the junitreport feature of ant? You can get test output in XML withr the formatter attribute of the junit ant task. Then use an ant junitreport task to aggregate test results into an HTML report.

I like to use Junit integration in IntelliJ IDE, click through from Junit error and failure output directly to the line of code.

I use the ant junit task for build testing, not development.

Mike Clark
(August 16, 2003 05:32 PM #)

Hi Bill,

I had to try this myself, since I have printsummary="false" by default. I do see the stack trace, but perhaps it's because I also use:

Rafael Alvarez
(August 16, 2003 09:04 PM #)

Overall I don't like the junit task for the very same reason: Why do I need to go throught all the troubles to generate a complete reports of all my test to know which test failed and why?

Using unit test is about getting feedback, and getting it as soon as possible. If I have a suite of 100 test (and that's a very small number, believe be) that ran in about 3 sec, and I need to run a report-collecting task that take about 4 more seconds, and then I have to start a browser to read it that take about 20 seconds, and browse the page (ok, 2 sec at most at that), then I have to wait about 29 seconds to know what test failed and why, being 26 of them just a waste of time.

Of course, you can reduce this time using the plain formatter and checking only the generating files that correspond to the failed test... But not by much.

I found that is faster to just run the TextRunner from a java task than to use the junit task. The overhead of using the junit task surpases the overhead of the java task.

Chuck Hinson
(August 18, 2003 01:07 AM #)

Yes, the output is useless in terms of figuring out what went wrong.

But (at least when it comes to tests), I find the stack traces are overkill. I almost never care about the whole trace - usually the first line is sufficient to determine what test failed.

I've always used log4j to track progress through my tests. I don't recall exactly why I started doing it this way, and I wont defend it, but it has served me relatively well.

Trackback Pings

TrackBack URL for this entry:
http://www.dehora.net/mt/mt-tb.cgi/1067