Update TestSuite class to report test-failure with correct class/method name instead of 'warning(junit.framework.TestSuite)'

[kishoretak]

The framework reports the test-class constructor initialisation with a hardcoded method name warning and class-name as TestSuite class and ignore the actual method name and class-name.
While its somewhat easy to find the issue locally when there are a few test-failure but in-case of CI pipeline where thousands of test are running and we rely on XML report, its not possible to differentiate between such test-failures and auto-assign to test-class owner as test-class itself is missing.
The class does optimise when the non parameterised constructor is failing by reporting a single test-failure instead of reporting all the test-methods with same error and there is might still be okay to keep a dummy test-name, however it should use the correct class-name.
And incase of non default constructor failure, it should report with correct test-name as well along with test-class name.

[kishoretak]

@marcphilipp please review the issue and I can start work on the PR, thanks.

[marcphilipp]

JUnit 4 is in maintenance mode and we're ship another release unless there's a critical security issue.

That being said, why/how are you using JUnit 3's TestSuite? Could you please post a complete example so we can take a look and, potentially, recommend a replacement?

[kishoretak]

These are legacy test in the system still using Junit3, below is the sample code where init of SampleUtil2 fails.

public class SampleTest extends TestCase {
   
    private final SampleUtil util = new SampleUtil(SampleUtil2.geSomeObject());

    
    public SampleTest(String name) {
        super(name);
    }
    
    public void testSample() throws Exception {
       ...
    }

[kcooney]

I know this is likely not the response you are looking for, but you should avoid having any code that could fail run at TestCase construction time. In the above example, you should do:

public class SampleTest extends TestCase {
   
    private SampleUtil util;

    @Override
    protected void setUp() throws Exception {
        super.setUp(); 
        util = new SampleUtil(SampleUtil2.getSomeObject());
    }
    
    public void testSample() throws Exception {
       ...
    }

Aside: One of the many advantages of JUnit 4 and JUnit 5 is that the pattern your example test uses will work like you expect (so you can reuse your best practices from writing Java when writing your tests).

It's likely you'll get a much better output if you use JUnit 5 to run the tests. That shouldn't require migrating all of your tests away from JUnit 3 style right away. I recommend moving to a more modern framework for running your tests (maven, gradle, etc). See Build Support in the JUnit 5 User Guide for details.

If moving to JUnit 5 is too big of a leap at the moment, you could try running the tests via the JUnit 4 APIs, which I believe will also produce better error messages. Perhaps you could update the code your CI runs to execute tests using something like this:

import org.junit.runner.JUnitCore;
import org.junit.runner.Result;

public class Runner {

    public static void run(Class<?>... classes) {
        JUnitCore core = new JUnitCore();
        core.addListener(new MyXmlRunListener());
        Result result = core.runClasses(classes);
        System.exit(result.wasSuccessful() ? 0 : 1);
    }
}

[kishoretak]

Thanks @kcooney for detailed explanation. We are using junit5 framework to run the test and have all versions (junit3/4/5) in the syetem.
In junit5, even if we add custom listener, it will not solve the problem as the test-name would be still 'warning'.

I understand the isssue is with the way tests are written and could be fixed even without migrating to junit4/5 as you mentioned in the example. The problem is with reporting such test to respective team so they can fix the same which requires atleast correct class-name if not test-name.