Test annotated with `@Test(enabled=false)` and `@Ignore` should be reported as skipped

[marcphilipp]

Currently such tests are not reported at all.

Related issue: gradle/gradle#1403

[marcphilipp]

I briefly looked into this. It seems methods with @Test(enabled=false) are available as part of ITestClass.getTestMethods but @Ignored ones are not.

[marcphilipp]

@juherr @krmahadevan Is that difference on purpose or accidental?

[juherr]

@Ignore is supposed to disable @Test thanks to an internal listener.
The only expected behavior is that @Ignore on a class will disable enabled @Test on methods but @Test(enabled=false) won't.

About reporting, I don't know what is the best option but not reported because disabled looks legit.

[marcphilipp]

When I'm in an implementation of IClassListener.onBeforeClass and call testClass.getTestMethods(), the returned array will contain an instance of ITestNGMethod for methods annotated with @Test(enabled=false) but not for methods annotated with @Ignore. Hence, we could report the former as skipped in the TestEngine but not the latter which would be inconsistent. I'm wondering why @Ignore methods are not accessible via testClass.getTestMethods(), though. 🤔

@krmahadevan Do you know?

[krmahadevan]

@marcphilipp - @Ignore on a class level is basically equivalent to having @Test(enabled=false) on all the test Methods.

That is why @Ignore at class level and @Test(enabled=false) for all test methods in the test class behave the same way i.e., testClass.getTestMethods() returns null.

The below example should demonstrate the behavior.

import org.testng.Assert;
import org.testng.IClassListener;
import org.testng.ITestClass;
import org.testng.ITestNGMethod;
import org.testng.TestNG;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Ignore;
import org.testng.annotations.Test;

public class LocalTestRunner {

  @Test(dataProvider = "testData")
  public void demonstrateBehavior(Class<?> clazz, boolean isNull) {
    TestNG testng = new TestNG();
    testng.setTestClasses(new Class[]{clazz});
    testng.setVerbose(2);
    LocalClassListener listener = new LocalClassListener();
    testng.addListener(listener);
    testng.run();
    Assert.assertEquals(listener.getTestMethods() == null, isNull);
  }

  @DataProvider(name = "testData")
  public Object[][] getData() {
    return new Object[][]{
        {TestClassWithAllTestsExplicitlyDisabled.class, true},
        {TestClassUsingIgnoreToDisableAllMethods.class, true},
        {TestClassWithMixedCombinations.class, false}
    };
  }

  public static class TestClassWithAllTestsExplicitlyDisabled {

    @Test(enabled = false)
    public void test1() {
    }

    @Test(enabled = false)
    public void test2() {
    }
  }

  @Ignore
  public static class TestClassUsingIgnoreToDisableAllMethods {

    @Test
    public void test1() {
    }

    @Test
    public void test2() {
    }
  }

  public static class TestClassWithMixedCombinations {

    @Test
    public void runnableTest() {
    }

    @Test(enabled = false)
    public void disabledTest() {
    }

    @Ignore
    @Test
    public void ignoredTest() {
    }
  }

  public static class LocalClassListener implements IClassListener {

    private ITestNGMethod[] testMethods;

    @Override
    public void onBeforeClass(ITestClass testClass) {
      testMethods = testClass.getTestMethods();
    }

    public ITestNGMethod[] getTestMethods() {
      return testMethods;
    }
  }
}

[juherr]

All the @Ignore logic is based on a listener.

So, it sounds like an issue when listeners are involved.

We should investigate and fix it on TestNG side.

[krmahadevan]

@juherr - Currently Testng is behaving the same way when

• @Ignore used on class or
• Test class has all test methods as disabled

That was what my sample was trying to show.

Since in the second case ( enabled=false) there is no listener involvement I didn't quite get you.

[marcphilipp]

@krmahadevan Thanks for the sample!

Given the status quo, I don't see how the engine could reliably report tests annotated with @Test(enabled=false) or @Ignore as skipped since they're only available from onBeforeClass in the TestClassWithMixedCombinations case above (and even then only the methods with @Test(enabled=false) not the ones with @Ignore).

[juherr]

That was what my sample was trying to show.

@krmahadevan Right! I didn't realize your sample already highlights similar behavior between @Ignore and @Test(enabled=false).

only available from onBeforeClass in the TestClassWithMixedCombinations case above (and even then only the methods with @Test(enabled=false) not the ones with @Ignore)

but it looks like some not-expected differences exist.

I don't see how the engine could reliably report tests annotated with @Test(enabled=false) or @Ignore as skipped

@marcphilipp Like https://github.com/cbeust/testng/blob/master/testng-core/src/main/java/org/testng/internal/annotations/IgnoreListener.java I think you can collect all methods into an IAnnotationTransformer and mark the not-run methods as skipped at the end of the run.

[marcphilipp]

Thanks for the suggestion! That could be done but it would mean somewhat duplicating TestNG's internals (i.e. what it considers a test method etc.) since the code currently gets most of its info from ITestNGMethod.

[juherr]

@krmahadevan @marcphilipp
If it is possible, WDYT if we add an internal listener which notifies every discovered ITestNGMethod?

[krmahadevan]

@marcphilipp - Can you please tell me if this is just a reporting requirement ? If yes, then can we consider building something on top of the IReporter interface that I believe will report skipped tests as well. Would that work here ?