Inconsistent filtering between ParentRunner.getFilteredChildren() and getDescription()

[paulduffin]

Background: I am working on Android and trying to use JUnitParams with the Android Test Support Library, specifically the AndroidJUnitRunner class which runs classes on the device. During that work I discovered that JUnitParams applies filtering inconsistently depending on whether it is for the getFilteredChildren() method or the getDescription() method.

In getFilteredChildren() the filter is applied to a flat list of FrameworkMethod instances, where each parameterized method has been expanded into N FrameworkMethod (where N is the number of parameter sets).

In getDescription() the filter is applied to a flat list of FrameworkMethod instances where each parameterized method is represented by a single FrameworkMethod. After the filtering has been applied then each parameterized FrameworkMethod is expanded to N Description objects.

e.g. in getDescription(), before filtering:

• method1
• method2
  After applying filter
• method1
  After instantiating
• method1
• method1[0]
• method1[1]
• method1[2]
• method1[3]
• method1[4]

in getFilteredChildren(), after instantiation, before filtering:

• method1[0]
• method1[1]
• method1[2]
• method1[3]
• method1[4]
• method2[0]
• method2[1]
• method2[2]
  After applying filter
• method1[0]
• method1[2]
• method2[1]

The filter that is being applied is the sharding filter which basically decides if a test should run with the following test, where numShards is the number of devices you have available and shardIndex is the number of the shard on the current device.

    boolean shouldRun(Description description) {
        if (description.isTest()) return (description.hashCode() % numShards) == shardIndex;
        return true;
    }

I also discovered a number of other Filter related issues.

Applying multiple Filters consecutively does not work. They are applied correctly to getFilteredChildren() but only the last filter applied affects the Description.

Calling getDescription() and then applying a Filter does not change the Description.

I have an initial change that I have created with fixes this issue, it needs some work (I think I broke JUnitParams.flat system property) but it does show the basic approach I took.

[paulduffin]

Any feedback on this?