IssueTracker

Hi Lukasz,

This is actually working as intended.

There was an ambiguity on passing the instrumentation test runner and the test runner arguments, since it could be passed either in the run configuration or in the build.gradle file. In order to prevent misbehavior caused by this ambiguity, we kept the possibility of passing these parameters only by the build.gradle file, being able to keep the coherence of results when running tests through the command line and through Studio.

You can pass the instrumentation runner by setting the parameter testInstrumentationRunner and the arguments by setting the parameter testInstrumentationRunnerArguments. For instance, your build.gradle file can contain:

android {
    defaultConfig {
        testInstrumentationRunner "android.test.InstrumentationTestRunner"
        testInstrumentationRunnerArguments size:"medium", foo:"bar"
    }
}

For further information, have a look at: https://google.github.io/android-gradle-dsl/current/com.android.build.gradle.internal.dsl.ProductFlavor.html#com.android.build.gradle.internal.dsl.ProductFlavor:testInstrumentationRunner

Hello,

Thank you for the response.

I agree this ambiguity may cause some problems with test results while running through Studio vs command line. However this solution decreases flexibility of run configuration by forcing gradle sync for each argument change.

Consider this case:
We have a test suite that is capable of working with both real and mocked API. This behaviour was controlled by a flag that was passed as runner argument. In AS 2.2.x switching between these environments was just a matter of selecting a different run configuration, whereas in 2.3.x we have to either modify build.gradle to change parameters or create different flavours/buildTypes with different values. Both build.gradle modification and selecting buildType in Studio requires gradle sync (!), which is much less convenient than selecting a predefined configuration from a dropdown list.

Even though setting extra arguments via build.gradle is a nice feature, it shouldn't prevent developers from running a desired configuration without having gradle sync after each argument change.

Hi,

I see your use case and it definitely makes sense.

We had to disable this option in the configuration because it was causing more troubles than solving for most users, but since it can be useful for some other users (as the case you suggested), I will keep this bug open and for the next version we'll try to support both with a kind of merge between the arguments set in gradle file and in the configuration. For awhile I would suggest to run the tests through the command line passing the arguments if sync takes too much time in your project.

I'd also like another feedback from you: do you consider useful being able to set also the InstrumentationRunner (not only the arguments) in the configuration?

Hi,
Same issue here. In my case the scenario is a bit different though.

I am using additional test runner arguments in run configuration options in AS to specify which tests to run with Cucumber. I pass a Cucumber tag as an extra argument which is then consumed by my instrumentation test runner which uses it to run only scenarios with the given tag.

This is useful for development if you have many scenarios and want to run only the selected ones to save time. Having this in build.gradle would mean having to sync project + remembering not to actually commit this since I would like to run all the tests by default.

I could move this to e.g. local.properties & have build.gradle take it from there, however the previous solution which was in AS 2.2 seemed much more convenient.

Hi,

Regarding the question: "do you consider useful being able to set also the InstrumentationRunner (not only the arguments) in the configuration?".

Honestly, it's hard to say. From one point of view, it may be useful while using tools like cucumber together with espresso (as cucumber requires to delegate some onCreate/onStart stuff to it's own runner). So in case of having 2 different tests suites (espresso and (cucumber + espresso)) it's much easier to have two runners.

On the other hand, above problem can be achieved with one "delegating" runner that is selecting of the other runners depending on extra argument. Moreover, (from what I know) build.gradle does not allow two runners at this moment, so in order to have them both working you would need to specify them in manifest manually (I'm not sure how gradle cAT will behave then).

But having two runners introduces even more ambiguity: what should happen after running tests using gradle task? Running default runner? Running both of them and combining results? And what about application? Kill & restart app for each runner?

So given this, I think multiple runners introduces even more (not so easy to solve) problems, and if anybody requires that complex behaviour custom implementation of one runner is a way to go.

Do we still want to implement this? I think this is low hanging fruit: show extra options and instrumentation runner fields in runner config editor always. When specified, override value otherwise use gradle's.

Yes please. 😄

+1! 😊

Fixed by https://android.googlesource.com/platform/tools/adt/idea/+/07db25399e807f495eaabff5b613ef38a0c13787

Thanks guys!