Android end-to-end test automation

Recapping: In the A journey through end-to-end tests article (Please, read it first) we was introduced to the end-to-end, user journey test or smoke tests terms. We also saw how to write test scenarios using Gherkin Syntax to structure our Domain Specific Language and here we are...

Looking to the following scenario, how can we automate it?
Given my application is installed
And I tap on the app icon
When the app is launched
Then I see the Login screen

Rules

By observing the given test scenario It's possible to extract some rules and infer some others.

• Test scenarios are composed by ordered steps

• Each step should be asserted
  It's possible to find the exact step that caused a failure.

• Each step execution result should be printed
  As each step should be asserted, each step execution result should also be printed as It is described in the test scenario (e.g. And I tap on the app icon FAILED).

• Test scenarios should also be ordered
  As we're simulating the real world, it should not be possible to open the account screen before authenticate an user (If the application requires authentication, of course)

Now that we have the test rules, let's select the tools that we're going to use to automate our tests.

Tools

On android we have two kind of instrumentation tools that helps to simulate the user interactions, Espresso and UiAutomator.
With espresso we have a sort of gray test box, as it allows the developer to access and modify some application values, on the other hand the UiAutomator simulate a real black box, where we interact with the app and the system (not possible on espresso) as a real user, and that's why we'll choose the UiAutomator.

Ok, what about the Junit? Is it possible to use Junit5? May it have some new feature that can help? Yes, it does, but the Junit5 is not officially supported on android (They also don't have plans for that) and the setup is not simple for the instrumentation tests. Let's keep using the Junit4 instead of adding a bunch of unnoficial library to our project.

And what about cucumber? You can give it a try, but It's not easy to configure, especially with UiAutomator. We can have something simpler by creating a custom test runner with the Junit4, let me show you how.

Test Runner

We have three important rules set for our test steps, they should be ordered, asserted and properly logged, but Junit4 doesn't offer the @Order and @DisplayName annotations as Junit5, so, how we can comply with these step rules?

On Junit4 we have the possibility to extend BlockJUnit4ClassRunner and override the getChildren and describeChild methods, let me explain...

The test methods in the Junit4 are filtered/found by the @Test annotation inside the Junit Test Runner, this process happens in the getChildren method, and then each method found is executed by calling the runChild (Undefined execution order), which also calls the describeChild method to define the test name to be displayed. Now we have all the information needed to modify the Junit Runner as we want.

So, the @Test annotation is just an information to Junit, and the same is valid for the @Ignore annotation, that tells Junit to just logging the given method as ignored instead of running it. With that, we can also add more information to the test method, as telling Junit the sort order of a test for example.

So let's create an annotation called @Order and try it.

@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION)
annotation class Order(@IntRange(from = 0L, to = 100) val value: Int = 0)

Now it's time to create our custom test runner extending BlockJUnit4ClassRunner

class AutomatorRunner(
    private val testClass: Class<*>
) : BlockJUnit4ClassRunner(testClass)

And then we can sort the test methods by overriding the getChildren method for that

 override fun getChildren(): MutableList<FrameworkMethod> {
    return sortTestClassMethods()
         .map { FrameworkMethod(it) }
         .toMutableList()
}

private fun sortTestClassMethods(): List<Method> {
    return testClass.methods
        .filter { it.getAnnotation(Test::class.java) != null }
        .sortedBy { it.getAnnotation(Order::class.java)?.value ?: 0 }
}

In the code above we first filter the existent test (annotated with Test::class) methods inside the Test Class sorting them using the Order::class annotation value.

Now let's create our test class using our the AutomatorRunner along with Order annotation.

@RunWith(AutomatorRunner::class)
class ApplicationStartScenario {
    @Test
    @Order(1)
    fun givenMyApplicationInstalled() ...

    @Test
    @Order(2)
    fun andITapOnTheIcon() ...

    @Test
    @Order(3)
    fun whenTheApplicationIsOpened() ...

    @Test
    @Order(4)
    fun thenISeeTheLoginScreen() ...
}

Running the test above will give us the following output:

givenMyApplicationInstalled SUCCESS
andITapOnTheIcon SUCCESS
whenTheApplicationIsOpened SUCCESS
thenISeeTheLoginScreen SUCCESS

Not that readable, right? Let's fix it. We need a display name and we have the option to create a DisplayName annotation or enhance the Order annotation, which is the selected approach here.

The @Order annotation can be renamed to @Step, as our test Scenarios are composed by Steps, then we just need to add the displayName: String field to the Step::class

@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.FUNCTION)
annotation class Step(
    val displayName: String,
    @IntRange(from = 0L, to = 100) val order: Int = 0
)

To modify the test method display name we need to override the describeChild method

override fun describeChild(frameworkMethod: FrameworkMethod): Description {
    val displayName = getDisplayName(frameworkMethod.method)

    return Description.createTestDescription(
        testClass.name,
        displayName,
        frameworkMethod.method.annotations
    )
}

private fun getDisplayName(method: Method): String {
    val annotation = method.getAnnotation(Step::class.java)
    return annotation?.displayName ?: method.name
}

That's it, if the method don't have the @Step annotation we return the test method name, if it's annotated, then we return the displayName value from @Step.

Here's the final version of our Automator Class

class AutomatorRunner(private val testClass: Class<*>) : BlockJUnit4ClassRunner(testClass) {

    override fun getChildren(): MutableList<FrameworkMethod> {
        return sortTestClassMethods().map { FrameworkMethod(it) }.toMutableList()
    }

    override fun describeChild(frameworkMethod: FrameworkMethod): Description {
        val displayName = getDisplayName(frameworkMethod.method)

        return Description.createTestDescription(
            testClass.name,
            displayName,
            frameworkMethod.method.annotations
        )
    }

    private fun sortTestClassMethods(): List<Method> {
        return testClass.methods
            .filter { it.getAnnotation(Test::class.java) != null }
            .sortedBy { it.getAnnotation(Step::class.java)?.order ?: 0 }
    }

    private fun getDisplayName(method: Method): String {
        fun formatMethodName(name: String): String {
            return name.replace("_", " ")
        }

        val annotation = method.getAnnotation(Step::class.java)
        return annotation?.displayName ?: formatMethodName(method.name)
    }
}

Time to update our ApplicationStartScenario test class

@RunWith(AutomatorRunner::class)
class ApplicationStartScenario {
    @Test
    @Step("Given My Application Installed", 1)
    fun givenMyApplicationInstalled() ...

    @Test
    @Step("And I tap on the app icon", 2)
    fun andITapOnTheIcon() ...

    @Test
    @Step("When the app is opened", 3)
    fun whenTheApplicationIsOpened() ...

    @Test
    @Step("Then I see the Login Screen", 4)
    fun thenISeeTheLoginScreen() ...
}

The test example above would print:
Given My Application Installed SUCCESS
And I tap on the app icon SUCCESS
When the app is opened SUCCESS
Then I see the Login Screen SUCCESS

Cool, but how we can have Ordered Scenarios? Test Suite

JUnit Test Suite

Ordered scenarios means that we should call the test classes in a sort order, and Junit4 already give us this feature, the Test Suite, here's how to use it..

@RunWith(Suite::class)
@Suite.SuiteClasses(
    ApplicationStartScenario::class,
    LoginScenario::class,
    HomeScenario::class
)
class SmokeTestSuite

That's it, the class order that goes inside @Suite.SuiteClasses represents that execution order of the SmokeTestSuite.