Understanding the Essentials of Mobile Test Automation

Preface

When it comes to test automation in general, it's easy to say that it's complex. If we were to stick to the classic separation of the Test Pyramid, we'd end up with:

- Unit tests, which can be implemented for both backend and frontend.

- Integration tests, which today can mean anything from API testing, microservices and third-party service integration, user authentication and authorization, to quasi-performance testing.

- End-to-end tests, which for the vast majority of test automation engineers and software developers would mean web application testing (to name a few major frameworks: Playwright, Cypress, Selenium).

There are also tools like Pact for contract testing, Storybook for advanced visual and/or snapshot testing combined with interactive documentation, and probably many more that I don't even know about.

However, these tools are well documented and have huge communities around them, and to be completely honest - I don't think any of these tests are hard (except maybe the contract tests).

When I started working on mobile test automation, I quickly realised that this was not the case. The more I got into mobile software development, the more confusing it became, the mere amount of dependencies and requirements to just get the application running was really bizarre, let alone automating tests on it. The documentation on the various technologies was sometimes not very helpful, and there were no articles on the full spectrum of possible mobile environments.

So let's fix that and talk about mobile test automation, shall we?

Development Frameworks Overview

First, before we go any further, we need to consider what technologies will be used to build the applications that will be tested with the mobile testing frameworks. The popular frameworks for mobile development are:

According to the 2024 Stack Overflow survey, Flutter is the most popular framework for cross-platform development. It's followed by React Native, Electron (not listed in our table because it only supports desktop app development), .NET MAUI (and it's older brother Xamarin, which has been replaced by MAUI), Ionic, and Cordova closing the list. Interestingly, there is no mention of NativeScript at all.

Mobile Test Automation Frameworks Overview

Below is a general overview of the test automation frameworks. There are some other frameworks that I have intentionally left out because they do not seem to be very popular (e.g. EarlGray, which is Google's framework for iOS automation), and they are most likely wrappers of the technologies listed in the table anyway (e.g. Selendroid or any of the cloud mobile test automation providers).

Appium-based Frameworks Overview

From the above list, Appium can be considered as the most comprehensive and therefore the most complicated framework of them all. In short, it consists of four basic elements: Appium Core, Drivers, Clients and Plugins.

We won't go into the details of what each element stands for, but to give you a better understanding of what technologies are supported by the Appium framework, here is a list of so-called Clients:

Scenario-Based Guide to Selecting Mobile Testing Frameworks

When you would have to make one of the key decisions in the context of testing strategy, that is, the choice of a framework for automating mobile end-to-end testing the choice is quite complicated, as it depends on many factors.

Let's do a little thought experiment and try to describe some possible scenarios to help you decide.

TL;DR:

• When developers are using native development tools and want to maintain grey/white box end-to-end testing and/or you’re building an MVP: Espresso (for Android) / XCUITest (for iOS)

• If you're developing an MVP and you only need UI testing (without physical iOS devices and with limited scripting possibilites): Maestro

• If you're developing a cross-platform React Native application and want to improve test scripting experience with Large Language Models: Detox

• For long-term projects that require stable solutions: Appium-based platforms

Scenario 1: Android or iOS Only

The first scenario is an example for likely early stages of development (e.g. seed stage of a startup) when the focus is on only one mobile platform. This means that the natural choice should be either Espresso (for Android) or XCUITest (for iOS).

However, there may be other things to consider, such as:

1. Resource-Constrained Environments:

For example, if the application is being developed on a tight schedule and/or budget, and the developers simply do not have the time to maintain the end-to-end test suites, choosing complex Espresso or XCUITest frameworks can end up being a disaster.

In this type of environment, I would suggest using Maestro as it seems to be the most hassle-free and does not require too much time and effort to maintain test suites and configuration.

2. Future Proofing for Long-Term Projects:

If your project's time and budget constraints are not the things that keep you up at night, then you should probably think a little more about the potential development path.

I want to emphasise that we should avoid over-engineering, but on the other hand, it's also important to choose a framework that we can use in a more mature test environment. By a more mature test environment I mean things like automated test data generation and cleanup (which can be achieved with API-based model) or implementation into the existing CI/CD pipelines. And of course we want these things to be relatively easy to implement.

In summary, if you are sure that your project is entirely Android or iOS based, developers are the ones who will write and maintain the test suites, and you do not plan to extend it to the opposite technology, I would suggest using Espresso or XCUITest, as they will give you the most solutions adapted to your existing codebase. Otherwise, I'd lean toward the Appium-based solutions because they're much more extensible and can be used in black-box environments.

Scenario 2: Android and iOS

Most of the time, companies want to target the broader audience, and since Android has ~72%* of the market share, which means that iOS users have ~28%* of the market share, it means that most companies want to target both platforms.

This scenario, at least from my perspective, is much more complex in terms of things that need to be considered before an informed decision can be made.

1. Separate development teams for Android and iOS

   

If your application is developed by separate teams using native development tools (i.e. Android Studio and/or Xcode), you can probably integrate Espresso and XCUITest to perform end-to-end testing. However, I would like to point out that this may be a subpar solution, as other frameworks allow you to test both platforms using the same test scripts.

So, for the situation mentioned in this sub-scenario, I'd suggest using Espresso and XCUITest for smaller scale tests (component testing if you will) that would be maintained by the developers themselves and integrating broader scoped frameworks for end-to-end testing (i.e. Appium or Maestro).

2. Cross-platform development

   

As you can see in the table mentioned in the Development Frameworks Overview section, there are many possible frameworks to choose from to develop cross-platform applications. Most frameworks support all platforms (mobile, web, and desktop), but when it comes to mobile development, we focus mostly on Android and iOS.

Unfortunately, in order to figure out which framework we should choose to write and maintain end-to-end mobile tests, we need to proceed with the sub-categorisation of possible needs and settings.

• If you really don't have the time to maintain test suites, and therefore need something simple, almost effortless to write and maintain, I would suggest going with the Maestro.

• If you are an SDET or an experienced test automation engineer, you may want to lean more towards Appium-based tools, as Appium gives you all the benefits of mobile test automation as well as the freedom to configure the entire test automation framework yourself. I'm not saying that Maestro can't do this, but please take a look at the Maestro Afterthoughts for more insight.

3. Cross-platform development with React Native

   

The tool I haven't mentioned yet is Detox. This is because Detox is specifically designed for grey-box end-to-end testing for React Native. I'd recommend giving it a try if you're developing a React Native application, especially given the fact that it's the first known framework to incorporate the Large Language Models to write tests in natural language.

However, if you and your team are developing applications for external customers (so-called enterprise-grade applications) that require stable tools, I'd strongly recommend using Appium-based tools because they are much better documented and have much larger communities built around them.

Scenario 3: Android, iOS Web and More

The last scenario we'll talk about is native mobile development combined with any other version of the app. This is paradoxically the easiest of all.

Appium.

There is simply no other solution that would allow you to test on all of the available platforms.

Of course, if you'd like, you can mix and match Maestro for mobile testing and Playwright for web testing. Or Detox for mobile, Appium for smart TV, and Selenium for web, if that's what you want. It just doesn't seem like a good idea, especially since some of the Appium-based frameworks allow you to create a single test script that would run sequentially on multiple platforms.

For example, WebdriverIO allows you to do something on one mobile device, then switch to a separate web application to change/verify values and perform some other actions on another mobile device.

Maestro Afterthoughts

I felt that Maestro might seem like a great solution in the end, which it can be in some cases, but I also want you to think about possible downsides if you decide to go with it.

Maestro gives you the ability to configure the test automation framework (e.g. run JavaScript code), but since it's written with a specific architecture in mind, it's more of an add-on to the standard YAML flow files than anything else.

Another thing to keep in mind when talking about the Maestro is last year's Cypress intellectual property update case. In a nutshell, the Cypress team removed the ability to use third-party services for dashboards that were previously free, essentially forcing companies to use their own cloud platform.

Not only is that platform not free, but it also prevents Cypress from being used where contracts with customers specifically block the use of cloud platforms due to GDPR or similar laws. I mention this because Maestro has its own cloud platform for running tests called Robin, and it's possible that the similar scenario to the Cypress update could happen to Maestro as well.

Development for the Apple ecosystem

The last thing I wanted to mention here is that if you want to write tests for the Apple ecosystem, namely iOS, iPadOS, macOS, or tvOS, you actually need to own a machine based on macOS (either a MacBook, Mac Mini/Studio/Pro, or iMac) because you need to have Xcode installed.

The reason is simple - it contains the Simulator application, which allows you to run your application in the virtual environment and is required to run tests on real devices. You'll also need an Apple Account, and if you want to publish your application on the AppStore, you'll need to sign up for the paid Apple Developer Program.