Using Maestro for Acceptance Testing in Flutter

Any app developer knows the importance of testing. Sadly, some time ago, we were left without a QA engineer. This meant that specific flows (such as authentication, generating a test order, or accepting an order) had to be manually tested. Fortunately, my engineering manager came up with a plan----Acceptance Tests!

Acceptance testing is a quality assurance (QA) process that determines to what degree an application meets end users' approval. Depending on the organization, acceptance testing might take the form of beta testing, application testing, field testing, or end-user testing - Tech Target

We found a tool that will help us automate the acceptance testing process. This tool is known as Maestro. On the official Maestro website, it was aptly described as the simplest and most effective UI testing framework.

At this point, I might as well clearly articulate the purpose of this article. Well, I want to show you how Maestro helped to automate acceptance tests simply and effectively. Come with me!

Setup

I use a MacOS machine, so I will be following the applicable steps to install Maestro. For other environments, please refer to the documentation here. To install Maestro CLI on Mac, you use the command:

curl -Ls "https://get.maestro.mobile.dev" | bash

This same command can be used to upgrade the CLI version. To use Maestro for testing your app in iOS simulators, you'll need to install Facebook's IDB tool as shown below:

brew tap facebook/fb
brew install facebook/fb/idb-companion

We're done with the needed installations. We will now move on to samples, and most importantly, the flow I used to test our application at work.

Testing

Maestro tests your application using "Flows". They are essentially yaml files that list the steps and actions taken to validate that your app runs correctly. An example of a flow looks like this:

appId: your.app.id
---
- launchApp

The first line of the flow file is appId:your.app.id. This line helps to run the tests in the file on the app with the same app id. The next line is - launchApp. It is a command to start the application as the first part of testing.

appId: your.app.id
---
- launchApp
- tapOn: "Text on the screen"

This snippet goes further with the next line being - tapOn: "Text on the screen". This line is a command to tap on the widget or area of the screen with the text Text on the screen. This provides a simple way to perform clicking interactions though an even better way will be explored below. Sadly, I can't cover every command there is, so take a look at the list of commands that can be used in Maestro here.

Apart from matching texts on screen, one Flutter-specific feature is matching widgets with the value of the semanticLabel field or the label field when your widget is wrapped with the Semantics widget. I will describe it below:

//Match via semanticLabel field
FloatingActionButton(
  onPressed: _incrementCounter,
  child: Icon(Icons.add, semanticLabel: 'fabAddIcon'),
)

//Match via label field in Semantic widget
Semantics(
  label: 'fabAddIcon',
  child: Container(
    color: Colors.yellow,
    width: 100,
    height: 100,
  ),
)

Then we can access it in our Flow file like this:

- tapOn: "fabAddIcon"

Now, let's take a look at my use case. It'll help us to understand how to use this tool better. After finishing your flow file, make sure your app is installed in a running emulator/simulator or attached device. You can then run this test like this:

maestro test <flow-file-name>.yaml

My Usecase

We had to ensure that at the very least, users could receive and accept order(s). That was our moneymaker and any failure would require the deployment of multiple hotfixes. This prompted me to construct the following flow:

Dry run
Starting the app from scratch helps ensure that the state and flow of the app are predictable. It also guarantees that caching and other optimization written to work after the initial installation and run doesn't give us false positives. The snippet to ensure a dry run looks like this:

appId: co.company.app
---
- launchApp:
    clearState: true

Accept permissions
We require certain permissions for our app to run properly, so we verify if the corresponding dialogs show depending on the type of emulator/simulator our flow runs on.

# Checks for background optimization permission dialog (will only
# show on Android)
- runFlow:
    when:
      visible: Let app always run in background?
    commands:
        - tapOn: Allow
# Checks for mandatory location request dialog as per Playstore (-_-)
- runFlow:
    when:
      visible: Location Request
    commands:
        - tapOn: Accept

The runFlow command executes a specific flow when it detects the text Location Request or Let app always run in background? . It then taps on the Allow or Accept buttons to go on the next screen.

- tapOn: Continue with e-mail
- tapOn: example@email.com
- inputText: "otpbypassemail@company.co"
- tapOn: Log in

Here, we use the tapOn: Continue with e-mail command to tap on the widget with the text Continue with e-mail. This action should navigate the user to the Email Login screen. Afterward, we tap on the element containing the text: example@email.com, which happens to be a text field. This action causes the keyboard to be highlighted allowing us to input the email address associated with the account that has an OTP bypass. We input the text on the now-highlighted keyboard using the next line; - inputText: "otpbypassemail@company.co".

Moving on, we tap on the Log in button which should make an API call to verify the email via OTP. We are navigated to the OTP screen if the API call is successful.

- assertVisible: "Verify"
- tapOn: ".*otp-field.*"
- inputText: "000000"

This next line: - assertVisible: "Verify" means that we're checking to make sure that the textVerify is on the screen to indicate that we're on the OTP screen. We tap on the OTP field using a semanticLabel to bring up the keyboard. The OTP field has a semanticLabel called otp-field.

Semantics(
 label: "otp-field",
 child: OTPField( ....

We tap on the OTP field via this label and type in 000000 . Once the OTP field is filled, verification occurs. A successful verification means that we navigate to the home screen. This is to ensure that the API calls required for authentication work correctly and that the flow remains as we left it.

Generate and Accept Order
We want our home screen to execute these actions properly: generation and acceptance of orders. The home screen on our app shows a few full-screen onboarding pages and we want to make sure we can tap on what we need to generate the order.

- extendedWaitUntil:
    visible: "Skip" # Waiting for the pop-up to show up, it always shows on a fresh run
    timeout: 10000
- back
- tapOn:
    text: "More"
    retryTapIfNoChange: false

So we wait for 10000 seconds until the Skip button is visible. This onboarding screen shows up all the time. When it shows up, we go back i.e. we close the screen.
We then tap on More , (an item on our bottom navigation bar) to pull up the drawer. The retryTapIfNoChange flag makes sure the tool just taps once regardless of whether there's a change in the UI or not.

- tapOn:
    text: "Send Test Orders"
    retryTapIfNoChange: false
- tapOn:
    text: "Accept"
    retryTapIfNoChange: false
- assertVisible: "Order's Accepted"

Once the More button is tapped, a drawer will come up with the Send Test Orders button. The next lines: - tapOn: text: "Send Test Orders" ensures that the button is tapped which causes a test order to be generated by the application.

When this generated order is retrieved by the app, a bottom sheet with the details and a giant Accept button pops up. This aspect tests out the speed of our API. By default, the Maestro tool waits a bit for the Accept text to show up on the screen. All things being equal, the Accept button is clicked via the lines tapOn: text: "Accept" retryTapIfNoChange: false. The next line asserts that the Order's Accepted label on the bottom sheet is present on the screen to indicate the current state of the order.

And that ends our flow!

Sample Flow

A small demo of the flow file running locally.

CI/CD Integration

It would be strange if we all had to run this command locally, thus, to avoid cumbersomeness, we added it to our Github Actions workflow. You can learn more about how to do this here. To be concise, we added the following section to our workflow file.

- name: "Start Accept Order Happy Path Flow"
  uses: mobile-dev-inc/action-maestro-cloud@v1
  with:
    api-key: ${{ secrets.MAESTRO_CLOUD_API_KEY }}
    app-file: build/app/outputs/flutter-apk/app-staging-debug.apk

We're using the mobile-dev-inc/action-maestro-cloud@v1 action. The API Key for Maestro Cloud API is added to our GitHub repository's secrets and the path to the generated apk file is referenced in the app-file field.

Maestro provides a Cloud Console for you to view screenshots in case of failed runs and to see your logs in general.

Conclusion

Maestro helped us ensure that our core feature still worked in all environments regardless of the code we pushed. It gave us more confidence in our workflow and allowed us to make changes quickly. I certainly enjoyed my time researching and writing up the code, and I hope you do too. If you have any questions, please feel free to reach out to me on Twitter or via email.

Have a great day!