Trends in Mobile Test Automation

Hello, Readers!

In this blog, we are going to discuss the trends in Mobile Test Automation.

Test Automation and Unlimited Concurrency

First of all, it is difficult to perform a huge number of test runs on device farm without test automation.

Test automation is the enabler that makes concurrent use of mobile devices possible and enables users to run their application and test simultaneously on any number of devices.

This is also one of the sweet spots of any testing vendor/test team as there is no limitation on the number of devices that users can use simultaneously for their tests.

It’s a great news for the native app and game developers as they can use the same device roster too. But, with slightly different test scripts and frameworks. The same foundation serves both the user groups extremely well.

Especially during the past two years, mobile test automation services has come into play and helped mobile game/native app developers to quickly and easily see how their games work across device combinations and versions to get the vital debugging information on which things should be fixed before the release.

This has produced tons of great data of how game/ app can be optimized for end-user devices.

Today, the mobile test runs hammer thousands of devices every day, producing an enormous amount of data, results, screenshots, recorded videos out of devices runs, and performance statistics.

This shows how a mobile app/game work on these different handsets.

From the recent survey data, close to 84% of mobile test runs were done on Android devices and less than 16% on iOS devices. This provided just a bit higher failure rate (17.8%) than on iOS devices (15.5%).

This sort of finding states that there are lots of issues also on iOS devices, with an increasing trend.

Mainly, this is because of new iOS versions and some new form factors with iOS devices plus new APIs, new notification system etc.

What’s really awesome to see is that concurrency has been going up quarter by quarter.

The average number of used mobile devices per test run is now over 10 devices per a test run.

However, there is still lots of outliers and variation, as many of users still use one-device-at-time and some test automation experts use tens (or sometimes even hundreds) of devices simultaneously for their test runs.

The average test time also varies a lot.

For example

Quick and fast tests typically complete quickly to some extent, through Appium where some test runs may even take 10x time.

As expected, it’s all about the scope of testing and the logics included in Appium tests.

The Most Popular Mobile Test Automation Frameworks

Appium is the most popular functional testing services of mobile apps, games and, to the certain extent, mobile web as well (we didn’t include Selenium in these numbers as the majority of all those selenium tweaks are more or less Appium-based).

There are lots of good reasons why people have been using Appium (cross-platform, support for literally any programming language, great coverage with API etc) and frankly, it has worked extremely well until recent changes with iOS UI Automation.

On Android, Espresso and UIAutomator have been very popular as well. And they’re also good reasons why people use/love these frameworks.

Espresso provides very fast test execution and UIAutomator provides a light-weight API that is easy to adopt and use for your native apps.

Both these frameworks, however, are somewhat limited to native apps. Again, the majority of the developers either use Appium or some internal grown/developed framework.

Another (promising) new framework that has come into play is the Robot Framework and it quickly took over the Robotium, which used to be one of the mainstream frameworks in the early days of Android test automation.

On iOS Appium and Calabash, both being cross-platform frameworks, have been popular choices for iOS test automation.

As many developers have been using Appium it’s also obvious from the data that usage has been on both major platforms – Android and iOS.

Another highly used framework on iOS has been UI Automation (until the last quarter of 2016). Since then, there has been (and will be) a clear replacement movement to some other framework and I’ll cover this with a bit more details in the next chapter.

On both platforms, there are also lots of other frameworks that didn’t get listed here. Some of those are internal, proprietary or just niche frameworks that their users happen to use.

As stated, there is no wrong choice for the framework if that does the job efficiently and provides exact results on how apps, games and web stuff work on real mobile devices.

NEW TRENDS WITH MOBILE TEST AUTOMATION FRAMEWORKS

Okay, let’s speculate a bit.

Currently, we don’t see any major changes taking place in Android mobile test automation.

Appium, Calabash, and Espresso will be definitely in high use and probably even grow stronger as Google tends to favour Espresso and there are lots of Appium fans out there as well.

Calabash hasn’t provided much new to the ecosystem lately but one of its strength is that “if you speak English, you can write a mobile test”.

The most significant changes are going on in the iOS side of things. As UI Automation got deprecated by Apple, XCTest and XCUITest got an excellent start to be the next big frameworks in iOS test automation.

However, the change by Apple hit Appium, as it was relying on UI Automation as its foundation.

Now, the big question is that what happens to Appium with UI Automation depreciation.

If things start to work well with Appium, there is a good chance that many of UI Automation users go to Appium (instead of XCTest/XCUITest, that we see on going now).

It’s always worth to consider which framework (XCTest vs Appium) could provide a more stable environment for test automation runs for next years.

Alright. Let’s speculate a bit more. Let us know in the comment section below which frameworks will fall, which will flourish, and why!