Automated mobile UI testing is a relatively new field in quality assurance automation. A few years ago, QA specialists were facing a myriad of challenges because of buggy tools, low environment performance, and issues associated with mobile platforms, early Android versions can be an example. The situation fundamentally changed with the appearance of Appium 1.0 on the market in 2014.
With this open-source test automation framework, testing became much easier, but some challenges still remain. While web and desktop applications are quite stable regarding testing, unfortunately, we can’t say the same about the testing of mobile applications. That’s why I decided to create this brief Appium tutorial to help you minimize the number of mistakes and make the testing process more quality and fast.
According to the World Quality Report 2016-2017, up to 48% of the surveyed pointed out they are lacking the right process or method to test mobile apps. The research states that in 2016 the situation became even sharper than it was in 2015 as statistics are getting 20% higher. The other issues the QA industry faces are associated with a lack of the in-house test environment, scarcity of time to test and an insufficient pool of test experts available. Today, as never before, testing professionals have to make a shift in their approaches, adopt new relevant strategies to address these challenges and provide customers with a better experience.
This article will give you a brief analysis of the main challenges in automated mobile testing and provide some useful tips on how to tackle these issues with the help of Appium. Many of these challenges I experienced on my own, so the advice is based on the real examples from the projects I’ve worked on as a QA Competency Manager.
So, what are the biggest dilemmas of mobile test automation?
As it was mentioned above, mobile testing is not an easy thing. Here are some challenges that every mobile application tester may encounter and my advice on how to deal with them.
- Test environment scalability (execution on multiple devices)
When you have to run tests on different devices, e.g. up to 3-5 of them, it would be more convenient to buy these devices for testing purposes. But imagine the number of devices is from 5 to 10 gadgets (smartphones, tablets, etc.). It’s difficult to organize automated testing on all these platforms as it requires a great deal more effort and resources. You will need a few separate computers with connected phones, a safe place for keeping all the devices, and additional time for servicing them (restarting, charging, etc.). So, if you are testing on a large number of devices, it might be more comfortable to use such Cloud services as Sauce Labs, Perfecto Mobile, or BrowserStack (currently only web apps). All these services provide you with almost any emulator you need. The real devices are also available, but the price is much higher. For example, using SauceLabs real devices requires an enterprise license. The great thing is that the services mentioned above support Appium. This means that you can run tests using Appium on devices and emulators hosted in the Cloud.
- Low test performance (e.g. emulators, clouds)
If you test on emulators, performance is an essential factor for test development and execution. To boost the performance of Android SDK Emulators, you have to make two simple steps. First, install the HAXM driver and second - check the “Use host GPU” option in the AVD manager settings (note: please perform these steps in the order they are mentioned). In case you are looking for some faster and more functional emulators, you can try Genymotion. The thing is that it’s not free, so you need to buy a license after the trial period ends.
- Device connectivity problems: issues related to drivers, specific device limitations (security, accessibility, etc.)
The devices you use for testing may differ a lot and may concern various features such as firmware, launchers, USB-drivers, etc., so the problem can emerge at any level. For instance, when I was running a test on my phone, I couldn’t access some elements of the application; however, I faced no problem while testing the same app on another phone.
- Access to local resources, a configuration of tunnels, VPN, etc.
The major step to avoid any problem with the access to the resources you are planning to test is to enable a connection with the database, web services, etc. If you are working with Cloud systems or remotely, you will need to tune up the tunnel or VPN-connection.
How can Appium help you overcome these challenges?
Web testing is usually associated with Selenium tools, desktop testing – with Ranorex or similar instruments, but when we talk about a mobile test automation tool then, for me, it’s definitely Appium.
Here are some reasons why I consider Appium to be the most convenient tool for mobile testing:
- It supports Selenium Webdriver API, so you don’t have to learn anything new, presuming you are already familiar with Selenium;
- There is no need to recompile or change the application under test. It is a significant advantage since many mobile tools require application instrumentation or integration of a third-party code), which can change application’s behavior;
- It supports major programming languages: Java, Python, Ruby, C#, etc.;
- It’s an open source tool and has a broad community;
- Appium supports both Android and iOS platforms.
I consider Appium to be a great tool for mobile app testing, but, as any other instrument, it has its limitations (I’m talking about the latest 1.4.16 version):
- It doesn’t support image recognition/comparison;
- Some features are available just on Android, but not on iOS and vice versa;
- Swipes and gestures work only in native applications, in web - you should switch contexts and use coordinates;
- Some API are not implemented;
- Unexpected errors may occur;
- There is no support for an Android API version older than 17;
- On iOS, we can execute only one test at a time, and configuration process may be tricky to do the first time.
I believe — despite all these drawbacks — for today Appium remains the best choice for mobile testing automation.
Which environment is the most favorable: emulators, real devices, or Clouds?
A few times in the article I’ve mentioned the testing environment regarding its importance for the overall mobile automation process. Most likely, you’ve already asked yourself which software and hardware are better to set up for smooth test execution. Below, you can find my summary of all the advantages and disadvantages concerning each of the test beds - emulators, real devices, and Clouds.
- Emulators: you can use them for development, PoC, white box testing, or run a smoke test. It’s the fastest way to verify something.
- Real devices: they are the best option to perform regression or acceptance testing, providing a real user experience; however, the minus here is that with the real devices, you won’t be able to scale easily, especially when the number of platforms is big.
- Clouds (Sauce Labs): allow you to scale your testing efforts on different platforms and provide up to 60 devices and emulators. Furthermore, with the help of a Cloud environment, you can:
- Capture video while testing
- Generate reports with screenshots for each test step
- Provide hundreds of emulators and 15+ (Sauce Labs) real devices
- Get reliable support as you won’t need to configure and bolster the test environment
The only drawback is that if the team is distributed, there is no way to share devices.
The table below would help you compare the main characteristics of every test environment and choose the most optimal one for testing:
