A 5-step approach to successful mobile testing strategy

Consumer and enterprise mobile applications have grown exponentially over the last few years. In 2016, the global mobile internet user penetration exceeded half the world’s populationⁱ. In fact, as of March 2017, Android users were able to choose between 2.8 million apps. Apple's App Store remained the second-largest app store with 2.2 million appsⁱⁱ. With the user base multiplying at a fast pace, it is essential to ensure performance and functionality of the application. A comprehensive mobile testing strategy therefore acts as a key differentiator.

The scope of mobile testing is vast due to numerous software platforms and versions, a variety of hardware, screens resolutions, and diversity in mobile devices. In addition, frequent updates in the operating systems, new mobile devices in the market, and increasing consumer expectation of quick software upgrades means that prompt and efficient test cycles are required. Covering this diverse scope with an inefficient mobile testing strategy can be expensive and time consuming.

At Nagarro, we have developed a mobile testing strategy that revolves around automation, maximizing coverage and at the same time optimizing the cost of mobile testing.

Step 1: Device sampling

The first step of our mobile automation strategy is to identify the set of real devices, platforms, simulators where the automation framework will run. We call it device sampling. The aim is to establish the right balance between simulators and real devices to optimize cost of testing without impacting quality. Simulators are effectively used for testing basic application functionality and flow while testing on real devices covers touch response and user experience. A stable mobile application (after simulator testing) is then tested on real devices.

Step 2: Choosing the right automation tool

After device sampling, we select the right mobile automation tool. The choice of tool is based on different factors such as cost, type of mobile platform, script reusability, ease-of-use, browser support capabilities and supported languages. To optimize the automation cost, we look for open source tools first and move to commercial tools only when we find some constraints with the open source tools. Some of the tools we use include Appium, Renorax, XCUI test, etc. for Android and iOS automation.

Step 3: Selecting the automation framework

After finalizing the right tool, we focus on the type of automation framework (data-driven, keyword-driven, hybrid) for the mobile application. Selection of the automation framework is based on our framework selection checklist which includes criteria such as size of the application, type of business scenarios etc.

Step 4: Minimizing automation risk for future

Once the framework is identified, to minimize automation risk in the future (basically, to make it future proof), a small PoC (Proof of Concept) of 2-3 weeks is done to automate critical business workflows. This helps in identifying any major issues that might be encountered during future automation. The PoC also helps automation engineers in accurate estimation of the volume of automation.

Step 5: Monitoring

After completing the estimations and PoC, automation scripting, execution, debugging and reporting starts and the KPIs (such as coverage, detected defects, etc) are closely monitored. The automation framework is also integrated with build pipeline (Continuous Integration) to ensure automated smoke, sanity and regression as soon as the build is deployed in the test environment for testing. It is vital that automation remains relevant and effective over a period as the mobile application grows. Thus, close monitoring and maintenance of automation framework is essential.

Mobile application testing is challenging.  However, with an optimal selection of simulators and real devices, we can maximize test coverage without the need to test every feature on each device. Nagarro’s mobile automation framework supports a variety of platforms and is integrated with build and deployment pipeline for continuous automated testing and regular quality validation.

Read more about key considerations to keep in mind while formulating an effective automation testing strategy.