Are You Selecting the Best Tools for Mobile Software Testing?

Are you confident your testing tools are well suited for the mobile challenge? We all know it’s critical to “get mobile right”. With the popularity of mobile applications exploding, software product companies face a tantalizing opportunity to expand their market reach by offering mobile products. Yet, the twin side of this opportunity is the danger associated with releasing a mobile product too soon, before all the major bugs have been discovered and fixed; today’s consumers are so demanding that a poor experience with mobile can often lead them to reject the entire brand. Ouch.
This creates particular stress for the mobile testing group. As the last line of defense in catching errors before unforgiving customers discover them, mobile testers need to be fast, accurate, and adept at their jobs. There’s a lot riding on your effectiveness, when you’re in this position.

Naturally, mobile testing is vastly improved when you’re working with the best possible tools. So how should you go about choosing your mobile testing tools? From our experience working with clients around the globe, the team at Nagarro has collected several recommendations to help you with your mobile tool selection process.

For starters, we encourage you to begin with a proof of concept on critical platforms before you narrow down to a single tool or tool set. It may well turn out that a single tool won’t be sufficient to achieve your goals for cross-platform automation. Additionally, you’ll need to consider how your tools integrate with test management environments. You may find it helpful to begin by identifying reusable scenarios across platforms, and then group them accordingly. This can be useful towards creating an appropriate, time-saving framework.

Next, when focusing on mobile applications, you should choose critical test cases that can be run across releases as automation candidates. It’s also important to remember the different form factors that need to be tested when you’re mapping out test coverage, especially as this relates to the mobile Web. Plus, you’ll need to consider different connectivity scenarios which will need to be tested with automation execution runs.

With these basic considerations in mind, we’ve prepared a checklist of specific areas to review, as you evaluate your tools for mobile testing:

Checklist for selecting Mobile testing tools

• Cross-Platform (and Multi-Version) Support – Mobile testing tools will need to support numerous devices with various platforms, as well as multiple versions of a given OS.
• Cross-Browser Support – Tools will need to test different versions of Internet Explorer, Chrome, Safari, Firefox and (possibly) Opera on multiple operating systems, to validate a consistent experience for all users.
• Reusability – Mobile testing tools need to support reusable test scripts to save time and effort. Look for avenues for early automation, because most mobile projects are Agile.
• Data-Driven Automation Support – Look for tools that will support iterations in the execution, to increase coverage and ROI. Good candidates for automation are test cases that have repeated business logic, functional flows or data-driven tests.
• Integration with Test Management Tool – The mobile testing tool should integrate easily with other test management/bug-tracking tools, as well as with application lifecycle management, to optimize communication with all stakeholders.
• Cost – Several factors will enter into your tool cost analysis, including maximizing automation (reducing costs), support across multiple platforms, script reusability, and total cost of ownership. Ensure ROI per each mobile platform and duration of execution.
• Ease of Use – The more difficult–to-use tools will cost more in terms of training time.
• Support From the Tool Vendor – Mobile testing tools should be supported by their manufacturer upon installation, as well as for ongoing maintenance, with less lead time expected for new OS versions and device availability on the tool.
• Hybrid Tool Combinations – Select tools that support both platform simulators and actual devices, because you can mix and match devices and simulators to optimize runs on different platforms.
• Support for Non-Functional Requirements – It’s critical for mobile testing tools to address NFR adherence, including the following areas: battery use, memory use (with respect to volatile, non-volatile, and memory card usage), memory leaks, CPU and network usage, and bandwidth use (the tool should be functional across 2G, 3G, and Wi-Fi).
  • Battery Consumption - On mobile devices, the battery is a scarce and valuable resource. Battery consumed by AUT (Application under test) is very important to measure and it cannot be done without the support of a tool.
  • Memory Usage - Most Mobile devices have low memory space. So, it is important to measure memory usage while running the AUT. A good testing tool can help us monitor memory usage.
  • Memory Leaks - The device should never run out—or run low—of memory. When a program needs to store some temporary information during execution, it dynamically requests a chunk of memory from the system from its fixed amount of total memory available. If one application uses up all of the system’s free memory, then other applications will not be able to obtain the memory that they require. It is the responsibility of each application to free dynamically requested memory when it is finished using it, so that the memory returns to the system and can be allocated to other applications as required. If the application does not free the memory after finishing its execution, this situation is called memory leaks. The testing too should detect such memory leaks as they can leads to slowing down or hanging of the device.
  • CPU Utilization - The tools should be capable of calculating per-process CPU usage statistics for all device ranges – small (slower CPU and small RAM), mid (average/optimal CPU, RAM) and high (dual/code CPU, higher RAM). CPU usage and network usage are the two major sources of battery drainage.
  • Bandwidth Usage - The tool should be capable to work and measure various network bandwidth situations and network types (2G, 3G and Wi-Fi)
  • Calculate Installation Time for Mobile Apps - Testing for mobile apps must include tools to calculate the speed of mobile app installation. The majority of users expect a mobile app to launch in 2 seconds or less.
  • User Interface Changes - Mobile testing tools must be capable of finding changes to fonts, layouts, and images, as well as capturing the exact screen location for different form factors, and fixing problems related to scrolling, text selection, back buttons, etc.
  • Interruptions - A testing tool will be needed to capture application behavior against external events such as phone calls, text messages, charger use, auto updates, etc.
  • Sensors - Mobile applications that make use of a sensor to receive external information need to be tested for the sensitivity and accuracy of the sensors.
  • Localization - Mobile tools need to test multi-language support for different content, dates, characters, postal codes, phone numbers, and currency, based on the markets and regions where the application will be targeted for use.

If you’re a CTO, Product Manager or QA leader looking for more ideas on mobile testing in general, I encourage you to read the latest article from Nagarro: