In today’s business environment, Enterprises are making huge investments in mobility space to ensure better user coverage and experience. Most companies are releasing the enterprise applications on various mobile handsets having varied operating systems, different screen sizes and hardware configurations.
In many ways, mobile testing is more challenging than testing on desktop or Web applications due to the multiple combinations of operating system, screen sizes and hardware configurations. Testing teams need to find better and more cost-effective solutions to ensure more test coverage in less cost but without compromising on quality benchmarks.
Challenges in Mobile Testing
The phenomenal growth of mobile devices has opened avenues for organizations to integrate them into computing environment. Today’s mobile applications deliver complex functionality on platforms that have limited resources for computing. Mobile applications have evolved significantly over the past few years. Every organization needs a mobile application and they make sure the mobile applications developed should be of high-Quality. Nowadays we have mobile applications for almost all the different categories whether its Game, Sports, Business, Movies, Music, Banking, Books, Education, Food & Drink, Lifestyle, Medical, Magazines, News, Social Networking, Kids, Utilities and list is endless.
Mobile app testing is more complex and varies from testing web applications and traditional desktops. It should be tested on a variety of hardware and software platforms of various versions under different network connectivity conditions. It required unique skills in a tester along with innovative thinking.
There are number of challenges Tester can face while performing Mobile Applications Testing:
- Choosing mobile devices with different resolutions and testing the app on all
- Its More time consuming than testing on Desktop browsers
- Testing a mobile application requires multiple tools for verification
- Network challenges
- Frequent upgradation of Platforms and hot fixes
- Frequent upgradation of Mobile applications to cope up changing users and platforms
- Test Lab Setup
Diverse Mobile Device landscape
Mobile devices differ in screen sizes, input methods (Touch, QWERTY, normal) with different hardware capabilities.
Diversity in Mobile Platforms
There are different mobile operating systems in the market. The major ones are Android, iOS and Windows. Each operating system has its own limitations. Testing a single application across multiple devices running on the same platform and every platform poses a unique challenge for testers.
- Manufacturers like iPhone, Samsung, HTC, Nokia, Sony, Micromax, Oppo, Vivo, MI etc.
- Versions of an operating system such as iOS 7.x, 8.x, 9.x, 10.x, 11.x, 12.x etc.
- Different Mobile Platforms
- Blackberry OS
- Spectrum – Chinese phones – Low Cost Mobile Platform – China
Market share of different Mobile platforms in 2018: As per Stat Counter Global Stats the market shares of different Mobile Platforms in 2018:
All the challenges can be mitigated by carefully choosing unique Test Strategies, Testing Types and Types of Devices chosen which will ensure thorough and end-to-end testing of Mobile applications.
Mobile Application Testing Strategy
Mobile app testing is time consuming and expensive. It is vital to make sure that the experience is a great one for every consumer every time they use the application, especially the first time. If the tester fails to do a good job app testing, this will result in your customer doing it for you—and unlike the testing teams, customers do not have the tools or the time to report back problems. Also, keep in mind that your customers do not want to be treated like guinea pigs. When they find a flaw, you’ll never hear a word from them and they certainly won’t come back.
Simulators vs. Physical Devices
Use of device simulators can be highly effective in the early stages of product development when features are under development. Simulators are useful to familiarize the test team with various application features.
Figure below illustrates various conditions under which testing with simulators and real devices can be effective. Simulators can be effectively used for testing basic application functionality. Testing on physical devices is essential to understanding application behaviour such as touch response and user experience. Stable, defect-free (based on simulator testing) features can then be tested on physical devices. Device usage can be optimized by distributing test cases across various types of devices.
Note that this strategy requires that the test tools being used support testing with simulators as well as devices.
Real Devices with Real Networks
Tester should test applications on real devices on real networks whenever required. This is important since the Mobile application will always be used on mobile devices by end users who may access the application from a remote area with fluctuating network signal strength. Tester can test everything their user can do, from pinching and swiping to double-tapping and Scrolling.
To ensure better test coverage on multiple devices, enterprises might need to invest in device cloud solutions. Though there are some limitations in terms of the time lag while testing but there are huge advantages in terms of the cost saving, better and early predictability of results on varied platforms.
Automation Tools to Avoid Manual Work
The QA team should automate the mobile applications task to avoid manual work. There is a wide choice of tools for automation. Some of them are free, some are rather expensive. Some automation tools were created years ago; some have just appeared on the market. Each tool is unique and possesses certain characteristics.
How to Make Automated Tests Effective
- Understand the Project scope and perform feasibility analysis for automation
- Identify potential tests that can be automated
- Perform ROI Analysis for automation
- Explore all available and suitable tools for automation;
- Select the most suitable one or more tools based on the research and experience (Some of the criteria for selecting automation tool can be easy of use, features, cost involved, extensibility, support etc.)
- Discuss the chosen automation tools with other project stakeholders, justify the choice and get their approval;
- Proceed to test automation.
Best Mobile Testing Tools
There are several Open source (free) Mobile Test Automation Tools available in market. That can check Functionality, Usability, Consistency and Security of Mobile app. Which can really reduce the time and efforts. Also make sure to evaluate and use the tools already available free of cost with Android and iOS SDK. Some of the widely used tools are mentioned below
Calabash (Android and iOS)
Calabash is an open source (free) tool that enables test code to interact programmatically with native as well as hybrid apps. Calabash helps automate and test your application on over 1,000+ devices in the cloud. It is an automated acceptance testing framework for mobile apps. Calabash could be compared to Selenium WebDriver. However, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen.
Calabash provides APIs that are specialized to native apps running on touch screen devices.
Appium (Android and iOS)
Appium is open source and it is meant to be a cross-platform test tool for the hybrid and local iOS. It is used for native app testing, hybrid app testing and mobile web app testing. Appium works like a server running out of sight like selenium server. It supports most platforms including iOS, Android, Windows, Firefox OS and Mac.
MonkeyTalk (Android and iOS)
MonkeyTalk computerizes useful intelligent tests for Android and iOS. It is an open-source instrument comprises three segments, for example, IDE, Scripts, and operators. Scripts used in tools are basic and justifiable. MonkeyTalk automates real, functional interactive tests for iOS and Android apps – everything from simple “smoke tests” to sophisticated data-driven test suites. Native, mobile, and hybrid app, real devices or simulators
It is tool for free Android UI testing which is anything but difficult to compose capable programmed black box test cases for Android applications. Along these lines, there is no requirement for additional data about the Android application’s structure or executed classes.
It is an iOS integration test framework. It allows for easy automation of iOS apps by leveraging the accessibility attributes that the OS makes available for those with visual disabilities. KIF builds and performs the tests using a standard XCTest testing target. Testing is conducted synchronously in the main thread (running the run loop to force the passage of time) allowing for more complex logic and composition.
Selendroid is a test automation framework which drives off the UI of Android native and hybrid applications (apps) and the mobile web. Tests are written using the Selenium 2 client API
Google has released a new open source test automation tool for iOS called EarlGrey.
EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests. It integrates with Xcode’s Test Navigator so you can run tests directly from Xcode or the command line.
There are few other leading commercial automation and cloud tools in market like SeeTest, Perfecto, SOASTA etc. that too needs to be evaluated for the features provided.
Mobile Application Testing Methodologies:
Depending upon the scope and test approach some of the following testing types can be used:
Certain mobile applications come pre-installed on the device whereas others have to be installed from the store. Installation testing verifies that the installation process goes smoothly without the user having to face any difficulty. This testing process covers installation, updating and uninstalling of an application
Functional testing ensures that mobile app under testing working as per the defined requirements. Mostly, testing performs on the user interface and call flows of the application. As like other UI applications mobile applications also require lots of human consideration.
Every application is different and it’s very important to perform testing on business logic on which the mobile app is intended to work. we verify the declared price and content correspond to the user got information.
Ensure the user can perform typical operations: buying, adding goods to the cart, ordering goods etc. Make sure the application supports payment transactions through payment systems like Visa, Mastercard, Paypal etc. Check the recovery of the purchase regardless of the device, but with an account binding.
Usability testing is carried out to verify if the application is achieving its goals and getting a favourable response from users. This is important as the usability of an application is its key to commercial success (it is nothing but user friendliness). Another important part of usability testing is to make sure that the user experience is uniform across all devices. This section of testing hopes to address the key challenges of the variety of mobile devices and the diversity in mobile platforms/OS, which is also called device fragmentation. One key portion of this type of usability testing is to be sure that there are no major errors in the functionality, placement, or sizing of the user interface on different devices.
Recovery test verifies the app under test in terms of its ability to withstand and successfully recover from possible failures caused by software errors, hardware failures, or communication problems.
Localization testing allows to test mobile application adaptation for a specific target audience in accordance with its cultural specifics. Using Localization Testing we test languages supported by the application, Ensure the correctness of the translation, Verify the correctness of the translation in accordance with the theme of the application, Check the date formats, Check the delimiters in numbers.
Finally, you have the pre-release full functionality version of your mobile app. It would be better to evaluate the possibilities and stability of the program in terms of its future users.
Beta testing is the stage of debugging and checking the beta version of the program. Its main purpose is identifying the maximum number of errors in its work for their subsequent elimination before the final release of the app to the market.
People who have experience with working with similar type apps, better yet, with the previous version of the application are chosen to the role of beta testers.
An application while functioning may face several interruptions like incoming calls or network coverage outage and recovery. The different types of interruptions are:
- Incoming and Outgoing SMS and MMS
- Incoming and Outgoing calls
- Incoming Notifications
- Battery Removal
- Cable Insertion and Removal for data transfer
- Network outage and recovery
- Media Player on/off
- Device Power cycle
Memory Leakage Testing
Memory leakage happens when an application is unable to manage the memory it is allocated resulting in poor performance of the application and the overall slowdown of the system. As mobile devices have significant constraints of available memory, memory leakage testing is crucial for the proper functioning of an application
This testing process is undertaken to check the performance and behaviour of the application under certain conditions such as low battery, bad network coverage, low available memory, simultaneous access to application’s server by several users and other conditions. Performance of an application can be affected from two sides: application’s server side and client’s side. Performance testing is carried out to check both.
This includes checks on server connection changes to WIFI from 2G/3G/4G or vice versa, shared images size used for the Application is as per the requirement, application response time, code
optimization for the CPU cycle, battery consumption, memory leaks, resources like GPS, CAMERA, etc., freed.
This includes encryption/decryption techniques used for sensitive data communication, checks for multi-user support without interfering with the data between them, checks for access to files saved in the app by any unintended users, detect areas in tested application so that they do not receive any malicious content.
There are certain rules for organizing an installation file (.apk) and rules for applications design for each application store. Certification testing verifies the app meets the requirements of the most popular stores like Google Play, the App Store, and Windows Phone.
- The installation file for the application (.apk) matches with Program Policies .
- The application meets the requirements of the UIG (User Interface Guidelines).
- There are no viruses in the app. Android market semi-automatically checks the application for viruses and could block your account if detect them.
- You should follow the order of version control in the case of publishing an updated version of your app.
- The application meets the requirements of the Human Interface Guidelines.
- The application must have a unique name.
- You need to provide a link for feedback from the developer.
- The application should be put to the determined particular category.
- App Store test the app for compatibility.
- App doesn’t contain prohibited materials, unforeseen delays in work or repetition of existing functions.
- The application meets the requirements of the App certification requirements.
- Clear description of the hardware and network requirements.
- The functions mentioned in the description or shown in the screenshots are fully realized
- Option to control auto-playable sound is required.
The significant challenges and risks involved in mobile application testing can impact the success of mobile apps. A buggy application can hurt not merely business but also the reputation of the organization. These risks and challenges can be mitigated by adopting the various testing types and strategy outlined. The various tools available in market for mobile application testing not only help with testing but also provide a platform to measure security and compatibility compliance. It is imperative to choose a cost effective and effective strategy that helps in the success of the application launched.