What is Mobile performance testing
The function of Performance Testing is not really understood by most people beyond the requirement to “make sure it works fast”. So how do we go about defining Mobile Performance Testing? While the majority of test planning and execution is not that much different than other types of performance testing there are a number of aspects that exacerbate risks, for example the technology involved and the increased number of variations and moving parts of a system’s ecosystem.
When we think of traditional Performance Testing, we think of Load or Stress testing, however these do not really apply for “on-device” mobile testing. Instead, we need to be concerned about things like battery life, user experience performance, app loading, screen/data rendering and screen interaction. For the purposes of this post I am going to focus on just the “mobile” aspect of performance testing, specifically focusing mostly on “on-device” testing.
Types of Testing
As with any good quality assurance approach to a project, testing begins at the requirements stage. Because of the different technology involved (we are talking about a large number of different variations of hardware and software, with each version of software being specific to the hardware in question) this changes the approach to how your application is to be designed.
The question of Native (Smoother user experience but harder to deploy changes) vs Web (Clunky user experience but easier to respond to change and control content served to users) or some kind of hybrid needs to be asked and answered and the decision will depend on what your target audience is and the focus and complexity of your application.
What is your intended audience? Is it a focused set of users for a distinct set of features or are you aiming for a more generic widely used set of features. The closer your relationship may allow you to set-up some sort of Pilot or Early Access program which will allow your to use monitoring tools to capture real life data on the performance of your application before release to the masses.
These are among just some of the questions to be asked and answered in the requirements phase that will play a big part in determining how much of an investment to put in to the technical performance testing.
The guiding rule of all performance testing (to the ley person at least) is “Make the app FASTER!!”. While this may sound facetious, there is at least an element of this that is true and that we can develop and test for. It does beg the question however, who is responsible for testing. The short answer, especially in this scenario, is that testing begins with the developer themselves.
Regardless of the platform you are developing for, each of them have a set of tools aimed at helping debug the performance of an application at the developer level, monitoring things like CPU, Memory and Data usage.
- Instruments Performance Profiling has long been used for Memory/CPU usage, etc
- Xcode 6 has also introduced a lot of new features aimed at helping test your app’s performance, eg: Unit test level performance testing of code where you are able to set and measure against baseline performance criteria
More information on what apple are working on can be found at WWDC 2014 Session Videos, in particular the sessions:
- Improving your apps performance with Instruments
- What’s new in XCode 6
- Turning developer options on on your device gives you access to a lot of tools such as:
- CPU/GPU Usage,
- Hardware Layer Updates
- Android is also good at logging pretty much everything going on (which can be pretty verbose). It always helps to monitor the logs as you are testing.
- Eg: adb logcat | grep Skipped results in:
Skipped 147 frames! The application may be doing too much work on its main thread
- Eg: adb logcat | grep Skipped results in:
- Android Device Manager tools have also got a lot of performance debugging tools embedded into it:
- Hierarchy Viewer
- How complex are your layouts?
- Thread monitoring
- How much time is spent in the Main thread and doing what?
- The biggest culprit of poor performance is too much happening in the Main thread.
- Hierarchy Viewer
One of the many things Microsoft does will is build an integrated development environment. While I haven’t used these tools myself for mobile development, a quick Bing search quickly results in the Windows Phone Application Analysis tool.
- You get this automatically as part of the Windows Phone SDK
- It’s fully integrated into Visual Studio IDE
- Runs against emulator or phone
- Enables monitoring and profiling
- App Analysis – performance and quality
- Execution and Memory profiling
- “..make quality assurance a part of the development cycle..”
When thinking about testing at the server level, the first question is “What is unique to Mobile performance testing?”. There are a lot of similarities, you are still testing at the service/api level. The main difference is that testing is exacerbated by the mobile platform. There are a lot of different Devices/OS/Network/Bandwidth combinations. There are more moving parts that can impact user experience. So at the end of the day there are a number of things the need to be focused even more on than other types of performance testing, some being:
- Data/Packet size – you cannot guarantee that everyone is on a high-speed or reliable connection. Keep data as small as possible
- Battery life – while there are no ‘standards’ on developing to maintain battery life, there are some guidelines to help in this area:
- When transmitting data, it is better to transmit in larger batches rather than many smaller transactions. This cuts down on the number of interactions and the amount of actual work the phone needs to do.
- Latency – It is a given that response times on a mobile device will be impacted. This will feed into your app’s design more than anything
- Real Device vs Simulators – Always test on real devices. When capturing traffic for your favourite load tool, always capture it using real device traffic going through a proxy.
So as we can see, “Mobile” Performance testing is not too different from other types of performance testing, we are simply dealing with a different technology platform.
I’ve mentioned some basic differences in the types of “Performance” testing, across all levels. As always, performance testing is not something to be left until the end of the development lifecycle. Requirements definition and the type/design of your app both greatly influence how much you performance test.
Most importantly though, Performance Testing is owned by everyone. While the mobile platform is still relatively immature, advancements in toolsets are allowing developers to begin technical testing at a debug/unit testing level, they are free, and there is no excuse to not be using them.