Industry’s first global benchmark for mobile testing coverage

Earlier this summer, Perfecto Mobile announced the release of its ‘Mobile Test Coverage Index’, the industry’s first data-driven benchmark to help organisations determine the appropriate mix of mobile devices, operating systems and form factors to test their apps and responsive websites and ensure optimal app quality for end users.

Today’s mobile market is extremely fragmented and constantly changing, challenging organisations with device mix obstacles, future releases, and the pressure to deliver flawless customer engagement. Currently, most enterprises have no standard approach for testing mobile apps and responsive websites, and the business impact that non-tested or poorly tested apps have is devastating to a brand. To help begin this testing process to ensure continuous quality and develop a comprehensive mobile strategy, enterprises need to define a set of mobile devices to test apps and responsive websites against, a strategy that best represents their target market and end users. This should be part of a larger definition of the user profile, such as the environment that the app is operating in (network conditions, GPS location, etc.). To answer the vital question “what devices should I begin testing on,” the first-ever industry benchmark report provides crucial information and insight into market trends and the correct device mix to ensure coverage.

The 32 most relevant devices

Perfecto Mobile’s ‘Quarterly Mobile Test Coverage Index’ aggregates 360,000 hours of exclusive Perfecto Mobile customer usage data, market data, and future device release timelines to create an extensive list of the 32 most relevant devices. It serves as a recommended starting point for organisations beginning the test coverage process.
The report provides an index for reaching 30%, 50% and 80% test coverage in North America, France, Germany, Italy, Spain and the UK. The device selections address five key sectors: device model, screen size, GA date, hardware system and recommended operating systems. The number of devices in each test bracket may vary from quarter to quarter to reflect the changes in the mobile market. The report will be continuously updated to meet these market changes.

Key points from the research

Organisations can use the index as a benchmark to take advantage of the three levels of mobile market coverage and scale as their needs grow. Key takeaways from the research include:

  • 30% of the market can be covered with 10 – 16 devices, based on market adoption, market device leaders, reference devices and device characteristics.
  • 50% market coverage can be covered with 17 – 25 devices. This level of test coverage yields an increased number of devices from manufacturers such as Sony, LG, Samsung, Apple, HTC and more.
  • 80% market coverage can be achieved with an index of 26 – 32 devices. This is the final group of devices to test on that will guarantee the most recommended coverage.
  • While Apple pushes its users to upgrade to the latest iOS version, there are earlier iOS 8 versions that should be included in test plans. Additionally, iOS 7 still accounts for 17% of total users, and should be included at the same level of iOS 8 test plans.
  • Including the most widely adopted Android versions KitKat and Jelly Bean is important to an Android test plan, Lollipop proves to be very important as well. Lollipop, the latest OS from Google, is running on the Google Nexus devices (considered reference devices).

This report analysed and aggregated a wide range of public, private and proprietary data from 4000+ device profiles and 360,000+ user hours across US and EU markets, which includes Spain, UK, France, Italy and Germany.