It is feared that traditional software testing practices are becoming unsustainable as device and browser numbers multiply.

In a recent announcement, TestPlant explained that its research data indicates software quality departments in the banking sector are facing scalability challenges when testing applications due to the ever-growing range of devices and browsers. Conflicts between the need to roll out new applications and updates quickly, and the need to adequately test these applications, present an insurmountable challenge if manual testing practices continue.

Scalability challenges

A typical banking application (web or mobile) today will require approximately 200 test scripts to be created and run for each release. In today’s highly fragmented consumer technology landscape, banks’ quality assurance teams are generally choosing to test 15 – 30 of the most popular device/OS/browser combinations – for example, Internet Explorer on Windows 8, or Safari on an iPhone 6. Assuming just 15 combinations are tested, the time taken to manually test a typical release is more than 380 tester-days.

Subsequent releases typically add 10% more functionality, but in a purely manual testing environment these have to be fully tested again. This increases the time required to manually test each new release to 422 tester-days.

Manual testing no longer enough

“In the past, companies were only testing on very small numbers of device and browser combinations, which would take a much smaller number of tester-days for each release,” said Antony Edwards, CTO, TestPlant. “This meant that banks could reasonably expect that a fully manual testing approach was enough, but that is no longer the case.”

As fragmentation increases, all of these numbers will also increase and it doesn’t take long to realise that testing departments will find it impossible to scale up manual testing processes. In five years’ time, TestPlant estimates the number of testing combinations to approach 60. This would see the manual testing effort for an initial release rise to more than 750 tester-days.

“The problem is linear – for every hour it takes to manually test an application on one phone or browser, then testing more phones/browsers multiplies the time required,” said Edwards. “Banks IT teams are struggling with this because they can’t just keep hiring more people to absorb the impact of fragmentation – but their marketing and customer service departments can’t afford delays.”

About technology fragmentation

Technology fragmentation refers to the large and increasing number of different devices, operating systems and browsers being used around the world. For mobile devices this means multiple iPhone and iPad versions, many Samsung Galaxy models, Nexus, etc. For desktop applications, fragmentation refers to the many browsers and browser versions in existence, including all the variants of IE, Chrome, Firefox, Safari, etc. That’s not forgetting the plethora of new point-of-sale devices and smart TV browsers.

To help provide some context to the scale of the challenge, consider Android devices alone. OpenSignal reported last year that it had recorded more than 18,000 distinct devices running the Android platform, and fragmentation is only increasing.

To effectively test a banking application, whether it be a mobile app or web-based interface, banks ideally need to test on as many devices and browsers as possible, but typically they only choose the 15 – 30 most widely used technologies. In the world of manual testing, where tests are conducted by human beings, even this fairly small cross-section presents a huge testing challenge.

Maintaining a manual testing approach means having to adopt one or more of the following unappealing options:

Pay for more manual testers, which has a significant cost implication.

Increase time-to-market (i.e. take longer to do the testing), which has a significant impact on competitiveness.

Test less, which will ultimately mean more errors in the app and customer service failings.

Automation is the answer to scalability issues

Automating software testing is the key to dealing with fragmentation. While there is an upfront cost, the costs scale much more favourably after that.

“The scalability question relating to the impact of technology fragmentation on software testing is undoubtedly addressed by test automation,” said Edwards. “Manual testing has its place, but banks who want to stay ahead of the competition when it comes to the quality of their apps need to consider the right mix of manual and automation testing.”