How New AI technology is already impacting test organisations and how it will impact yours in the coming months 

What is wrong with automated testing today?

As application complexity continues to increase, and test budgets grow larger, throwing more people at the problem is not the solution.

A few items that are creating enormous pressure are:

• Application complexity increasing faster than test teams and tools can keep up
• According to Gene Kim, noted DevOps expert, testing is the number one bottleneck to a true DevOps culture.
• With about 40% of many IT budgets spent on QA and test (according to CapGemini), the budgets are already beyond the breaking point.
• Users report substantially different functional, performance and security results (ie issues) than were found or even can be found by test tools.
• Often load testing, UX-level performance testing and security testing are dropped in order to meet deadlines.

Defining the true costs of software QA

Today, access to large amounts of compute power in the cloud is cheap, but burdened labor cost is US$25/hr to well over US$100/hr depending on location and competency. Employing highly competent QA and/or dev team members to write 1000 Selenium scripts, for example, can cost US$100,000 (of labor). Multiply that by thousands of applications in an enterprise and it’s easy to see how costs continue to rise.

IBM estimates that the cost of finding and fixing a bug early in the development phase is $100. But if found by the QA team rises to u$1500 and when found by a customer in production is at least $10,000, ignoring brand damage. So, testing deeper and wider is far more important in each release than testing fast, but ideally you want to achieve both.

The world’s first AI-driven test automation

In order to consider automatically generating tests that are meaningful, one must consider the inputs that would be required. These include:

• Knowledge of how users are using (ideally) or will use an application (user analytics)
• Expected results and/or responses (how do we know a pass versus fail?)
• How does the application function and what is its purpose?
• How does one form requests which will not be rejected by the server, even if the server is now a different server (for instance the QA server instead of the production server)?
• What data is required for valid forms (such as credentials)?
• How can one create correlations to take server responses and place them back in to future requests (such as session ID’s)?
• How does one handle changes in a new build versus the old build?

Results to date

The speed and accuracy of the system seems unbelievable at first

The AI system has been in use by several large companies since spring of 2017. In typical cases, after a short learning phase, the system automatically generates 1200 valid test cases in 5 seconds. The resulting tests increased test coverage from under 50% to over 90%, and represent real user workflows far better than achievable with traditional scripting.

In comparing scripting times with automatic script generation, we see AI can generate 1000 scripts in a few seconds versus 3.6 million seconds for humans. AI clearly has the advantage here.

We also see improvements in test or code coverage from less than 50% to over 90%. But now also able to predict actual coverage compared to what users do in the system. In the past, we simply wanted code coverage regardless of what user flows might look like. Now with full analysis of production user flows, the system can intelligently create scripts which more closely match what users actually do. And attaining user-flow coverage of nearly 100%.

And as a bonus of AI, false positives (commonplace in manual testing) are almost nonexistent.

AI driven test generation is now available from Appvance.

Kevin Surace, CEO, Appvance

Kevin is a Silicon Valley entrepreneur. He was Inc. Magazine‘s Entrepreneur of the Year, listed as one of the top 15 innovators of the decade by CNBC, Tech Pioneer by World Economic Forum – Davos, nominated as Innovator of the Year by PlanetForward, listed as a Technology Innovator by Wall Street Journal and inducted into the RIT Innovation Hall of Fame. He has been awarded 28 patents.