Do you have the required quality to be agile? asks Jun Zhuang, Senior Performance Test Engineer, Hobsons.

Almost all the time when people talk about performance testing in agile software development, they talk about the process. For example, plan and prepare early, start testing as soon as a story has passed functional testing, continuous testing and communication, etc. – the quality associated with the performance testing practice itself, which is also critical to the success of the agile process, is rarely mentioned.

The quality associated with the performance testing practice can be viewed from several perspectives: the quality of the engineer, the quality of the testing practice, the quality of the testing tool and the quality of the people & practice beyond the performance testing team.

The quality of the performance test engineer

To start with, it is people who participate and drive the process. The performance test engineer not only has to be familiar with the agile process but also the lifecycle of performance testing from planning to test development to execution to result analysis to reporting and diagnoses.

There is probably no need to emphasise the criticality of the planning phase, it is in this phase where performance requirements are gathered, the test plan is drafted and important questions such as test coverage, expected load and acceptance criteria, whether the existing tool is capable of doing the work should be asked and answered. These would come naturally for a seasoned tester but probably won’t be so for someone inexperienced.

Agile development requires tests to be developed and executed in a timely fashion. From this perspective, the better the test engineer knows about the tool and the application being tested the more productive he becomes. There is no denying that some tests are challenging to develop and may take a long time, but if the test engineer is always falling behind due to lack of working knowledge of the tool or the application, it is a problem that must be fixed now.

Every tool has its limitations, the quality of the engineer also reflects in whether he has sufficient ammo in his toolkit to complement the tool. Generally speaking, knowledge in every aspect of software engineering and computer engineering helps. Furthermore, there are times the cause of certain performance issues are challenging to pinpoint, even for an experienced tester. The knowledge and experience of the engineer often determine how long it takes to track down the cause.

It is necessary to highlight the roles that performance test engineers should be playing. Our primary role, of course as expected, is to take part in the software development process and provide necessary testing coverage. Our other roles, which are rarely mentioned and not being realised even by some testers, include, but not limited to, promoting performance awareness among all parties involved in the software development life cycle (SDLC) and educating people about the broad range of tests we can perform.

Believe it or not, it’s not uncommon for people to think that the only thing we do is LOAD testing, i.e., measure an application’s response time under certain load. Without the support and participation of other teams, it is going to be very difficult for the performance testing practice to be successful.

The quality of the performance testing practice

The quality of the performance testing practice involves all phases of the performance testing lifecycle from planning to test development to execution to reporting. I have already covered the planning in the previous section, here I am going to focus on the artefacts such as test scripts and any documents that are produced plus the test execution.

I once worked on an electronic health record (EHR) application used by more than 200,000 physicians in the US. This application has a page that allows a physician to write clinical notes (SOAP notes), along with more than a dozen other sections for patient related information such as vitals, surgical/medical histories, allergies and test results. The test for this page was initially built with just a few sections and pretty much all static data and it worked, surprisingly, for many releases with acceptable response time.

During one of the releases, I decided to rewrite it to accommodate the variations in test patients’ data and incorporate all the sections; as a result, the response time for the same request went up several‑fold comparing to the established baseline. What caused the elevated response time? Was it code related or something else? It only dawned on me a few days later that the acceptable response time before was the result of using static data in the request, which would never happen in the real world where the information for every patient is unique. But, as pointed out by a developer, including all the sections in the request is probably not realistic either. Further tests showed that the response time is quite sensitive to how many sections are included in that specific request, i.e., the payload of the request matters.

The main takeaways from this story are that the outcome of a test could vary significantly depending on the quality of the test script and, for the test to be valid, the behaviour of the test must closely mimic that of a real user.

As with software development, the test scripts we develop could have flaws, how can we be sure they do what we intend them to do? There are a few things to think about:

• Follow the software development best practices. For example, establish coding standards and institute periodical code reviews.
• Make sure all dynamic data in the script are correlated correctly.
• Add validation to check the response of critical requests.
• Test the script with common data variations.
• Run the script and manually verify it did what it is supposed to do via the UI or database.

Did it ever happen to you that you intended to run the test against one environment but instead it was run against another one by mistake? Did it ever happen to you that the test was ran against a wrong build? How about running tests with incorrect configurations and wrong test data? Admittedly, I made all those mistakes at some point of my performance testing career. Time wastes as a result of these mistakes can be characterised as ‘lost‑time’. If you keep making these mistakes, how can you not be stressed out while trying to keep up with the fast pace of agile? If that’s the case, it’s probably time to stop and think hard how you can avoid them. Here are some tips:

• Try to automate the test setup and configuration as much as possible.
• For the first few minutes after starting a test, manually check the target environment and make sure you see expected activities as a result of your test.
• Start with a small load and gradually increase it to the desired level. If you try to apply the max load off the bat, chances are you might bring down the system.

Performance testing tools, at least the commercial ones, typically come with some sort of report generating capability, some even allow the testers to create templates based on which reports can be generated. But as much as the automatic report generator can help, I cannot see how it is ever possible to avoid certain customisations, i.e., if you want to create a high quality report.

To begin with, the content of the report should reflect the purpose of the test. For example, it won’t make sense to include in the report a CPU graph of the application servers for a test run directly against the database.

Secondly, you may want to include only the minimum amount of information that is necessary to support the test findings and make sure they are easily understandable by your audience. During a test you could gather a huge amount of data from throughput to transactions response time to servers CPU usage to network bandwidth usage, which, if all are included in a report, especially in their raw format, could be overwhelming as well as confusing. Depending on the audience, a one‑page executive summary might be sufficient for upper management, but you might want to include some technical details for developers.

As long as it can accurately communicate the results, the report does not have to be very formal, sometimes an email might just suffice.

Despite what else you deem appropriate, I would always include following in the report: my observations, recommendations (if applicable), and additional actions to be taken (if applicable).

The quality of the testing tool

Cost aside, it is probably safe to say that not every performance testing tool, open source or commercial, suits the testing needs of your application. If you have not decided on a tool yet, following is a list of questions you might want to ask before choosing one:

• Does it allow easy development, relatively speaking, of test scripts?
• Can it be integrated into automated build and deployment processes to run tests automatically?
• Does it allow you to ramp up any number of virtual users anytime you need to?
• How good is its performance monitoring and result analysis capability?
• The quality and responsiveness of the support.
• How big is the pool from which you can select additional tester(s) when needed?

As for the details of any of these, many people have written about them, I don’t feel it’s necessary to repeat what they said here.

If you have chosen a tool but are not happy with it, then you may or may not have an easy way switching to a different one depending on how much assets you have built and terms of your contract with the tool vendor.

The quality of everything beyond the performance testing team

All people know that the earlier to find and fix a defect the lower the cost and everyone knows, in theory, how to do that. But, in reality, application performance is still an afterthought in many places and performance testing is often underutilised or not being given the proper priority.

A common problem is the lack of non‑functional requirements, without which will lead to insufficient performance coverage, which, in turn, will lead to mediocre performance in production – the application might work most of the time, but when hit with high load, all kind of problems will be exposed. Some of the problems can be easily addressed by adding more resources, i.e., if scalability was not forgotten when designing the system and additional resources are readily available, but some, such as memory leak and bad queries, may require lengthy and extensive investigation. During one such accident, the production of one of the companies I worked for was completely shut down for about two weeks. Try to imagine the stress people had to endure when that happened, not to mention the financial impact.

Have you ever wondered why it takes forever for some web pages to load? Among many possibilities, one could be that the developer who wrote that page did not employ the many performance optimisation technologies to speed up the page loading; as a result, resources used by the page might be making unnecessary trips to the server, or not compressing images, or not caching data that should be cached, etc. Performance issues such as these could be reduced if developers keep performance in mind while coding and run sufficient unit tests. It’s doubtful that any company can deliver high quality software before its developers get over the idea that QA is the organisation that’s responsible for the quality of the application and start paying attention to the quality of code they write.

Certain performance issues can happen with just a few users, but some require heavy load, which can only be done against a fairly powerful system. For example, database queries often perform differently depending on how many records are in the database. If it is too expensive to have a clone of the production environment for performance testing, at least you should test against a database with production sized data.

Finally, let’s talk about production monitoring. When customers complain about production slowness, is there quality monitoring data to tell us what was going on in the system at that moment? What was the request rate? What was the resource utilisation? This information is critical for timely diagnosis and to fix potential performance issues.

Summary

A smooth and productive agile process is like a well‑oiled machine, to get the max output, all parties involved – business analysts, development, testing and TechOps – have to operate efficiently. Software quality is the responsibility of everyone involved in the SDLC, and no company can deliver high quality software before that happens.

This article first appeared in the November 2016 issue of TEST Magazine. Edited for web by Jordan Platt.