Henrik Rexed, Performance Specialist, Neotys, explains the power of performance testing in agile. TEST Focus Groups.

Let’s face it: agile is a fact of life. Perhaps you’re not a ‘full agile’ shop and maybe you’re not doing continuous integration or even talking about DevOps yet, but the reality is that pressures are increasing to realise many of the benefits, like quality and speed, that are inherent in agile or ‘agile‑like’ development methodologies.

When you start becoming more agile, developers churn out code at a rapid pace, and many testers struggle to keep up with this pace. Furthermore, testers in agile teams often have responsibility for automated testing, unit testing, regression testing, as well as load and performance testing. In this environment, you need to be able to keep up with the speed of development while also meeting heightened expectations of quality.

Performance testing in agile is even more powerful than in standard development environments because you can:

1. Avoid late performance problem discovery: When load and performance testing are pushed off until the end of a development cycle, there is often little to no time for developers to make changes. This can cause teams to push back release dates and delay getting features out the door that customers need.
2. Make changes earlier when they are cheaper: By including load and performance testing in continuous integration testing processes, organisations can catch performance issues early before they get much more costly to fix. This is especially true on agile teams when discovering a performance problem weeks later could mean that it actually occurred several builds ago which makes the task of pinpointing the root cause a nightmare.

In the same way that combining agile with load testing can provide unique benefits, it can also present your teams with unique challenges they may not have experienced in the past, such as:

1. Shorter development cycles require more tests in less time: With agile, development cycles are much shorter, and load and performance testing can get pushed off until the last day of a sprint or sometimes it’s done every other sprint. This can often result in code being released without being adequately tested or user stories slipping to the next release once they have been tested. Conceptually the solution is to do the testing earlier in the development cycle, but that’s easier said than done with many teams lacking the resources and tools to make it happen.
2. Developers need feedback now: Agile developers need to know more than just the fact that their code is causing performance issues: they need to know when their code started causing problems and what story they were working on when the issue started. It’s a huge pain for developers to be forced to go back and fix code for a story they worked on weeks or months ago. It also means they can’t spend time working on getting new features out the door. Detecting performance issues early in the cycle so you can deliver important feedback to developers quickly is crucial to saving costs.
3. Automating the handoff from Dev to Ops can feel risky: While DevOps and continuous deployment are still fairly young practices, the fear felt by operations teams that new changes in code will slow down or even crash the application when it is deployed in production has been around forever. Automating some of the testing in the continuous integration process can help to ease some of this fear, but without adequate performance testing included, the risk is still real. Ops teams know well the huge impact application downtime can have on the business.

Overcoming challenges

The following best practices can help you maximise the advantages while helping you overcome the challenges – of load testing in an agile environment.

Put performance SLAs on the task board

Every application has minimum performance service level agreements (SLAs) to meet, but agile teams are often more focused on adding features and functionality to an application than optimising the application’s performance. User stories are typically written from a functional perspective without specification of application performance requirements. Performance needs to be somewhere on the task board if the team is going to give it attention.

Work closely with developers to anticipate changes

One of the benefits for testers working in an agile environment is that they typically learn about updates on development tasks during daily stand‑ups or similar meetings.

In order to get the maximum benefit from this level of collaboration, testers should constantly be thinking about how the stories that are currently being coded will be tested. Will these require completely new load tests? Will they cause errors in current test scripts? Can you get away with slight modifications to current test scripts if you plan ahead? Most of the time, these are small changes, so testers can stay ahead of the curve if they keep engaged with the team.

Integrate with build server

Even if you aren’t completely on the agile bandwagon yet, you probably have a build server that kicks off some automated tests, unit tests, smoke tests, regression tests, etc. In the same way that performance goals need to be added to the task board, performance tests should be among the tests that occur with every build. This can be as simple as setting up a trigger to have the build server kick off the test, but could include displaying test results within the build tool depending on how sophisticated the integration is. Ideally, you want the person who kicked off the build to instantly see the results and know which changes went into that build so they can be fixed if there is a performance issue.

CI + nightly build + end of sprint load testing

The difference between continuous integration builds, nightly builds, and the builds produced at the end of sprints can be huge. We’re talking the difference between a single change committed to a version control server versus all the changes committed in a day versus all the changes committed during a sprint. With this in mind, you should adjust your load tests to the type of build you’re running.

The best practice here is to start small and internal. For CI builds that are getting kicked off every time someone commits a change, you want these tests to run quickly so that you can get results back to that developer about how his/her changes affected the system. Consider running a small performance test with the most common scenarios covered with the typical load on your application being produced from your own internal load generators. For nightly builds, ramp it up to include more of the corner case scenarios and increase the load to what you see at peak times to see if any performance issues were missed during the CI tests. At the end of the sprint, you’ll want to go all out: consider generating load from the cloud to see what happens when users access your app through the firewall. Make sure every SLA on the constraints list is passed so that every story completed during the sprint can be marked as ‘done.’

Summary

Agile development is increasing the productivity of teams and the quality of applications. When adding load and performance testing into this process, careful planning should occur to ensure that performance is a priority in every iteration. To ensure you are getting the most value from combining agile methodologies and load testing, it is advised you:

Make sure performance SLAs are on your task board or constraints list to guarantee the code associated with a task performs well before that task is marked ‘done.’

Collaborate with developers to anticipate when new code changes require changes in performance test scenarios.

Have performance tests automatically kicked off with every new build and track performance trends from build to build.

Ramp up complexity and load for tests with the size of the build to keep build times down (CI => performance smoke test, nightly build => full load test, end of sprint => stress test with cloud load generators).

Edited for web by Jordan Platt