Evgeny Tkachenko, QA Lead, EPAM Systems, shares tips on how to get your house in order when it comes to legacy bugs.
If you’ve been in software quality assurance business long enough, sooner or later you’re going to be faced with a horrible problem: handling a huge legacy “bugs heritage”. If you got an “ancient” project with more than 300 bugs in its “backpack” the first thing you have to do – analyse where the main source of bugs is.
According to world statistics, the main source (about 56%) of bugs is requirements (no offense to Business Analysts), the second one is design errors (about 27%), coding errors come fourth (with 7%) after other errors (10%).
Start with introducing some changes into the delivery (development and QA) process. Try to help developers preventing bugs instead of finding them.
Tip #1: Introduce a requirements analytics stage which has to be done before coding of this functionality starts. This type of bugs is the most expensive to fix (requirements errors – 82%, design errors – 13%, coding errors – 1% of the total cost to fix defects) and their cost grows exponentially over time. I think, no one knows a product as good as a tester who works with it every day (no offense to Product Owners and Business Analysts). The QA department has to check all specs (requirements and designs) for feasibility, fullness, testability, consistency, clearness, truthfulness. It helps to make requirements clearer and avoid situations where we missed some dependencies or implemented something that we didn’t have to implement, or something we can’t find the way to validate. It saves a lot of QA and Dev teams efforts and time. Developers mustn’t write any lines of code until requirements are tested and everyone is on the same page about the acceptance criteria for this functionality.
Tip #2: Create acceptance tests based on acceptance criteria and share them with developers and BA to show everyone what you (QA) expect to get after the implementation in code. Prepare some test data in Dev environment to help developers to check whether their implementation meets business (and QA) expectations. During coding, programmers can use these data and acceptance tests to make sure they do everything right. As a result, the number of new open bugs tends to zero, because everybody knows what will be implemented and how it will be validated.
Tip #3: Conduct acceptance tests of features as soon as possible, share with the owners of this feature (product owner, customer or client) what did you get to receive a feedback when it is not too late to make any changes. “Fail Fast, Fail Often” is my advice, the purpose of failing fast is to learn and adjust course more quickly. This saves time and money. The trick here is that adhering to the fail fast principle, we improve the feedback loop. We quickly reveal all problems in our software making it easier to spot and fix them. If it’s impossible to get a feedback from the feature owner then try to demonstrate this functionality at least before releasing to the production environment to avoid situations when something unwanted “goes live”.
Tip #4: Establish strict requirements for appropriate bug description and become (at least temporary) a filter of all newly created bug to check whether they meet these requirements or not. Try to suppress creating of not-fully-investigated problems, try to find the root cause of the problem, put as much information you can find (logs, service response, screenshot, data and steps to reproduce).
Tip #5: Create a united entry point for all complaints and suggestions about improvements. We created a “hotline-problems” mailbox (rfbug@copyright.com , RF is the name our project) and make it the alias of all QA guys of this project. If somebody notices something wrong or just want to make sure how a functionality (in terms of RF) works he can send an email to this address and he receives (almost immediately) an answer on it from members of the QA team. It helped to control all bugs which have been created in BTS by different persons (from different projects, departments, branches) and avoid duplications of existing in BTS issues. At the same time, testers started applying their expertise and supporting in the investigation of “urgent-emergency” issues in production.
All of these changes will help to stop a “flood” of new bugs, and now it’s the time to perform a spring-cleaning of your bug tracking system. It is a one-time exercise to leave only 100% clear bugs without false conclusions, duplications and other “trash”.
Tips #6: Find the best time to perform such activities. Allocate some time and efforts of QA team to do that in short time frame. In our case, we had a whole spare week in January without new functionality to test (because our dev team located in another country and they had long Christmas holidays). You can ask your product owner to create a separate story to do this spring-cleaning by QA team while developers will be busy with spikes, code refactoring, test coverage, etc.
Tip #7: Find and close all unreproducible bugs. Having more than 300 bugs makes managing them almost impossible – your application grows and changes, some bugs could be fixed or became outdated because of those changes. About 25% of bugs have been closed with a resolution “Cannot Reproduce”. It helped to avoid wasting of developers time on investigating of a unexisting problem.
Tip #8: Find root causes of bugs and get rid of issues which contain words: “Sometimes”, “Intermittent”, “For this user”, etc. All of these “bad” words are signs of not-fully investigated problems. If a bug is only reproducible for this user – find out what the difference between this one and others, try to find all conditions which help to reproduce bugs with sign “sometimes” and “intermittent”, etc. If you can’t find out the root cause right now, put it on the list for future investigations. Help developers to get all information about a user, conditions, browser, attach log files, etc., make all bugs 100% clear to everyone.
Tip #9: Find all duplicates. After applying of procedures described in previous tips it’s much easier to search for duplicates because now you have to have only reproducible issues with a specified root cause of them instead of described external presentations. There were almost 10% of duplicates.
Tip #10: find and close all functions as designed and “ancient” low priority bugs which don’t deserve to be fixed. Find all requirements for your application, compare expected results from issues with requirements. Also, search for low priority bugs which don’t harm users and which are not worth fixing. Frankly, there are always some bugs stored in BTS which have been pushed from release to release, because they are not important, find them and reconsider them with stakeholders. Most likely, you will find bugs which actually are not bugs at all and they should be reviewed by BA and PO to find out do we need to update requirements and fix this issue or it should be closed. In our case, more than 15% bugs have been closed with the resolution FAD.
Before I go to the next tip, I want to raise the question: should we consider bugs as a part of the product backlog. I mean bugs that are unrelated to functionality being coded during the sprint. If someone finds a bug during a sprint that is related to the features being worked on, it definitely should be fixed, It’s obvious.
Problems your team finds during feature testing:
- Issues introduced by the changes (new ones)
- Issues that were there before and doesn’t have a lot of impact on the “touched” functionality (old ones)
- Issues that were there before and has a critical negative impact on the “touched” functionality (old ones)
The developers need to fix the items that were there beforehand and which negatively impact their work, as well as the problems they accidentally introduced.
Now for the more difficult side – honestly, most of the pre-existing bugs that a QA team finds, do not have a major impact on the feature, because they are already in a production environment (and no complaints from customers) and it really isn’t the developers’ responsibility to fix them. And in this case, all of these old low-priority bugs go into the bug tracking system.
From another side, ideally, we should put the bugs right onto the product backlog. Because from a user’s perspective it does not matter whether something is considered a New Feature or a Bug (related or unrelated to coded functionality). The user just wants the system to work appropriately.
In these cases, the most common solution team’s use is to have two backlogs
- A product backlog of features
- A bug backlog
The product owner takes on most of the additional work by having to prioritize two separate queues. But QA folks can help with that, providing a list of issues which, from their point of view, are the most important. So, before the sprint starts PO can say something like this: “My top priorities are the first 6 items on the product backlog, they are “must have”, then switch to bug fixing – RRS-1203, 1205 and 1206, then these next 3 stories from the product backlog”.
I still insist in a perfect world bugs should belong to the product backlog just like any product story.
After all, as the “Manifesto for Agile Software Development” says, “Working software is the primary measure of progress.” It is a little insincere to mark a feature with known bugs as complete — “Yes, it’s done… but there are a few bugs in it,”
Tip #11: Transform some bugs into improvements and stories. We have conducted a series of triage meetings to explain BA and PO the root causes of issues and ways how we can fix them, some of the issues has been transformed into stories because business decided to combine bugs related to some functionalities and fix them in one fell swoop (for instance, issues reproducible on mobile devices). Also, we found some bugs related to the parts of the application which will be removed (or replaced by another functionality) soon, one part of these issues has been closed another one has been transformed into improvements.
Tip #12: Pull bugs, related to a functionality which will be changed in terms of the next sprint, in the scope of stories. We created some components in our BTS to mark all bugs related to appropriate functionalities (components), like: “order process”, “user management”, “account settings”, etc. It made easier the process of finding bugs related to particular functionality.
Before each sprint started, QA guys were finding a group of bugs which bound to changing in an upcoming sprint functionality. We proposed PO to include them as a part of these stories and fix them. Usually, we transformed these bugs into additional acceptance criteria of these stories.
After these activities, we got a list of 100% clear bugs with a specified root cause, without unanswered questions inside, with an appropriate priority and which definitely should be fixed.
This list contained only 49 bugs. We allocated a whole sprint (stabilisation sprint) to fix most of them. After that, we go to our “dream” (to have 0 bugs in our BTS) with small steps sprint by sprint. At the moment, we only have 7 bugs.
Don’t be afraid trying new things in your process! Do not be afraid to take on more responsibilities! And everything will pan out!
Edited for web by Cecilia Rehn.
