As Scrum is the most popular framework adopted by organizations adopting an Agile approach for project management, many companies are trying to find financial facts that justify its adoption. This article discusses the topic of evaluating the return on investment (ROI) of using Scrum and Agile project management approaches. It suggests some hints about mistakes to avoid and on how to get meaningful results from this activity.
Author: Neri Lavi, QualiTest Group, http://www.qualitestgroup.com/
Many strategies are supported as the “best” management of a software development process. While each programmer has his favorite way to code and every manager has his own method of administration, a software company’s goal is to always find the most efficient process to maximize profits.
A primary concern in any industry is the rate of return of an investment, or Return on Investment (ROI). The ROI represents the benefits gained from the investment versus the costs that was expended. For each decision that managers make, they should be considering the ROI, and collaborating with the customer to ensure that everyone agrees upon options that maximize the ROI. Any backlog item should add value to the project. While it clearly costs to implement a new process for a team, the benefits are more difficult to calculate.
Scrum is a process that has become popular recently. Its main purpose is to increase the entire project’s ROI. To choose the approach that is correct for the project, you must determine a method to effectively compare the different approaches and to establish which will gain a better ROI. However, an issue exists in comparing development methods. Numerous studies used pseudo-science and estimation to evaluate each method. Their conclusions are based on how the method fits into Agile principles or how the companies feel that the method is working out for them. In one study, project managers were asked to compare their experience with popular processes, and a conclusion was based on the popular choice. This is bad science. In other studies, metrics have been developed to estimate the costs of these methods and perceived benefits. For example, one study calculated costs and benefits by the lines of code that were produced. However, there is no guarantee that a greater number of produced lines increased the effectiveness of the team, and the same method may not work for your company or your project. The field of software development process evaluation needs to be advanced in a more empirical and studious manner.
While a methodology may work for one company, one size does not fit all. You should start your development process by evaluating different methods and assessing the gains for your own business. The first step in comparing methods is to establish a scenario where two or more processes can be evaluated through the same guidelines. This can be achieved in two ways. The first approach entails comparing two projects that are similar in size, scope and cost, then developing each project using a different process. However, this approach does not guarantee success, as two projects are never exactly the same. It can also be difficult to transition a team from a traditional methodology to an Agile methodology. A better option is to take a large project and break each process into a small, medium and large piece, allowing you to compare the processes together. You should perform the testing processes against the traditional methodologies, and then transition the team into the Scrum methodology by performing the same processes in an agile way. While this approach has some of the same pitfalls as the two projects approach, the risks will be mitigated by the number of pieces and by the transitory approach. The “piece approach” could take a couple of iterations, as your team needs to get used to the Scrum process.
After the test scenario is set up, we need to measure hard values that can be used to calculate a ROI. One such measurement is the time required establishing a stable build. While this definition may be different per company or project, a good measurement of a stable build is one that contains no critical bugs. By taking this measurement and calculating the man-hours used, we can have a hard value of which method costs less. Another method is the number of critical bugs found, as they require considerable resources to correct. By correcting the bug in both methods, you can measure the resources used in both methods and compare the expense of both. By measuring these values, you can compare any savings to the amount of money invested, thus calculating your ROI.
It is important to note that Scrum and other Agile methods boast benefits that are hard to measure empirically. While Agile methods are implemented because they improve flexibility, how can we concretely measure the consequential flexibility of a method? One option is to analyze how our test scenarios deal with added features and measure the time it takes to incorporate them. Scrum boasts a collaborative process that increases communication between your employees. You could track communication errors through emails or other modes and see how many mistakes were made. Another principle of Agile is customer involvement; the primary goal of this is to increase customer satisfaction. We could survey the customers after the projects are delivered to see which process they felt was better. Furthermore, some employees could work better under different processes, and their opinions should be taken into consideration. However, these benefits are difficult to empirically measure and thus are not easily added into the ROI.
There are many proposed ways to estimate or speculate the ROI of Scrum. Studies may suggest methods for your company’s project based on size or scope, but these proposals will never be as good as testing the process in your environment and measuring your own results. Scrum and other Agile processes have great values and intentions, stressing business value and delivering a better product, but that still does not mean that these are the best practices for you, your company or your project. Furthermore, you must take into consideration that when learning a new methodology, a team’s initial attempt may not be representative of the best work that can be produced with that methodology. It is therefore important to find a method that fits you and your team.
About the Author
Mr. Lavi is a Sr. Test Specialist at QualiTest Group and manages QualiTest’s western US operations. Prior to joining QualiTest, Mr. Lavi acted as a QA manager at a global software company managing distributed testing teams as well as acting as a ScrumMaster. Mr. Lavi is a software testing expert with strong experience in managing test teams and implementing Agile processes. He has published multiple QA Testing and Scrum articles and has delivered lectures and training seminars worldwide. In addition, Mr. Lavi has extensive experience in implementing automated testing tools and methodologies.