Abstract
Scrum is an agile project management technique with high transparency, adaptive control, reasonably accurate release forecasting, and high productivity. It helps organize efforts involving teams and creativity (including software engineering, product design, venture capital management, charities, civic organizations, marketing communications, etc.). Because Scrum exposes managers and team members to greater visibility and accountability than traditional waterfall approaches, and because it requires different management structures, some organizations encounter resistance implementing it. Scrum focuses on management, so it can be combined with field-specific agile techniques, such as Extreme Programming.
Problem
The most common project management technique today is called "waterfall", reflecting that development flows from phase to phase, ending in the release of a completed project. In waterfall management, all the requirements are gathered up-front. Requirements gathering can take months. Product managers consider the existing marketplace, hypothesize future needs when the product will be released, estimate the development time needed to implement each requirement and analyze dependencies between development activities, putting together a Gantt chart that forecasts a release date. The development team develops the product, then the QA department tests the product, then the system is released as a beta, then bug reports and feature requests come in, then developers change the system, and out pops a completed product. Right on time. In theory.
However, once development starts, the Gantt chart typically loses credibility rapidly. Developers working under optimistic assumptions discover implementation difficulties—the creative process is not so easy to predict. Product requirements thought to be important at project initiation become irrelevant, superseded by competitive forces, new market developments, or better market understanding. The product managers add new requirements to the list. Finally, when the customers view the beta, surprises ensue. Typically, customers provide new requirements no one imagined, and ignore features all assumed were essential. The release goes out late. In an ongoing effort to "fix the situation" with the next product release, management often demands more meetings and more advance planning, taking more time before development starts.
Historically, waterfall forecasts are notoriously wrong; no one is shocked by projects forecast to release in 18 months that took 36 months. With late releases, organizations often sacrifice quality for speed. Companies can achieve bankruptcy this way; developers can lose their jobs and careers. In some situations, people have likely died.[6]
Solution
To solve this problem, reconsider some assumptions. If requirements obtained upfront are likely to radically change, a system that adapts rapidly to change would likely work better. If developer work estimates are reliably bad, find a way to map developer work estimates into actual developer time. If expensive advance planning results in marginally better or worse results than no planning at all, then plan over shorter horizons rather than longer.
Scrum is a software management and planning technique that balances these goals:
- Stakeholders can re-prioritize developer efforts at least monthly,
- Outsiders can experiment with full implementations of some features first hand at least monthly,
- Developers can measure their productivity empirically,
- Stakeholders can forecast release dates based on feature set, or tune the feature set to meet a release date,
- Developers can focus on their work uninterrupted, while simultaneously communicating beneficially in highly controlled, structured meetings, and
- Developers can refine their processes and work activities through feedback loops and self-organization.
Scrum was originally developed by Jeff Sutherland [1] and Ken Schwaber[2]
People
A Scrum Development Team comprises a Product Owner, between three and seven Developers, and a ScrumMaster. The Team is jointly responsible for the product, not only functionality, but quality, releasability, documentation, etc.
Team members are called Pigs; outsiders are called Chickens. The terminology arises from a joke: A chicken proposes to a pig that they form a restaurant together called "Ham and Eggs." The pig demurs, stating "I don't think it's a good idea. You'd just be involved, but I'd be committed."
The Product Owner in Scrum articulates stakeholder needs in the form of Product Backlog Items or User Stories, and assembles those Product Backlog Items into a prioritized list called a Product Backlog. There is only one Product Owner per Team (but several teams can have a single Product Owner), ideally there is only one primary Product Owner per Product (which may involve multiple Teams). Having only one Product Owner provides a "single wringable neck" for upper management, and a single authoritative source of prioritization and clarification for the Team. The traditional equivalent of the Product Owner is the "Product Manager."
Developers in Scrum include software engineers, user-interface designers, quality assurance engineers, database engineers, operations "shepherds", and other people who commit to a significant amount of work for the team. Ideally, development efforts are self-contained within the Team, so outsiders are not required to produce a releasable product. Furthermore, Teams and Developers are encouraged to be self-contained and cross-functional. If Developers don't know a skill, but that skill is needed by the team, they are encouraged to spend the time to learn and apply it.
The ScrumMaster facilities self-organization, supports the Team and Product Owner, enforces the process, encourages activities that improve the quality and likelihood of developing a releasable product, and serves as a sacrificial lamb to interruptions that would otherwise defocus developers. The ScrumMaster protects Developers from direct appeals from the Product Owner or other stakeholders. This supports Developer productivity: when Developers are interrupted, their productivity can drop dramatically. The ScrumMaster also works on behalf of developers to remove impediments, especially when those impediments arise from outside the team. The ScrumMaster is responsible for pro-actively providing transparency into the Team's progress and historic productivity information to outside stakeholders. The ScrumMaster encourages the team to improve its practices and make its work more feasibly releasable.
Process
Developers attempt to produce a Feasibly Releasable Product in a fixed time-period, called a Sprint. Each Sprint contains a Sprint Planning Meeting, Sprint Work, a Sprint Review (or Sprint Demo) and a Sprint Retrospective. Sprints occur in succession; in most cases there is no delay between the end of a Sprint and the beginning of the next.
>Using the same duration for successive Sprints provides for easier planning and more reliable release forecasting. The maximum Sprint duration is a month; other common choices are four-weeks, three-weeks and two-weeks. In my experience, new practitioners of Scrum often start with longer Sprints. As they gain facility with Scrum and incorporate more efficient quality automation, they can reduce the duration.
All Scrum meetings are Time Boxed, meaning that they start exactly on time, and they last no more than a pre-established duration. A good meeting leader (in Scrum, almost always the ScrumMaster) monitors a meeting's quality: when its cost begins exceeding its value or veers from its goal, the leader should immediately discuss that observation with the team, and help redirect conversation or end the meeting. Meeting leaders may jot down "topics for later" to keep a meeting on track, inviting optional ad hoc discussion on those topics after the official meeting ends. If a meeting leader is not present, another member starts and runs the meeting. If members arrive late, they are penalized (a fine donated to charity is common). By enforcing start times vigorously, the process recognizes the enormous cost of idle developers, not only in squandered salary but also in defocused productivity.
Sprint Planning Meeting
In the first half of a Sprint Planning Meeting, the Product Owner presents the highest priority Product Backlog Items. For each item, the Product Owner describes it to Team member satisfaction, and obtains a jointly established effort estimate from the team. Many teams use Planning Poker to obtain this estimate[3]. New practitioners typically estimate Product Backlog Item work effort in units of Ideal Developer Days, but sophisticates use Story Points. When more than a Sprint's worth of effort has been estimated in total, the team selects a subset of high-priority backlog items it believes it can complete in the Sprint. This subset is called the Sprint Backlog; it is comprised of Sprint Backlog Items. With the Product Owner's consent, the Team can choose to skip a high-priority item. This can help focus the Sprint on a single theme, or defer items that need further evaluation.
As an Arab or Israeli user, I want the system to display text right-to-left.
- Code unit-tests and right-to-left language support in client application
- Create Arabic and Hebrew localization strings
- Store Arabic and Hebrew strings in properties files
- Code unit-tests first, then code web server transmitting localization strings to client
- Design acceptable user-experience for Arabic and Hebrew language and users
- Code unit-tests first, then code web server providing designed user-experience
- Code integration test confirming system operates end-to-end in Arabic or Hebrew when Saudi Arabia and Israel locales are selected, respectively. System passes
Sprint Backlog Item with Tasks
In the second half of a Sprint Planning Meeting, the team breaks each Sprint Backlog Item into development Tasks, as in the example above. For each Task, one or more Developers volunteer, and those volunteers agree on the estimated work hours for the Task. Group estimation processes are rarely used for Tasks. There is no need to make sure the estimated hours in the Tasks sum to the estimated days for the Sprint Backlog Item. When the second half is complete, the Sprint Planning Meeting is over and developers begin their Sprint Work.
Sprint Planning Meetings should be time-boxed to 8 hours, typically 4 hours for the first half, 4 hours for the second. New teams or new backlogs may require all of this time. The Product Owner and ScrumMaster can make these meetings much shorter with Backlog Grooming.
Sprint Work
In Sprint Work, the Team starts tackling the work, beginning the highest priority Task in the Sprint Backlog, and moving down the list. A productive subset of Developers focuses on the topmost Sprint Backlog Item, taking on individual tasks. Another subset focuses on the next highest Sprint Backlog Item they can productively address (there may be dependencies or skill set issues), etc., until all Developers have something to do. If a Developer finishes all his or her Tasks, that Developer looks at the remaining work to see where he or she can help others, or move on to another Sprint Backlog Item. If no other Sprint Backlog work seems suitable, the Developer considers whether he or she can improve overall product quality, automate build processes, or otherwise improve the productivity of the team. As a last resort, a Developer might pull the next highest priority Product Backlog Item outside the Sprint Backlog, and attempt to complete it.
Each working day, the Team attends a Daily Standup Meeting (sometimes called a Daily Scrum or Scrum Meeting). Ideally, this meeting occurs at the same time every day and lasts no more than 15 minutes. Each Pig answers three questions: What did you do since the last Daily Scrum? What do expect to do before the next Daily Scrum? What impediments are currently blocking your progress? When all Pigs have answered, the meeting is done. Pigs can ask questions during the meeting, but, by virtue of the time-limit, answers must be kept very brief.
Sprint Burndown Charts
Sprint Burndown Charts help a Team track progress during a Sprint. The vertical axis is the total remaining estimated work hours for the Sprint, the horizontal access is work days. Every day, developers update the current Sprint's task list with the remaining estimated hours. If developers encounter difficulties their estimates can increase or if they add new tasks to a Sprint Backlog Item (both are allowed by Scrum), a task's estimated hours can go up. The ScrumMaster sums the remaining estimated hours at the end of the day and puts it on the graph.
In an ideal world, the Sprint Burndown Chart shows a straight line from the total estimated hours on the first Sprint day, converging to the X axis on the last day of the Sprint. It never happens; if it does, you should question whether the ScrumMaster is "cooking the data" to make it look perfect. Developers often discover estimated hours that were forgotten. When they are added to the estimates, the burndown line may diverge for a day or two.
Triage is a common activity mid-Sprint, for some Teams. If a Team discovers it cannot complete all the items in a Sprint, it revisits the Sprint Backlog with the Product Owner, makes sure that it can do enough backlog items to deliver a Feasibly Releasable Product, and moves low-priority items out of the Sprint (or leaves them in, but all agree they will be tackled only after everything else is done). Teams can also refactor Sprint Backlog Items to make them address part of a feature, preserving releasability, and add another Product Backlog Item to complete the job in a later Sprint.
Sprint Review
At the end of the Sprint, the Team hosts a scheduled Sprint Review Meeting or Sprint Demo. All stakeholders are invited. For each item in the Sprint Backlog, the team demonstrates, to the satisfaction of the Product Owner, that the Sprint Backlog Item was completed properly in the running product. In the best case, a third party installs the product following instructions written by the team, and the third party attempts to demonstrate each Sprint Backlog Item, thus showing that the product is releasable. In the worst case, the team asserts that it is done, with no validation. A responsible Product Owner in this situation should reject the Sprint Backlog Item, causing it to remain in the Product Backlog, to be later picked up in another Sprint. If all or most Sprint Backlog Items cannot be third-party validated, the Product Owner may consider sidelining or reconstituting the team.
Following the Sprint Review Meeting, the iterative cycle begins again with the next Sprint Planning Meeting. Note that during the Sprint, no one is allowed to change the Sprint Backlog without the unanimous consent of the entire team. However, Product Backlog Items outside the Sprint Backlog can be re-prioritized, rephrased or eliminated, and new Product Backlog Items can be added. This allows product development to adapt to new understandings of markets, product ideas, better assessment of Team skills, thoughtful assessment of risk, etc.
Sprint Retrospective
After the Sprint Review, the Team hosts a time-boxed 4-hour Sprint Retrospective to discuss and refine the process. This first involves taking pride in what was accomplished during the Sprint: What did we do right? Scrum is much harder than it may first appear, especially in dealing honestly and transparently with stakeholders and management.
The meeting also talks about improvements: What can fix, add or remove from our process that would achieve our goals better? Typical retrospective topics include revising the Done Criteria for Sprint Backlog Items, Tasks and Feasibly Releasable Products, finding more efficient ways to groom the Product Backlog, deciding how to split up Product Backlog Items better among multiple teams, considering new seating arrangements, deciding who can communicate a release forecast, etc.
Teams often spend 30 minutes to an hour listing possible improvements, then prioritize the list, then spend the rest of the time discussing specifics about how the highest priority items will be implemented in the next Sprint. Organizational impediments may block some improvements; if so, the ScrumMaster should work with managers to remove those impediments.
Release Forecasting
Velocity is a Team's historical average productivity in estimated-programmer-days per Sprint. The Sprint Backlog Items used to compute this number must have been completed in the Sprint where they are counted. If the Team estimates Product Backlog Items in advance, the team's Velocity can be used to forecast when any subset of those items could be completed.
A Product Burndown Chart displays a Team's Velocity, its progress in estimating future items, and the addition of new Product Backlog Items as the Product Owner and stakeholders gain experience with each Sprint's Feasibly Releasable Product.
The figure shows 100 estimated programmer days at the onset of Sprint 1. There were Product Backlog Items that had not yet been estimated. They were estimated during Sprint 1, and appear in the Sprint 2 bar, minus the estimates of Sprint Backlog Items completed in Sprint 1. From Sprint 2 to Sprint 3, you see the pure effect of the team's Velocity in completing about 22 estimated days of work. From Sprint 3 to 4, we see that the team burned down about 44 estimated days of work, but about 15 estimated days in new Product Backlog Items were added after the Product Owner showed the Sprint 2 feasible release to stakeholders. From Sprint 4 to 5, the team burned down 40 estimated days of work, but the Product Owner added 15 estimated days. At this point, we can project a possible release product at the beginning of Sprint 7.
Product Owners can consider how dropping different features from the Product Backlog might affect the release date. Of course, this must be done with thoughtful consideration of dependencies that might exist in the ordering of Backlog Items.
Challenges
To waterfall managers, Scrum is an alien process. To adopt Scrum is to admit that years of reliance on defined process planning—with its immediately-out-of-date Gantt charts and never-reached project deadlines—was wishful thinking all along.Scrum places release forecast responsibility on the shoulders of Product Managers, who tune the release date based on the feature set they demand, and on empirical productivity metrics about the team. Using a waterfall process, upper-managers could blame their optimistic developers for a late release. But let's state the truth, it was usually worse: Most of the time developers had no chance to set a realistic schedule, rather a manager created an unrealistically aggressive schedule, perhaps in optimism or perhaps in an attempt to motivate productivity.
For developers, Scrum can be a challenging process. The term "Sprint" is very descriptive. I've had more than one developer state that "Scrum is intense." Scrum and other agile methods make it hard to hide sloth: developers have to report to the team daily, they have to release a working product monthly. Typically I've found that insecure people, rather than challenging the process itself, seek ways to limit transparency by modifying Scrum. They might suggest that Daily Standup Meetings occur only when needed, rather than daily, might find ways to avoid investigating the releasability of a sprint product, or might avoid test frameworks that can expose regressions.
Scrum challenges our job descriptions. Engineers become members of a highly-communicative, multifunctional team, and the team succeeds or fails together. The team is highly motivated to create automated unit, regression and integration tests. The team must determine "what is done", and hold itself to that standard. Does this mean the team is entirely responsible for QA? The answer to this question is different depending on the organization, but I can say, in Scrum, the team becomes gradually more and more involved in preserving and improving product quality. New projects have more flexibility; the team takes on QA tasks from its inception.
Group managers and directors have less-defined roles. They become change advocates, auditors, teachers. Sometimes they must get their hands dirty getting teams started, or making up for deficiencies that emerge. They must diplomatically push people in the right direction, making the process more efficient, keeping people happy and preserving their own integrity. To do decent performance reviews or allocate developers to teams, they must attend a lot of Daily Standup Meetings—if raises are misaligned with performance, it will be clear to the whole team.
With transparency comes unvarnished honesty and vulnerability. At one point, I was managing a Scrum-based project, but working for a waterfall manager. Every day was exhilarating, because my team led the way for others, and was scary, because I worried I would annoy people too much with my bluntness and get fired.
An obvious challenge: How do you scale this to a larger engineering group? Most all-Scrum departments use “Scrum-of-Scrums,” where teams aggregate all their product backlog items together into a giant list, and plan many backlog items together [4]. Sometimes they create and manage priorities around "Epics". I have never seen either approach used to manage multiple products. My company, Citrix Online, developed something different, which we call Enterprise Scrum. It handles multiple products, it is efficient, and it makes impediments and velocity visible at the highest levels of the company. You can find out more at my blog [5].
Summary
Scrum is an adaptive software management process that provides more visibility, more control, more accurate forecasting, and more productivity than standard waterfall processes.A Scrum Development Team has three participant types: one Product Owner, one ScrumMaster and 3-7 Developers. Development proceeds iteratively, with fixed-duration development periods, called Sprints, lasting no more than a month. Each Sprint contains a Sprint Planning Meeting to determine what will be tackled in the Sprint, Sprint Work to complete development tasks, a Sprint Review Meeting showing off running code that could be suitable for release, and a Sprint Retrospective to tune the process.
Scrum includes three feedback-loops: Daily Standup Meetings help the team adapt rapidly and resolve problems together, Sprint Burndown Charts allow the team to review its progress and perform Triage to produce a working product in a short period. Sprints allows marketers and stakeholders to change development priorities on a monthly basis.
Product Burndown Charts provide the ability to forecast releases empirically, incorporating both developer productivity metrics and requirement addition trends into the forecasting process. They make planning and resource allocation easier. They provide a fourth feedback-loop, showing how product management choices affect release dates.
Jeff Sutherland assesses a company's agile maturity by asking CEOs these two questions: What are your teams' Sprint velocity? What organizational impediments have your teams identified, and what are you doing about them? To answer these two questions thoughtfully: your company must divide into Scrum Development Teams; each team must perform all the functions listed here; the teams must report velocity and impediments; upper-management must roll these data up for the CEO; and the company must be able to analyze and unblock team impediments. Simple questions are not always easy to answer.
As a software engineering manager, I've not found anything better than Scrum. I've worked in large and small companies, as everything from developer to CEO. In managing developer groups, I've long looked for a methodology I could apply with some rigor to deliver products on-time with a predictable feature set. I've used Rational Unified Process (RUP), Unified Modeling Language (UML), defined-process planning with Microsoft Project and Gantt Charts. None of these satisfied, though I have found some value in each.
In applying Scrum, I've seen greater productivity levels than I expected, and refreshing honesty and adaptability. It makes my work much more satisfying: I feel closer to customer needs, I think team members work at nearly peak performance. While we are typically exhausted at the end of a day, we also go home evenings and weekends and enjoy them.
References
- Jeff Sutherland, Scrum Blog, http://scrum.jeffsutherland.com
- Ken Schwaber, Agile Project Management with Scrum, Microsoft Press (2004), Amazon
- Mike Cohn, Agile Estimating and Planning, Prentice-Hall (2008)
Amazon - Mike Cohn
Advice on Conducting the Scrum of Scrums Meeting - Dan Greening's Scrumerati Blog
http://scrumerati.com - Anthony Finkelstein, Report of the Inquiry Into The London Ambulance Service
(February 1993), International Workshop on Software Specification and Design Case Study
http://www.cs.ucl.ac.uk/staff/A.Finkelstein/las/lascase0.9.pdf
Postscript
At the time this was written, Citrix Online was my employer, and supported my explorations and advocacy of agile management and marketing techniques. Many of its development teams are geographically distributed; we used Citrix Online's GoToMeeting product for our Daily Standup Meetings. Ron Lussier assembled the Scrum Cycle figure shown above.
This article was originally posted to Google Knol on October 30, 2008. It quickly became a top-pick, as well as a “Knol for Dummies” post. At the time of this post, it had over 12,000 page views. Google Knol is being decommissioned, so I have transferred this article to Scrumerati.com. If you are interested in Scrum and agile methods, particularly relating to enterprises or product management, I invite you to subscribe to Scrumerati.com by entering your email on the right side, or by adding its feed to your RSS reader.
