Scrumerati

Enterprise Scrum:Scaling Scrum to the Executive Level

My paper on scaling Scrum to the executive level has been published by IEEE. This paper was selected as the best paper in the agile track at HICSS. Download it here:

Daniel Greening, “Enterprise Scrum: Scaling Scrum to the Enterprise Level,” 2010 43rd Hawaii International Conference on System Sciences (HICSS), Koloa, Kauai, Hawaii January 5-8, ISBN: 978-0-7695-3869-3 (10 pages)

Here is the abstract:

Our company manages 25 software engineering teams across 6 products using a single top-down Enterprise Scrum. We know of no other company doing this, yet it provides extreme visibility and control at the CXO level. It promotes agile thinking enterprise-wide, driving non-engineering departments to adopt Scrum. We believe it is making us more profitable.

We estimate effort in team months, run quarterly Sprints, assign whole teams to projects, meet in weekly stand-ups. We start, postpone or cancel whole projects.

Within individual projects, we still use 1-4 week Sprints and all the trappings of the classic Scrum process, including, in some cases, Scrum-of-Scrums.

New challenges arise: Shared resource constraints suggest Kanban methods. Net Present Value can justify prioritization, but creates controversy. Moving teams between projects requires rapid programming environment setup. The process forces executives to justify decisions. We want simple improvement metrics, but they seem elusive.

We are actively adapting this process at Citrix Online. I would love feedback and suggestions related to either the process or the paper itself.

Need Your Help: Agile Abandonment?

Several companies with a strong history of using agile methods, even "hyperproductive" variants with published metrics, have abandoned their commitment to agile, sometimes accompanied by dramatic staff defections. There seems to be a correlation (not necessarily causation) with departure of key staff members. I am especially curious about this because, years ago, this happened to an employer of mine. Whether due to management failures, or failures of the agile framework, I want to understand and improve this situation.

Open Space Session, Scrum Gathering Orlando 2010

I am currently researching this phenomenon, to explore the root causes in a future paper. I do not want to bias our research by hosting a public debate, instead preferring to have one-on-one email or phone discussions with parties involved.

I will share drafts of our work prior to publication with any participants.

If you are aware of a company abandoning a commitment to agile methods, whether you were directly involved, or are aware of others I could interview, please contact me. Dan Greening dan-abandonment@greening.org, cell +1 (415) 810-3693. If you wish to remain anonymous, whether in referring to others or discussing your own direct experience, I will honor that request.

Please refrain from publicly commenting on this post, and rather contact me directly, to avoid biasing research participants. (Crossing my fingers and hoping that works.)

Enterprise Scrum: Scrum Gathering

On 9 March 2010, I gave a talk on Enterprise Scrum at the 2010 US Scrum Gathering in Orlando, Florida. I am grateful that about 50 people showed up for my talk, from about 300 total Scrum Gathering attendees. People were intrigued by fractal thinking, by the blunt assertion that engineering teams rapidly burn money, and by the prioritization of work using forecasted Net Present Value. Enterprise Scrum can create very healthy product lines and companies. In a sense, Enterprise Scrum turns enterprises into internal venture capital funders.

Despite that large Scrum implementations can dramatically increase the productivity of 250 to 30,000 developers, the exotic world of agile methods confuses or scares most large company executives, if they even know of agile methods at all. It's too bad, because deploying Enterprise Scrum or other "large Scrum" variants could likely save a few dinosaurs from extinction.

I love sharing these ideas with others. My personal mission is to help society produce more value (happiness, etc.) with less work, for everyone. I would be happy to present Enterprise Scrum to your organization or executive team, if it resonates.

Here is the background information on the talk.

Speaker Biography

Dr Dan Greening is Citrix Online's Director of the Agile Program Office. This team provides Program Management, ScrumMastering, User Experience Testing and User Interface Design services to the company. He was the primary evangelist in converting Citrix Online to be a pure Scrum organization, and advocates for Lean, Theory of Constraints, and other continuous improvement frameworks. His paper on Enterprise Scrum was nominated for best paper in the 2010 Hawaii International Conference on System Science.

Greening is co-founder of three previous startup companies, two of which realized moderate liquidity.

He was Principal Investigator on 3 National Science Foundation SBIR awards.

He holds Ph.D. and Masters degrees in Computer Science from UCLA, and a Bachelors in Computer Engineering from University of Michigan. He is a Certified Scrum Professional. He published a paper on stochastic optimization and fractals in the proceedings of Santa Fe Institute's Complex Systems Summer School.

He has studied the emergent properties of large organizations for decades. In 1986, while a computer science graduate student, Greening was elected President of the University of California Student Association, representing 144,000 students at 9 campuses. Under his leadership, UCSA created self-organized teams that successfully championed state legislation for improved student health care, student safety, and financial accountability in the University.

Abstract

Enterprise Scrum is "Scrum Writ Large," a continuous improvement process modeled after Scrum, but scaled, using fractal thinking, to a much larger organization. It is not Scrum-of-Scrums. It was developed in late 2008 at Citrix Online, and has been iteratively improved since then.

Enterprise Scrum, as implemented in Citrix Online, has quarterly sprint cycles, weekly standup meetings, effort measured in "estimated team months," and prioritization partly based on Net Present Value. Planning is iterative, to avoid squandering developer time estimating effort until the last possible moment. Several resource allocation strategies have been explored, everything from total self-organization to traditional approaches. Through a series of retrospectives and adaptation, a hybrid is emerging.

Evidence for Enterprise Scrum's success is anecdotal, but strong: Executives know and plan against the engineering department velocity, remove high-level productivity impediments, and promote agile in their own departments. Because the value of key engineering projects, including technical-debt reduction, is understood and prioritized at the CXO level, not hidden in the priesthood of the Engineering Department, there is a healthy appreciation of the value engineers provide. We have seen situations where other departments voluntarily redirected funds to engineering, because engineering could prove its high profitability.

Topic

Scrum practitioners regularly face problems gaining advocates at the top levels of medium to large organizations. In part this results from Scrum's focus on individual teams and small features, while large organizations assess and prioritize major projects largely around profitability, mission synergy and cost.

To successfully promote agile and continuous improvement processes at the executive level, we must understand corporate values, incorporate them in our processes, and explain process utility in ways that executives understand. We must provide executives with tools that let them easily prioritize projects and, once they have started, identify systemic and project-level blockers that impede progress.

This talk will likely help agile practitioners working within large organizations, and help people considering investing in or initiating startup companies. The concepts are similar for both. If you have a large organization, and you are competing with startups, you may need an approach like Enterprise Scrum to remain viable.

Objectives

• how to easily drive agile understanding into the whole organization.
• how to apply fractal thinking to Scrum and other continuous improvement techniques.
• how to reduce apples and oranges comparisons (such as comparing a feature project against a technical debt reduction project) to Net Present Value.

Pushing Agility Upstream

Most agile teams live in a waterfall ecosystem. When powerful stakeholders—like business partners, customers, other departments or executives—demand future commitments, while interrupting contributors with unprioritized demands, smart Scrum teams raise a protective shield. They make the Product Owner manage stakeholder priorities, and make the ScrumMaster defend them against interference. If anyone provides a date to a stakeholder, it's going to be the Product Owner: the Single Wringable Neck.

But instead of just defending ourselves against an external onslaught of unprioritized need, we can go on the offense. We can evangelize agility upstream. Impossible requirements point out that stakeholders could themselves benefit from agility. If we teach them agility, they and we will both go faster. Together we can develop a strong, sustainable and profitable business ecosystem.

My buds and I have pushed agile upstream twice recently, and it's more fun than you might think.

Example 1: Marketing

Traditional marketing departments demand that staff respond to immediate needs: the "hair on fire" approach to planning. In the past, engineers working closely with marketing have accepted these interrupts, and some of our own past work reflects the ragged architecture and appearance you might expect. Expensive rework wasted engineering time. Bugs caused significant problems.

In 2007, our engineering department adopted Scrum. After that, marketers ran headlong into ScrumMasters defending engineering sprint boundaries. Marketing's first reaction was to find ways around Scrum restrictions, demanding their own dedicated teams, or time-boxing vaguely defined marketing related work that engineers would have to fulfill.

Engineering colleagues at first tried to satisfy these needs, but then realized marketing demands were distracting them from important work. I assumed we just needed to explain ourselves better. Scrum meant we wouldn't be able to satisfy marketing needs the same way. Marketing would have to prioritize needs earlier, based on our work estimates. They would have to allow us to say "No" to low priority work. Nick Kim (another evangelist ScrumMaster) and I started a conversation with key managers.

But as we talked, we realized that marketing suffered the same problems our we had experienced in the Bad Old Waterfall Days. Creative contributors in marketing looked overburdened and frazzled. They talked about trying so hard to meet requirements, but no one controlled the flow, so they were always late. We started to advocate that the marketing department adopt Scrum. Nick became their ScrumMaster for a few months. Over time, we helped motivate adoption of combination of Scrum and Kanban in much of marketing.

Marketing folks seem a lot happier now. Engineering and marketing are working together more harmoniously, and, I claim, producing better work.

Example 2: The Company

To meet market needs, executives want everything as soon as possible. Traditional companies hire whole product silos with dedicated engineers, or acquire whole companies, to address newly defined products. At first, it works: you have all the people you need for the current market demand profile. But gradually those historic assignments become less aligned with company value. Over time, your product silo produces less and less value per engineer. Engineers working within siloed Scrum teams might boast of high productivity based on code output, but tied to last year's markets, their ongoing work earns less money over time. This exposes the fallacy of measuring work in "function points," "lines-of-code," and other code-based metrics: a team can produce lots of function points, but make little money.

Startup companies can often produce value more rapidly than large companies. Unhindered by long-standing work responsibilities, desperate for liquidity and operating intimately with customers, startup engineers have more motivation to work on projects that bring money in fastest.

We are a big company with several products, so we have to create a process substitute for startup-motivation. Faced with unranked needs from executives, and concerns about engineering productivity, we created a single Enterprise Scrum, with a three-month sprint cycle, to organize all engineering work. We asked decision makers to prioritize work by dollar value. We concentrated our best efforts on the top-ranked projects. This structure has transformed Citrix Online engineering to become a more responsive group, with growingly predictable deliveries.

As engineering becomes more sophisticated in tackling external projects, friction between engineering and others can reveal learning opportunities. Urgent projects require a just-in-time mechanism to handle sudden market changes without disruption. Productivity problems around user-experience, software architecture and operations indicate the need for multi-quarter portfolio planning. Difficulties prioritizing "apples against oranges" may reveal a deeper problem: Sometimes people don't know what their projects are really worth; they should be trying to figure it out.

These efforts are making the whole company more agile. Our President knows our engineering velocity, and we manage priorities to that velocity. VPs and Directors intercede frequently to remove impediments in engineering. Things are a lot better.

Promote Agility, Develop Strength

Agile methods can help most organizations get better stuff done in less time. A key philosophy drives Scrum: If we are going to work on something, it should matter to someone outside our team. If you're a skilled agilist, these concepts are second nature. You could help many people around you, by carrying the agile message upstream.

Ideal stakeholders and Scrum practitioners assess value thoughtfully, respect focused work, discuss uncertainties, accept responsibility for problems, and help reveal embarrassing root causes. Unfortunately, when you start promoting Scrum to the unconvinced, expect some disappointments, pushback and politics. But if you persistently and diplomatically promote agility, carefully wending the shoals and backwaters of other departments, you will strengthen your company. If you do it with integrity, you should emerge with a great reputation. And that will help your career.

Marketing Scrum Experimentation

Agile methods can organize marketing efforts: asset production, brand development, strategic brand alignment. However, it may not be a perfect fit: marketing tasks are often short-lived; "debt" is a different concept; prioritizing is perhaps more important, estimation less so. We are experimenting with these concepts at Citrix Online. We hope to gain insight by looking at other organizations.

Scrum challenges the status quo. Engineering Scrum radically rejects traditional assumptions: that engineers must meet external deadlines, that engineers bear no responsibility to disseminate in-progress work, that market conditions do not change. Scrum forces other departments—marketing, finance, product management and operations—to adapt to it, causing anxiety and sometimes rejection. But in the process of struggling with Scrum, some non-engineering departments are discovering Scrum has something to offer them in their own work.

As Citrix Online, the company where I work, has gained comfort with Scrum, our marketing department has started seeing its advantages. They wondered whether Scrum could be used in marketing.

We interviewed Richard Leavitt, Senior VP Marketing at Rally Software, which uses Scrum to organize its Marketing Department. We gained insight into how stakeholders and forces pull marketing work. We started to understand why marketing needs Scrum.

Here is a video that Nick Kim and I produced to convince them. It's a Pecha Kucha presentation, so it's only 6 minutes 40 seconds long (20 slides, 20 seconds per slide). Does it communicate the concepts well?

Agile Marketing Screencast

We are in the early stages of applying Scrum to Marketing efforts. I may post later about our experiences. If your marketing department uses Scrum or another agile organizing process, please contact me. I may include your experiences in a later post.

Postscript: The Agile Marketing screencast was recorded and edited in GoView, an experimental product developed using the Scrum process. I was the ScrumMaster for the GoView project.

Scrum Self-Similarity: Creating Organizational Fractals

Scrum exhibits fractal self-similarity, a property you can use to scale Scrum. Large organizations can deliver higher corporate productivity, revenues and agility in the face of rapidly changing markets. Everything scales, including the challenges.

We use a unique Enterprise Scrum process at Citrix Online (we are aware of no other organization using it) to manage a large, multiproduct engineering department. Anyone who cares, from the President on down, knows what engineers are working on, and has discussed and agreed on priorities. Everyone knows what projects are succeeding, and which are having trouble. When a problem could impede a release, it becomes rapidly visible and can trigger immediate executive action. We are producing more frequent releases, and our engineers are becoming more flexible and better aligned with company success. Enterprise Scrum works.

To create Enterprise Scrum, I analyzed normal Scrum as a fractal, focused on Scrum's fundamental properties, and then scaled them up.

Fractals in Scrum

Figure 1. Fractal Self-Similarity

Fractals are geometric shapes that exhibit self-similarity: some properties remain the same regardless of scale. For example, Figure 1c is a fractal. If you look at the gross structure in Figures 1a, 1b and 1c, the images are similar (squint if you are having trouble). Going from Figure 1a to 1c, the metrics, complexity and fine detail become greater, but the general outline remains the same.

Normal Scrum exhibits self-similarity, so we'll use it as an example. Most readers know how normal Scrum works—if you don't, please read Scrum: How to Manage Agile Projects and Teams.

Scrum has two feedback loops at different time scales. Every sprint, the whole team discusses what they did (Sprint Review), what they will do (Sprint Planning) and what blocks progress (Sprint Retrospective). Every day, the standup meeting requires each team member to discuss what they did, what they will do, and what blocks progress. The rough purpose of each feedback loop is the same: share information, get feedback, adjust goals, escalate impediments. Everything but the time-scale is roughly the same.

Scrum has two estimation scales. Whole teams groom and estimate effort for Product Backlog Items, using story points, a unit often roughly the size of estimated programmer days. The whole team is considered responsible for a Product Backlog Item. Individual team members usually generate and estimate effort for Tasks, using estimated programmer hours. That same person is responsible for completing the Task.

The purposes of estimation is the same: allow the team and individuals to prioritize activities on both relative value and relative effort, limit effort to respect the capacity of the team and individuals, allow the team and individuals to focus on a small number of activities, and gain greater understanding of the actual capacity of the team and individuals. They differ in scale.

Typical Scrum teams establish done criteria at different scales, though people sometimes don't realize it.

For example, a Task Done Criteria might include

1. unit tests were written and succeed,
2. a local build with all tests still succeeds,
3. the code was reviewed by another team member,and
4. the work was checked into the source control system. 

A Product Backlog Item Done Criteria might include

1. automated feature test was written and succeeded,
2. continuous build system on all platforms still succeeds,
3. team agrees it is good quality and marks it Done, and
4. deployment package was produced and checked into the repository.

Finally, a Sprint Done Criteria might include

1. upgrade and revert processes were performed on a clean integration system,
2. operations staff did a successful dry run install,
3. Product Owner reviewed the Sprint Backlog Items, marked them Accepted or rolled them forward, and closed the Sprint, and
4. product package was queued up in the Operations Backlog.

Let's review the Done Criteria parallelism. The 1s confirm the work was done. The 2s ensure other components in the ecosystem won't fail. The 3s cause an external party to validate the work. The 4s publish the work. They differ in scale. Now, when you look at the subcaptions of Figure 1, they might make more sense.

Scaling Scrum to the Enterprise

We've analyzed what exists, normal Scrum. Let's create something new, by scaling Scrum up to meet the needs of a whole engineering department. Enterprise Scrum looks like regular Scrum, with everything writ large:

• Story point size is estimated team months, estimates come from architects,
• Enterprise Backlog Items (EBIs) can be no smaller than 1 team-month of work,
• Sprint size is three months, i.e., a Quarter,
• Standup meeting frequency is every week, and
• Teams (not people) sign up for Enterprise Backlog Items.

If you were thinking Enterprise Scrum is just Scrum-of-Scrums, you'd be wrong—Enterprise Scrum is top-down, Scrum-of-Scrums is bottom-up. Citrix Online does both: in a big project, we do Scrum-of-Scrums; to prioritize and staff multiple big projects and products, we do Enterprise Scrum.

Sound simple? Once running, Enterprise Scrum is mechanically simple, but initiating is not. Your executive staff will take time to adopt Enterprise Scrum. Enterprise Scrum increases executive accountability, always a little scary. Enterprise Scrum exposes executive decisions so publicly that they can be second-guessed, and this threatens control. Without Enterprise Scrum, other departments could blame delays on the engineering department. I don't think you can adopt Enterprise Scrum unless your CEO and relevant Vice Presidents support you.

In other words, in this fractal the diplomatic challenges scale up with the rest. The same organizational impediments you faced with Scrum adoption will emerge in Enterprise Scrum, writ large. But the rewards scale too; if your company can do it, you win big in productivity.

I'll detail Enterprise Scrum in a forthcoming post on Scrumerati.