Scrumerati

Pushing Agility Upstream

Most agile teams live in a waterfall ecosystem. When powerful stakeholders—like business partners, customers, other departments or executives—demand future commitments, while interrupting contributors with unprioritized demands, smart Scrum teams raise a protective shield. They make the Product Owner manage stakeholder priorities, and make the ScrumMaster defend them against interference. If anyone provides a date to a stakeholder, it's going to be the Product Owner: the Single Wringable Neck.

But instead of just defending ourselves against an external onslaught of unprioritized need, we can go on the offense. We can evangelize agility upstream. Impossible requirements point out that stakeholders could themselves benefit from agility. If we teach them agility, they and we will both go faster. Together we can develop a strong, sustainable and profitable business ecosystem.

My buds and I have pushed agile upstream twice recently, and it's more fun than you might think.

Example 1: Marketing

Traditional marketing departments demand that staff respond to immediate needs: the "hair on fire" approach to planning. In the past, engineers working closely with marketing have accepted these interrupts, and some of our own past work reflects the ragged architecture and appearance you might expect. Expensive rework wasted engineering time. Bugs caused significant problems.

In 2007, our engineering department adopted Scrum. After that, marketers ran headlong into ScrumMasters defending engineering sprint boundaries. Marketing's first reaction was to find ways around Scrum restrictions, demanding their own dedicated teams, or time-boxing vaguely defined marketing related work that engineers would have to fulfill.

Engineering colleagues at first tried to satisfy these needs, but then realized marketing demands were distracting them from important work. I assumed we just needed to explain ourselves better. Scrum meant we wouldn't be able to satisfy marketing needs the same way. Marketing would have to prioritize needs earlier, based on our work estimates. They would have to allow us to say "No" to low priority work. Nick Kim (another evangelist ScrumMaster) and I started a conversation with key managers.

But as we talked, we realized that marketing suffered the same problems our we had experienced in the Bad Old Waterfall Days. Creative contributors in marketing looked overburdened and frazzled. They talked about trying so hard to meet requirements, but no one controlled the flow, so they were always late. We started to advocate that the marketing department adopt Scrum. Nick became their ScrumMaster for a few months. Over time, we helped motivate adoption of combination of Scrum and Kanban in much of marketing.

Marketing folks seem a lot happier now. Engineering and marketing are working together more harmoniously now, and, I claim, producing better work.

Example 2: The Company

To meet market needs, executives want everything as soon as possible. Traditional companies hire whole product silos with dedicated engineers, or acquire whole companies, to address newly defined products. At first, it works: you have all the people you need for the current market demand profile. But gradually those historic assignments become less aligned with company value. Over time, your product silo produces less and less value per engineer. Engineers working within siloed Scrum teams might boast of high productivity, but because of ossified product responsibilities, the value of their work becomes lower over time. This exposes the fallacy of measuring work in "function points," "lines-of-code," and other engineering metrics: a team can produce lots of function points, but make little money.

Startup companies can often produce value more rapidly than large companies. Unhindered by long-standing work responsibilities, desperate for liquidity and operating intimately with customers, startup engineers have more motivation to work on projects that bring money in fastest.

We are a big company with several products, so we have to create a process substitute for startup-motivation. Faced with unranked needs from executives, and concerns about engineering productivity, we created a single Enterprise Scrum, with a three-month sprint cycle, to organize all engineering work. We asked decision makers to prioritize work by dollar value. We concentrated our best efforts on the top-ranked projects. This structure has transformed Citrix Online engineering to become a more responsive group, with growingly predictable deliveries.

As engineering becomes more sophisticated in tackling external projects, friction between engineering and others can reveal learning opportunities. Urgent projects require a just-in-time mechanism to handle sudden market changes without disruption. Productivity problems around user-experience, software architecture and operations indicate the need for multi-quarter portfolio planning. Difficulties prioritizing "apples against oranges" may reveal a deeper problem: Sometimes people don't know what their projects are really worth; they should be trying to figure it out.

These efforts are making the whole company more agile. Our President knows our engineering velocity, and we manage priorities to that velocity. VPs and Directors intercede frequently to remove impediments in engineering. Things are a lot better.

Promote Agility, Develop Strength

Agile methods can help most organizations get better stuff done in less time. A key philosophy drives Scrum: If we are going to work on something, it should matter to someone outside our team. If you're a skilled agilist, these concepts are second nature. You could help many people around you, by carrying the agile message upstream.

Ideal stakeholders and Scrum practitioners assess value thoughtfully, respect focused work, discuss uncertainties, accept responsibility for problems, and help reveal embarrassing root causes. Unfortunately, when you start promoting Scrum to the unconvinced, expect some disappointments, pushback and politics. But if you persistently and diplomatically promote agility, carefully wending the shoals and backwaters of other departments, you will strengthen your company. If you do it with integrity, you should emerge with a great reputation. And that will help your career.

Personal 2009 Retro/2010 Planning

My yearly retrospective mind map summarizes what went well and badly in 2009, and what I plan to do in 2010. Click the image to see the full view.

Dan Greening 2009 Retrospective Mind Map

I plan to iterate in 2010, with monthly or shorter sprints, progressing each month to make usable progress on as many 2010 goals as possible.

One thing that deserves explanation: "software project assessor". This is not a commonly used term. An example is Elliot Fishman, who analyzes startup companies to determine their intrinsic bottom-up value. Venture capitalists pay him for his analysis. Being a sophisticated software project assessor means you can look at a software project or startup company and determine its likely value in the future. There are obvious implications to this.

I hope those who know me will ask me how it's going; it will help motivate me.

Have a great 2010!

Marketing Scrum Experimentation

Agile methods can organize marketing efforts: asset production, brand development, strategic brand alignment. However, it may not be a perfect fit: marketing tasks are often short-lived; "debt" is a different concept; prioritizing is perhaps more important, estimation less so. We are experimenting with these concepts at Citrix Online. We hope to gain insight by looking at other organizations.

Scrum challenges the status quo. Engineering Scrum radically rejects traditional assumptions: that engineers must meet external deadlines, that engineers bear no responsibility to disseminate in-progress work, that market conditions do not change. Scrum forces other departments—marketing, finance, product management and operations—to adapt to it, causing anxiety and sometimes rejection. But in the process of struggling with Scrum, some non-engineering departments are discovering Scrum has something to offer them in their own work.

As Citrix Online, the company where I work, has gained comfort with Scrum, our marketing department has started seeing its advantages. They wondered whether Scrum could be used in marketing.

We interviewed Richard Leavitt, Senior VP Marketing at Rally Software, which uses Scrum to organize its Marketing Department. We gained insight into how stakeholders and forces pull marketing work. We started to understand why marketing needs Scrum.

Here is a video that Nick Kim and I produced to convince them. It's a Pecha Kucha presentation, so it's only 6 minutes 40 seconds long (20 slides, 20 seconds per slide). Does it communicate the concepts well?

Agile Marketing Screencast

We are in the early stages of applying Scrum to Marketing efforts. I may post later about our experiences. If your marketing department uses Scrum or another agile organizing process, please contact me. I may include your experiences in a later post.

Postscript: The Agile Marketing screencast was recorded and edited in GoView, an experimental product developed using the Scrum process. I was the ScrumMaster for the GoView project.

Scrum Self-Similarity: Creating Organizational Fractals

Scrum exhibits fractal self-similarity, a property you can use to scale Scrum. Large organizations can deliver higher corporate productivity, revenues and agility in the face of rapidly changing markets. Everything scales, including the challenges.

We use a unique Enterprise Scrum process at Citrix Online (we are aware of no other organization using it) to manage a large, multiproduct engineering department. Anyone who cares, from the President on down, knows what engineers are working on, and has discussed and agreed on priorities. Everyone knows what projects are succeeding, and which are having trouble. When a problem could impede a release, it becomes rapidly visible and can trigger immediate executive action. We are producing more frequent releases, and our engineers are becoming more flexible and better aligned with company success. Enterprise Scrum works.

To create Enterprise Scrum, I analyzed normal Scrum as a fractal, focused on Scrum's fundamental properties, and then scaled them up.

Fractals in Scrum

Figure 1. Fractal Self-Similarity

Fractals are geometric shapes that exhibit self-similarity: some properties remain the same regardless of scale. For example, Figure 1c is a fractal. If you look at the gross structure in Figures 1a, 1b and 1c, the images are similar (squint if you are having trouble). Going from Figure 1a to 1c, the metrics, complexity and fine detail become greater, but the general outline remains the same.

Normal Scrum exhibits self-similarity, so we'll use it as an example. Most readers know how normal Scrum works—if you don't, please read Scrum: How to Manage Agile Projects and Teams.

Scrum has two feedback loops at different time scales. Every sprint, the whole team discusses what they did (Sprint Review), what they will do (Sprint Planning) and what blocks progress (Sprint Retrospective). Every day, the standup meeting requires each team member to discuss what they did, what they will do, and what blocks progress. The rough purpose of each feedback loop is the same: share information, get feedback, adjust goals, escalate impediments. Everything but the time-scale is roughly the same.

Scrum has two estimation scales. Whole teams groom and estimate effort for Product Backlog Items, using story points, a unit often roughly the size of estimated programmer days. The whole team is considered responsible for a Product Backlog Item. Individual team members usually generate and estimate effort for Tasks, using estimated programmer hours. That same person is responsible for completing the Task.

The purposes of estimation is the same: allow the team and individuals to prioritize activities on both relative value and relative effort, limit effort to respect the capacity of the team and individuals, allow the team and individuals to focus on a small number of activities, and gain greater understanding of the actual capacity of the team and individuals. They differ in scale.

Typical Scrum teams establish done criteria at different scales, though people sometimes don't realize it.

For example, a Task Done Criteria might include

1. unit tests were written and succeed,
2. a local build with all tests still succeeds,
3. the code was reviewed by another team member,and
4. the work was checked into the source control system. 

A Product Backlog Item Done Criteria might include

1. automated feature test was written and succeeded,
2. continuous build system on all platforms still succeeds,
3. team agrees it is good quality and marks it Done, and
4. deployment package was produced and checked into the repository.

Finally, a Sprint Done Criteria might include

1. upgrade and revert processes were performed on a clean integration system,
2. operations staff did a successful dry run install,
3. Product Owner reviewed the Sprint Backlog Items, marked them Accepted or rolled them forward, and closed the Sprint, and
4. product package was queued up in the Operations Backlog.

Let's review the Done Criteria parallelism. The 1s confirm the work was done. The 2s ensure other components in the ecosystem won't fail. The 3s cause an external party to validate the work. The 4s publish the work. They differ in scale. Now, when you look at the subcaptions of Figure 1, they might make more sense.

Scaling Scrum to the Enterprise

We've analyzed what exists, normal Scrum. Let's create something new, by scaling Scrum up to meet the needs of a whole engineering department. Enterprise Scrum looks like regular Scrum, with everything writ large:

• Story point size is estimated team months, estimates come from architects,
• Enterprise Backlog Items (EBIs) can be no smaller than 1 team-month of work,
• Sprint size is three months, i.e., a Quarter,
• Standup meeting frequency is every week, and
• Teams (not people) sign up for Enterprise Backlog Items.

If you were thinking Enterprise Scrum is just Scrum-of-Scrums, you'd be wrong—Enterprise Scrum is top-down, Scrum-of-Scrums is bottom-up. Citrix Online does both: in a big project, we do Scrum-of-Scrums; to prioritize and staff multiple big projects and products, we do Enterprise Scrum.

Sound simple? Once running, Enterprise Scrum is mechanically simple, but initiating is not. Your executive staff will take time to adopt Enterprise Scrum. Enterprise Scrum increases executive accountability, always a little scary. Enterprise Scrum exposes executive decisions so publicly that they can be second-guessed, and this threatens control. Without Enterprise Scrum, other departments could blame delays on the engineering department. I don't think you can adopt Enterprise Scrum unless your CEO and relevant Vice Presidents support you.

In other words, in this fractal the diplomatic challenges scale up with the rest. The same organizational impediments you faced with Scrum adoption will emerge in Enterprise Scrum, writ large. But the rewards scale too; if your company can do it, you win big in productivity.

I'll detail Enterprise Scrum in a forthcoming post on Scrumerati.

Andy Warhol explains Incremental and Iterative Development

In any project, you should complete your your work in stages. This lets you show customers (those who pay) and users (who might not pay) your work in progress, to ensure you satisfy their initial expectations and adapt to new ones.

You will likely stage your project incrementally or iteratively. Most people don't know the difference, but if they stage a project incrementally, it is easy to waste a lot of time. Iterative development is usually better.

I wrote this, hoping to help.

Incremental development completes system components in a sequence, finishing each component completely before moving on to the next. Incremental development makes it hard to satisfy customer priorities early, because many components are typically required for a usable system.  You have to finish most of the development before a customer can try something. The advantage is that you never redo anything; it seems great until you try it.

For example,  suppose a patron commissions you to paint Ms. Lisa.  Since you know exactly what your patron wants, you can use incremental development.  You will paint the top 5cm of your work, Mona Lisa, in the first increment, which takes a week, the next 5cm in the second increment, etc. until you finally finish the Mona Lisa. After 10 increments, you might be able to try it out on your patron.  “Is this what you want?”  You really hope that’s what your patron wants, because it took 10 weeks.  But it turns out your patron is Andy Warhol, and Mona Lisa is not what Andy wants.

Figure 1. Ms. Lisa, Increment 1

Iterative development first completes a barely viable portion of each essential component, enough that the components can connect with each other, and tests the system with customers.  In the first iteration, the skeletal structure of the system emerges. (Some people call this first try at a skeletal structure a "tracer bullet", because it shows the path all the way through the system.) Each subsequent iteration adds greater detail to every essential component. Iterative development is better if (you can admit) you aren’t sure what’s important, because the whole system emerges and can be shown to customers for their feedback.

For example, in the case of Ms. Lisa, drawing a pencil sketch on paper with a drawn frame in Iteration 1 might be a good approach.  You show it to your patron and say “Is this what you want?”  Your patron, Andy Warhol, says, “Well that mountain there, it messes up the pastoral nature of what I’m looking for.  But could you add some vibrant colors?” 

Figure 2. Ms. Lisa, Iteration 1

In Iteration 2 you erase the mountain, put some nice rolling grass, sketch a wan smile, and put in some bold colors with pastels. You bring it to your patron, with a frame sample. Andy says, “I love this sketch! Here's money for a copy, I’ll hang it right away. I’ll call to let you know what I like and don’t about this sketch, as I enjoy it in my home.” He ponders it for a moment, "I'd like to hang the frame sample next to it, and do you mind if I write 'R. Mutt 1917' on it?" You don't mind at all, of course.

So with iterative development, the patron gets a feel for your progress, obtains some benefit before the whole project is completed, and provides real-world feedback to you. 

Nine iterations later, you finish the Marilyn Lisa, in fluorescent colors, a bizarre cross between Marilyn Manson and Mona Lisa in a pastoral setting.  It is exactly what this patron wanted.  Andy gives you $300,000, a tidy profit, and provides your first 15 minutes of fame.  Your work hangs in Andy's foyer.  Andy introduces you to Shepard Fairey, Shepard teaches you how to make serigraphs, and you make millions.

Good iterative development requires feedback from your patron, namely the user, customer or Product Owner.  Directing development team priorities without early customer feedback is a recipe for wasted effort. Iterative development is one of the best ways to gain thoughtful customer feedback early, and create something they will buy.

Credits

1. Jeff Patton provided the idea about the incremental Mona Lisa.
2. Leonardo di Vinci painted the Mona Lisa.
3. Dan Greening, the famous artist, sketched the first iteration of the Marilyn Lisa.