Managing large stories on agile projects

Obie Fernandez: "Where have you experienced limitations of a story-driven process and how did you deal with it?"

Insightful post from Obie on two fronts. First the common functional themes in Web 2.0 apps:

"Hashrocket does a lot of Web 2.0 apps, and most of them have a similar assortment of basic functionality and social networking features: user accounts, event notification, photo uploading, commenting, etc. Naturally, there is a desire to take our sum knowledge of building these types of features and codify it in the form of a base application and suite of plugins."

Very true - typically these Web 2.0 features revolve around socialising and sharing rather than content (someday I'll post on why "websites" are an anachronism). Second was the issue of breaking down big ticket items into measurable stories, which is what I wanted to highlight. Here's Obie's card:

"Story: Adding Photos

In order to provide photo collections for my client,
As a developer,
I want to be able to add photos to a domain model


    * Installation instructions
    * Declarative macro makes model act like a photo collection
    * Generator-created: Photos controller that includes photo resource module
    * Generator-created: Views (new, form partial, javascript, FancyUploads)
    * Generator-created: migration
    * API documentation"

and here's the issue:

"One of my guys called it a "12-pointer" to denote how much it needed to be broken down in order to fit in with our typical process (limited to 1, 2 and 4 point stories).

The problem is that as the stakeholder, I don't want this story broken down! This is the level at which I want to do acceptance, for at least a couple of reasons: 1) I already went through a long acceptance process for many of these stories the first time they were implemented, in the language of the end-user. 2) In order to properly define acceptance criteria for this story, I would have to know exactly how it's going to be implemented, and that information is not available until they actually sit down to figure out how to do the extraction into gems and plugins. In fact, one of the reasons that the storycarding session was feeling so painful was exactly that we were spending so much time arguing about implementation -- out of place in a typical storycarding process where you leave implementation details for later."

I've seen this granularity problem on every project, product or program I've worked on. Often in non-agile methods it comes it up in the form of traceability requirements on top of the actual requirements. As someone whose has the word architect in their job description I come across a similar problem with systems, operations and non-functional stories that less to do with the features and more to do with the features being used by a lot of people (I know, you can argue they're not really stories in the pure XP sense, but good luck explaining *after* the iteration why the service failed ;). I find that business owners often want to think about the software at this level and having them into decompose the work into stories that are meaningful to developers, but which need to reconsitituted, that isn't always the right thing and we need different views of the work depending on context. Interestingly (to me) story cards are not requried by either XP or Agile, but we tend to fetishize index cards (iirc Extreme Programming 2nd ed mentions index cards just once, and as an option).

Mike Cohn has a nice decomposition for this, a notion of epics, themes and stories from Agile Estimating and Planning:

"Although in general, we want to estimate user stories whose sizes are within one
order of magnitude, this cannot always be the case. If we are to estimate every-
thing within one order of magnitude, it would mean writing all stories at a fairly
fine-grained level. For features that we’re not sure we want (a preliminary cost
estimate is desired before too much investment is put into them) or for features
that may not happen in the near future, it is often desirable to write one much
larger user story. A large user story is sometimes called an epic.
    Additionally, a set of related user stories may be combined (usually by a pa-
per clip if working with note cards) and treated as a single entity for either
estimating or release planning. Such a set of user stories is referred to as a
theme. An epic, by its very size alone, is often a theme on its own.
    By aggregating some stories into themes and writing some stories as epics, a
team is able to reduce the effort they’ll spend on estimating. However, it’s impor-
tant that they realize that estimates of themes and epics will be more uncertain
than estimates of the more specific, smaller user stories.
    User stories that will be worked on in the near future (the next few itera-
tions) need to be small enough that they can be completed in a single iteration"

In that light I would call Obie's card a "theme". Alternatively your shop might call it a feature or capability. It rightly should be broken into measurable stories, but equally, for the owners it's important the theme is not lost or splattered across a set of tickets, cards or ATs so it can't be easily tracked. It should remain a whole thing to the owners. So the idea is simple - treat the software as an Epic (part of the infinite game) and Themes are large and meaningful things the software should do, that can be broken down into Stories, chunks of work that the team can reason about. 

That still leaves two issues; the tracking mechanism and acceptance.

On the first, the tracking mechanism, I like how some open source projects manage these big ticket items simply as tickets. Here's are two examples I know of from Hadoop:

HADOOP-2510: Map-Reduce 2.0, re-work Hadoop Map-Reduce to make it suitable for a large, static cluster.

HADOOP-3719: contribution of Chukwa by Yahoo!, a data collection and analysis framework.

there's a lot of separately scheduled work implied by those, but there is one place around which the work can be dicussed, scheduled and closed (Hadoop use Jira which has features to relate tickets, but they're incidental here - being able to have links to tickets is way more important for the document of record). In no way are they the kind of things you put on an index card and blutack to the wall.

On the second, acceptance, this can be tricky and probably has more business impact than how to notate large stories. A common problem in Agile projects is having a theme scale card open across iterations. This has a number of negative knock on effects - the worst probably is the reintroduction of the 80% done antipattern that agile all but eliminates. It can also impact upstream acceptance scheduling organisation such as systems or beta testing, even marketing plans. It can allow for subtle scope creep where "splitting" of the card is in fact new work. 

How to represent work like this? One simple approach is to enhance your story or kanban board from a set of vertical swimlanes into a matrix where Themes are each given a horizontal slice on the board and each Theme has its own set of Stories. The entire board represents the Epic. This provides visibility on how software is progressing overall at a level above story cards, which is suitable for people who will care about different themes at different times, which features are bottlenecking and where. It will work well for integrations and visualising dependency issues day to day  - there's nothing to say a theme can be written from the point of view of another system or team - "As the Billing System, I would like ...". It also fits in with other non-team toolchains such as rollups required by management or program teams, mostly because it scales to a portfolio view by creating a master portfolio board where each aforementioned Epic is in turn given a horizontal slice. Another approach described by Mike Cohn is a treemap, but this requires more sophisticated tooling,



    Have you looked at User Story Mapping by Jeff Patton? He's put out this excellent presentation:

    It's a bit long, but the general idea is a bit like the epic/theme/story guideline, except that it is more precisely defined as goal/activity/task.

    This also relates to feature thinning. You don't have to split a feature in two, but you can thin it down to a bare minimum. Again, Patton has put a lot of thought into it:

    Combining these gives a compelling way of handling large stories in a backlog.
    Once again, Patton has insight to share with us:

    Hi Bill, I'm sure you have tried User Stories Applied:

    I have felt the problem you describe. I actually think its a feature most of the time given what the human mind is capable of.

    The chances of anyone getting the original story set precisely right is pretty slim.

    I think that the "5 levels of planning" described in User Stories Applied works pretty well:

    1. Vision

    2. Roadmap (we update every couple months and re-estimate it)

    3. Release planning (every couple months)

    4. Iteration planning (every 2 weeks for us)

    5. Daily planning

    At least on my most recent experience on the TriSano project, this + a Kanban board / continuous improvement + incredible people works great.

    As epics need to be split, you split them. As the users see progress each iteration through demos / deployments, new stories are written to fill in the details of themes or ones off the backlog are pulled or what is still missing. It takes disciplined users committed to the process.
    Users become committed by seeing results. It also helps when they have seen failure with other methods.

    Here's a sample of what we've evolved to (the team is about 1 year old and has reached the 'high performing team' stage):

    Our most recent release plan (pretty simple):

    We just put that together after a release planning session in person last week. Its enough for us to know the size of the release, to steer it, and to get to work asap :)

    Also, here's our simple roadmap.

    We re-estimated it last week to fit into our 2009 story point budget that is based on our historical velocity:

    Anyway, sorry for blathering on, in short, I don't think that problem is easily solvable and think its better to just have everyone accept it and instead deliver faster so no one really cares because they get to try again and (perhaps) perfect it or (perhaps) move on to something of more business value.