Over the last couple of years I have been increasingly exposed to Agile Methodologies for delivering BI projects. I’ve even once accidentally got myself trained as a Scrum Master. The topic itself isn’t exactly new (I blogged about it at the end of 2009) but adoption is starting to pick up as recognition that it can significantly improve the results of BI projects. What follows are some early stage ramblings on my experience so far as I try to pull together a more coherent story as I think its one that needs to be told and sold.
For those of you unfamiliar with Agile, it stems from the Agile Manifesto drawn up by some experienced developers who figured there was a better way than waterfall. The aim is to deliver faster, earlier and closer to user expectations. This is achieved by focusing on high value deliverables, delivering them piecemeal in descending order of value and engaging very closely with the business. All sounds pretty sensible so far.
There are a large variety of Agile Methodologies in play – Scrum, Extreme Programming, Lean Startup…. and none of them matter. Or at least, which one you pick doesn’t really matter. What matters more is whether your team does – or doesn’t – have the ability to work in an agile fashion. What I mean by this is that team members are self starting, self managing individuals with a low level of ego – for example they are happy to pitch in for the team and test some other developers work, rather than go “No, I code C++ and that’s all I will do.”. Good Agile teams are made up of members who understand they are in a team and that a team is not a fit of discrete lego bricks of skills but a fuzzy stew of ingredients where contributions blend into each other.
Now “the team” may just be your little pod of developers going out on a limb and trying soemthing new. Or, in a case I worked with last year, it may be a collosal financial services organisation where “the team” also encompassed very senior executives. Agility is a fragile thing and can just as easily be broken by a bottom rung developer not pulling their weight as it can be by a senior manager not understanding what the team is doing and trying to influence them to behave in non-agile ways.
The take up of Agile is often fraught with frustration and disappointment that a Nirvana of productivity doesn’t arise as soon as the Agile team forms. The reasons behind this are manifold – not least because it takes a while for the team to start working together with the new approach. Most methodologies acknowledge that initially productivity will be worse, rather than improved, until the team has stormed formed and normed its way to a higher productivity level. Many businesses struggle with the idea that they must let the patient get worse before they can get better.
Something else I have seen frequently is that the business engagement is poor – and not for want of trying. Before Agile is attempted, the business will often complain IT doesn’t involve them enough. Once it is attempted, they then grumble that involves them too much. This is incredibly frustrating to see as it underscores the business percepetion that IT is the servant and they are the master, rather than a valued partner trying to help them achieve their goals.
The Road to Success
I’m still gathering my thoughts on this but I suspect part of the road to success in doing BI in an Agile way is going to involve raising some red flags in the same way that you do for BI projects for the classic modes of failure (e.g. no business sponsor). Too often i’ve seen Agile fail because it is driven out of the development team wanting to do better but being unable to get the support needed to make it happen. The other killer is the “WAgile” trap where the team tries to operate as an Agile unit under a Waterfall project management approach.
I’m also keen to hear any readers war stories in the comments.