Agility in BI

Over the last couple of years I have been increasingly exposed to Agile Methodologies for delivering BI projects. I’ve even once accidentally got myself trained as a Scrum Master. The topic itself isn’t exactly new (I blogged about it at the end of 2009) but adoption is starting to pick up as recognition that it can significantly improve the results of BI projects. What follows are some early stage ramblings on my experience so far as I try to pull together a more coherent story as I think its one that needs to be told and sold.

Agile Principles

For those of you unfamiliar with Agile, it stems from the Agile Manifesto drawn up by some experienced developers who figured there was a better way than waterfall. The aim is to deliver faster, earlier and closer to user expectations. This is achieved by focusing on high value deliverables, delivering them piecemeal in descending order of value and engaging very closely with the business. All sounds pretty sensible so far.

Agile People

There are a large variety of Agile Methodologies in play – Scrum, Extreme Programming, Lean Startup….   and none of them matter. Or at least, which one you pick doesn’t really matter. What matters more is whether your team does – or doesn’t – have the ability to work in an agile fashion. What I mean by this is that team members are self starting, self managing individuals with a low level of ego – for example they are happy to pitch in for the team and test some other developers work, rather than go “No, I code C++ and that’s all I will do.”. Good Agile teams are made up of members who understand they are in a team and that a team is not a fit of discrete lego bricks of skills but a fuzzy stew of ingredients where contributions blend into each other.

Now “the team” may just be your little pod of developers going out on a limb and trying soemthing new. Or, in a case I worked with last year, it may be a collosal financial services organisation where “the team” also encompassed very senior executives. Agility is a fragile thing and can just as easily be broken by a bottom rung developer not pulling their weight as it can be by a senior manager not understanding what the team is doing and trying to influence them to behave in non-agile ways.

Adoption Issues

The take up of Agile is often fraught with frustration and disappointment that a Nirvana of productivity doesn’t arise as soon as the Agile team forms. The reasons behind this are manifold – not least because it takes a while for the team to start working together with the new approach. Most methodologies acknowledge that initially productivity will be worse, rather than improved, until the team has stormed formed and normed its way to a higher productivity level. Many businesses struggle with the idea that they must let the patient get worse before they can get better.

Something else I have seen frequently is that the business engagement is poor – and not for want of trying. Before Agile is attempted, the business will often complain IT doesn’t involve them enough. Once it is attempted, they then grumble that involves them too much. This is incredibly frustrating to see as it underscores the business percepetion that IT is the servant and they are the master, rather than a valued partner trying to help them achieve their goals.

The Road to Success

I’m still gathering my thoughts on this but I suspect part of the road to success in doing BI in an Agile way is going to involve raising some red flags in the same way that you do for BI projects for the classic modes of failure (e.g. no business sponsor). Too often i’ve seen Agile fail because it is driven out of the development team wanting to do better but being unable to get the support needed to make it happen. The other killer is the “WAgile” trap where the team tries to operate as an Agile unit under a Waterfall project management approach.

I’m also keen to hear any readers war stories in the comments.

2 thoughts on “Agility in BI

  1. With the new fiscal year, our government client went from “Agile in name only” to pretty hard-core Agile. It’s been an interesting transition, with a few big wins and a few big bumps in the road.

    The two biggest bumps in the road were contracts and communication. All of our task orders had been written up in the previous fiscal year, before we went Agile. As such, they had requirements that weren’t framed as user stories and schedules that were only vaguely organized as sprints. We had to strip out most of that work, and re-frame everything. The communication hiccup was largely from the government PM to their user base. The user base had ZERO idea what Agile meant. They showed up at the first Sprint Planning meeting completely confused and unprepared. We’ve started getting to the point now where we can properly frame things as user stories, but the users are still very uncertain as to what their role is.

    The two biggest wins come in tying off the projects, and the very basic Agile advantage of interactive development. Prior to this, we had a huge recurring issue of never being able to properly deliver anything. Projects were never really done, they just kept getting revised until we ran out of money. The Agile concepts of “definition of done” and pushing requests for refinements into future sprints has really helped to solve that. Our project is considered done when we do a demo and turn it over for UAT. It is then up to the user group to test it and decide if it is ready to go into production, or needs another sprint. If a defect is found (something that means that the report fails the acceptance criteria), we fix it for free. If an issue comes up that isn’t a defect (generally either a refined request or a realization that the acceptance criteria weren’t framed correctly), that goes into the backlog for a future sprint.

    The interactive development has been really good. We concentrate on delivering small chunks as rapidly as possible, and getting immediate feedback. For most reports, this is easily done. We can then immediately fold that feedback right back into our development. The smaller chunks make the comments much more specific, and much easier to track. Having the quick feedback loop is both better for us as developers (we waste a LOT less time on work that isn’t actually wanted) and for the users (who get to see constant progress instead of a big black hole of promises).

    The one issue we’ve run into is that SAP BO is not well suited to team programming. You can’t have two developers both working on separate reports in the same document, and then merge the efforts later on. This is even more of an issue when you get to SAP Dashboards. We are going to be starting up a couple Dashboards projects here in the next couple of months, and we’re a bit nervous about how it’s going to go.

  2. Lugh, thanks for the detailed and insightful comment. I particularly like the feedback on Agile approaches providing a sense of progress in what can be effectively endless programs.

    In my current client we also suffer from the team programming issue on our cube tech – Microsoft Tabular models are single files regardless of how many dimensions and facts are in there. Unfortunately the only way we can manage this is through careful coordination, communication and plenty of backups!

    Cheers, James

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>