How is what we are building helping you make better decisions?
This is a followup to a good article by Timo Elliott on making information understandable. It underlines a good point – good information is useless unless it is presented in a way the users can understand. I always try and build any reports with the philosophy; “How do I make it so the most number illiterate recipient can grasp what this report is showing?” – working for a bunch of Lawyers in one job really honed this mode of thinking!
Why Bus Timetables are a great example of BI
For a comparable transport analogy (and general moan), Sydney has terrible bus information. At most stops is a sheet of A4 (if you’re lucky) with bus numbers and times in order of time. No maps. No explanation of what bus number goes where. Unless you know in advance which bus you need, this data is useless. It’s not a surprise I never bother getting the bus – I can’t hop on one because I have no idea when a useful one is coming, and if one does come, whether its route is good for me. Compared to London with its simple route maps and timetables ordered by route and time, I hopped on the bus regularly, because if I passed a bus stop in a strange part of town I could rapidly figure out if it was a good option.
What question makes the difference between a Report and Information?
A common failure point for BI is that too much emphasis has been put on the tools, and getting data in shape – so you have a fantasic warehouse and toolset to read it – but users who aren’t “educated” enough to use it, because there seems to be the mindset among developers that BI systems are just that – systems. Training is usually functional – “how to access Report B, how to rerun the Data” – not practical – “How does Report B give me the ability to make better decisions”? There is a gap in expectations – the developers assume management know what they are asking for. Managers assume what they get will get the answers from the reports. No-one in the equation is brave enough to pipe up and ask: “Do you know what you really want? Because I don’t think this will help.”
A good consultant should continually be asking this question of their clients – “How is what we are building helping you make better decisions?“
Why you should check your consultants
I’m currently working on a near-real time reporting system for a big client, and one of the components of the work is to fix the ETL system put in place by the consultancy who were in previously. So what, you might say, they’ve been burned by a tin pot little outfit who overreached. Well, er, no. The consultancy who built it are a huge global player, and are people who should have the brightest and best working for them. They seem to have made 2 critical mistakes:
- Applied a generic methodology without customising it to the tool or situation
- Outsourced development offshore
Why can a general approach fail?
It’s true, ETL processes have general principles that should be followed in most processes, but each ETL tool has its quirks that need to be allowed for, and performance features that should be embraced. Similarly each project will have its own unique requirements that will mean the generic approach won’t be a 100% fit for the project you are working on.
For example, in the case I’m dealing with now, they have a control framework which is designed to do all sorts of logging and control – except about 50% of it simply doesn’t work because of the way the data of the project has to be handled. Similarly they have control tables whose contents have been stuffed with chunks of queries simply to make the existing framework function, rather than rework the framework to suit the need.
A general approach will fail when it is applied too rigidly by people who don’t understand the reasons for the approach in the first place. This is why lawyers have the concept of the spirit and letter of the law. With any methodology you need flexibility in how you handle each situation – the reason why there isn’t a universal ETL method used by everyone is because such a system cannot exist given the variety of demands and constraints on such systems.
Offshoring saves us money so is good for the business, surely?
Offshoring works in an IT project in a very narrow range of circumstances, and BI projects very rarely meet those circumstances. What happens when you offshore is you send your specification to a third party who cannot – because of time differences, cultural differences, language issues – communicate with you constantly and get an understanding of what you are asking for. As a result your specifications will be met exactly, with no allowance for what you wanted, because any experienced IT developer will admit the specifications often overlap – but rarely match exactly – with what the business needs. This will be fatal for a BI project because as seasoned BI professionals will warn – BI projects are first and foremost Business projects, not IT projects.
Combined with the risks in having insufficiently skilled developers sticking rigidly to a general approach that doesn’t really make any sense, and you have what we are dealing with – a dysfunctional system that may simply need to be thrown away.
So how do I protect myself against bad consultants?
This is probably a series of articles in itself, but I would give the following quick suggestions around development:
- Have development work done on site as much as possible. If the words outsource or offshore are mentioned, run away.
- Ensure that lead developers are experienced – certified if possible – in the technology you are using. Being a SSIS ace doesn’t mean you will be bale to do great work in Data Integrator
- If possible get a third party to review the work at an early stage
If anyone else has had similar experiences and ideas on how to mitigate the risks of engaging poor consultants, comment away!
Testing Business Rules
Everybody hates testing. I hate testing. You hate testing. Bob down the corridor, well – actually he quite likes it, but then he files his pencils in order of length in a special drawer, so he’s probably not right in the head.
However, get it right and you save yourself alot of grief. This is in reference to one client I have worked with who didn’t have a test plan as much as a “Well, if the numbers match the old system when we’ve finished, well, it’ll be right and we’ll sign off”. Not in itself an invalid test, but a lousy way of testing that the details work as expected. As a result of this approach the testing phase of this project dragged on for months, people on both sides were blaming each other for the slow progress and the relationship has gone to pot – driven heavily by inadequate test plans.
So what was so bad? Well, whenever anything didn’t match – on a test case such as “numbers of widgets moved from location A to location B in March” – the only way to find out why it didn’t match was to dig through all systems, find out why they didn’t work – applying all the business rules along the way – and then find out it was down to “ah, we didn’t apply business rule A in the BI layer” – or the client had forgot they didn’t apply business rule B in the old system, but they did in the new. The amount of effort put in digging into this a) frustrated the consultants working on the project and b) made the clients lose patience as every test case that failed to ages to resolve.
Fogged into all this was that often we’d resolve one problem on the test case, only to reveal another. This further dented the client’s confidence (because obviously when it didn’t work it was always the consultancy’s fault – some clients have an odd ability to completely gloss over their own contribution to a project’s problems).
What the tests should have been doing was testing the specific rules that were being applied to the data were being done so – on a rule by rule basis that would allow us to isolate it’s function. So test cases should have been “If Widget is of type A, but not in class C then they should be moved to location B”. This way we could have picked up whether rules were being applied correctly far earlier in the process, and with much less effort.
As an addendum to this, reconciling against old reports should only be done once it’s understood how the old reports work. We had significant problems because the old reports we were trying to align against weren’t applying the same business rules as the new system.
So, what is BI Monkey’s concluding thought? Test the details. Test them in isolation. Make sure that the details work long before you try and test the whole system, and you’ll have a much better chance of success.