Agile BI, or why Big Bang warehouses should (sometimes) be shot

[James Beresford]

Recently I wrapped up*on a project that could have been a money pit with uncertain results, but managed to be steered into something that delivered results quickly and made the business very happy. To give a real high level overview, the client in question had a collection of mainframe delivered text files which fed Excel reports via Access via SQL2000. The project was being pushed by IT (usually a Bad Thing) primarily to solve a business problem (which meant*it turned*back into a Good Thing). The root business problem was mostly*being the Access databases kept falling over for various reasons so the analysts couldn’t do their work.

As the requirements were scoped out, it became apparent that the resultant*Data Warehouse was going to be a very large and complex thing that would take a long time to develop. We did a quick pilot to demonstrate that the ETL could deliver results in a structured and controlled manner into the Warehouse, which delivered about 5% of the files through from end to end. IT were pleased as it all went well. Their first thought was then to deliver the remaining 95% of data.

Do you need to put*all the Data in the Warehouse?

I asked the client to pause for thought at this point. In delivering the 5% of data we had proved that the DW process was feasible to IT – but had*given nothing to the business. We had solved an IT problem, not a business one. In order to get the business excited we needed to deliver someting to them. The IT side of the client saw the wisdom of this – if the business had something tangible and helpful it helped sell the project. Also, if they hated it, we could find out we were on the wrong track in a far cheaper manner than after delivering a monster DW that nobody wanted!

So what did we do next? We focused on satisfying just one business problem – in this case the data needs of a single department, and set about building the parts of the DW required to meet those needs. At the end of this we could throw them a shiny new*cube and see if it stuck. This way we would:

• Deliver initial results at minimum cost
• Establish if we were on the right track from an early stage
• Get business visibility of the DW faster

If all went well, we could focus on delivering the next 5%, then the next…* and probably find that there was probably a big chunk that wasn’t used anyway, thus giving more savings – or identifying areas for new insight. Much better that delivering the big bang DW that may actually be a damp squib!

So was that Agile BI then?

It was a form of it, yes! The idea of Agile*Development is to deliver results quickly, allowing the business to see what they are getting sooner and tune the results. Now, this isn’t all good – it can lead to massive scope creep, drift of purpose and even*failure to deliver anything at all. So there is a need to be very cautious and apply this methodology only where appropriate. Cases such as Cubes for analysis and Reports are things that can benefit from a rapid development cycle carried out in close conjunction with the business. A good example of where not to apply it is the ETL – it’s something that needs rigour and structure to be delivered effectively, and in practice is something business users neither understand or care about.

Sometimes*with BI it is the right approach in greenfield sites because of one*significant issue*- clients*can have a poor grasp on the capability of BI solutions and need leading into the wonderful world of BI. Often they can be skeptical about the promises made by consultants and by using an Agile approach on the first build you can get them into the BI mode of thinking rapidly. The client sees what they are getting and can get excited – whereas with standard waterfall delivery *they can wait months to see results after an long analysis / design phasse, and if the consultants have misjudged the business needs – kill the DW in its infancy.

But how can we deliver BI without the Warehouse?

I’m now on another project which has become lost in the design phase and the client is wondering what is going to come out of it all. We shifted approach to an Agile method where instead of focusing on getting the solution letter perfect in Design (Waterfall style) we are going to deliver a functioning SSAS Cube based on a sample of real data and build the design off of that. This way the client gets to see the light at the end of the tunnel and we know we are building the right solution even before the design is finalised.

The key*in this case*is to establish what the end deliverable is as quickly as possible and demonstrate that we, as consultants, have identified the right target in order to give the business confidence. The DW can follow to support that deliverable.

So when should a project go agile?

There’s no hard and fast rule, but I would suggest the following indicators may guide you towards an initial Agile build:

• Requirements are sparse / undefined – and the goalposts can move
• It is*the first BI project and BI capabilities are poorly understood
• Business users are highly available
• You need to give confidence you can deliver the right thing

In case you want to know more, here are a few additional articles on Agile BI that I have found:

• Agile Best Practices for Data Warehousing (DW)/Business Intelligence (BI) Projects from Scott W. Ambler @ agiledata.org
• Agile Development for*BI from BI Guru
• Architecting an Agile Data Warehouse from*Lee Arnett*@ information-management.com
• Agile Data Warehousing: Delivering World-Class Business Intelligence Systems by Ralph Hughes – a comprehensive book on the subject

Get More from the original blog...