Is a data warehouse necessary for good data management?

[Robert Hillard]

Our profession is not a large one globally.  In fact, a couple of years ago I did a back-of-the-envelope calculation that there are about 50,000 to 100,000 people around the world who would regard themselves as primarily working in data or information management.  We worked this out by estimating that to specialise you would have to be working in one of the 5,000 largest organisations (private and government) and a reasonably constrained number of professional services organisations.

Regardless of whether the numbers are right, the numbers are not large given the impact that the profession has.  Most of us are connected in some way through the conferences we attend or the communities that we participate in (such as this one).  Looking at who are in the roles, anecdotally there appears to be a disproportionately large number of people with a data warehousing background.  I include myself in this category.

Over the past five or even ten years the field has broadened out to apply standardised approaches to data management.  The scope of this includes data quality, master data management, data integration, linking metadata to taxonomies and so on.  With a reach and impact that touches every aspect of the organisation, why is it that so many people in the field still come from a data warehousing background?

What is even more interesting, when you pull groups together to talk about data management, they almost always end-up referring back to the role that the data warehouse plays in the enterprise.  A discussion on master data management will almost always include reference to the standardisation that has happened within the data warehouse.  A discussion on data quality generally refers back to the process to cleanse the data warehouse.  A discussion on data integration seems to include a debate on the operational role of the data warehouse.

I am very interested in the view of our community.  Do we refer back to data warehouses because that is where so many of us came from, or is it that an architecture which has a data warehouse playing an important role is conducive to good data management?




Get More from the original blog...

[MattC]

In my experience people don't start worrying about good data management until they are using it to support decisions, and this is usually at the point they are building a data warehouse. I have never seen a DW project that does not uncover data issues in the source systems, so I don't think it is necessarily true that a warehouse is needed, but the process of building one brings data quality to the fore.