Data is everywhere but using it effectively poses challenges

[admin3]

The Australian, Edition 1 TUE 23 NOV 2010, Page 040

Data is everywhere but using it effectively poses challenges - SYSTEMS INTEGRATION: SPECIAL REPORT

By: Ian Grayson

While businesses focus on the task of application and infrastructure integration, attention is also being drawn to the challenges around integration of data.

In many organisations, data is stored in a plethora of places that have little or no linkage between them. Customer billing systems often cannot talk to marketing databases and information gleaned by sales teams can disappear as staff change roles.

The challenge is exacerbated by the growing volumes of data being generated and gathered every day. To extract any sort of commercial advantage from this deluge, businesses need to find ways to improve both data integration and analysis.
Gartner analyst Donald Feinberg says the trend, dubbed ``big data'', is nothing new but the challenges it presents are rapidly increasing.

``For some companies, 500GB of unmanaged data is a challenge,'' he says. ``For others, a petabyte of data is no longer considered that much. The key is what you do with it.''

Chief development officer at data warehousing specialist Teradata Scott Gnau says the challenges brought about by the big data trend are multi-faceted. They include finding ways to collect it, store it, manage it and analyse it.

``Most enterprises generate petabytes of data that they simply throw away,'' he says. ``Yet market leaders have already figured out the benefits of integrating data so it can be used in new ways.''

Gnau says that once a business has integrated its data into a single location, analytic tools allow valuable insights to be extracted from it.

``Data can be analysed in near real time, which allows a company to respond to rapidly changing market conditions,'' he says. For example, an insurance company could change its policy conditions or rates of acceptance based on events that have just occurred.

He also points to the location-based data now being gathered from mobile phones, GPS devices, RFID tags and other smart chips. All this unstructured data must also be captured and stored before it can be of use.

``Analytics will increasingly drive the world's economy,'' Gnau says.

The director of sales analytics and reporting at telecommunications company AT&T, Matthew Boos, says such data analysis will become more critical for companies as volumes increase.

``Thanks to mobile broadband networks, it won't be long before millions of devices and sensors will be connected and enabled,'' he says.

Boos gives the example of putting SIM chips in the tops of pill bottles. The chips can record information on when the bottle was opened and transmit that data back to a doctor, enabling remote health monitoring.

``There is an almost unimaginably endless array of devices and data that can be transmitted and captured,'' he says. ``Soon not just you will be on the internet, but much of what you own will be on the internet. The volume of data will exceed our ability to understand it. Building the infrastructure needed to gather, store and analyse that data will become increasingly important.''

Mark Jeffrey, senior technology lecturer at the Kellogg School of Management, says the organisations that undertake data integration projects will be best placed to take advantage of the looming era of ``big data''.

``Things like measuring customer lifetime value become possible when you store details of every transaction they make with your company,'' he says.

Jeffrey says data integration will become vital for all organisations as they fight to both deal with the flow of information and make sense of it from a business perspective.