Business intelligence and in-memory analytics

[Sean Lew]

Its been a year since I last wrote anything on this blog. Work has been absolutely crazy and school work hasn’t quite helped as well. I am now on my Christmas break and have abit more time to do the stuff I enjoy.

One of the technologies I am following this year is called in memory analytics. This is really an extension of Business Intelligence (BI). Let’s discuss BI before moving into in memory analytics.

BI is synonymous with data warehouse, data cubes, dashboards and reporting.

First of all, data warehouses do not provide full real time data of all your transactional databases. There are issues with performance of the legacy systems, integration, costs and hardware limitations. Therefore, reports that business users receive are 15mins late or a week old (depending on the type of data).

Moreover, data cubes require developers to build it, test it and deploy it. This takes time, effort and money. Most companies nowadays can get the top 20% of the long tail of reports right. However, the rest of the 80% is not met.

There are then reporting dashboards that try to do two things.
1) Provide a pretty front end graphs and charts to the data
2) Allow user manipulation of data and try to meet some of the unmet business reporting requirements. This generally requires a good understanding of the data, a comprehensive data dictionary and some technical skills. I am not sure how many people in businesses have such a skill.

Additionally, once a report is generated and the contents have been verified to be correct, analysis of the data, discussion/collaboration and actions needs to be taken based on what the report says. I have told a few people this: data sits in the database, information is in the report and wisdom is the outcome of the analysis and actions taken because of the report. Information management tackles the reporting side of things but getting people to take appropriate actions on the data in the report that is aligned to the overall business strategy and actions of other teams within the firm is not easy. Deriving cooperative wisdom is important but difficult.

In memory analytics aims to solve some of the current BI problems. Using a combination of better multi core processing, advanced multi threading technologies, 64 bit architecture and cost/speed of memory, it is technically possible to store all data in the memory (RAM) of the computer instead of the harddrive. What this means is that data will be real time and possibly a reduction of costs as only one database is required (instead of the transactional database and data warehouse).

The benefits of this is enormous in some industries like airline, retail, national security and banking. Real time data and analysis is critical for the optimisation of yield management, profits and identifying any arbitrage.

However, this still does not solve the issue of converting data into wisdom. Businesses need to create a culture of measuring the data in the reports to the performance metrics. Businesses track earnings and spending against budgets but tracking budgets are the result of operational performance and tracking. This is where the next step of BI should be focused on. Businesses have to encourage business units to be agile and nimble to react to the constantly changing business environments.

I do foresee that:
1) improving the technology around in-memory analytics and
2) improving processes around BI and reporting
to be the two key focus areas of organisations in the next couple of years.



More from the original blog...

[GoldenBoy]

Sean, Good Article.

The main problem in todays In-Memory is that most players load all data into memory, this creates a limit to the amount of data you can load. you can see power-pivot and qlikview struggling with large amounts of data trying to load all schema into memory.

There are other ways using in-memory today i.e. loading only what is needed in query time like SiSense Prism for example

I agree with you that data itself is not the answer, you need people to ask the right questions and have the right tool to answer them quickly.