Do we really build systems to improve decision making?

[Peter O'Donnell]

Just wrapping my mind around a new direction for a research project. Some of you know, I've been around the traps in the last little bit doing my schtick on interfaces. Of course everybody says the interface is important but I don't think we do a lot about it.Main message of my talk is that we don't devote enough effort or resources to the design of the interface. We just use the "orthodox" tables and charts provided by the vendors in our data displays (and the same for the navigation between views of data) - and many of them are not very good.

We just did a study where we can two different groups of people the same data - but one group got the data in "orthodox" BI charts, complete with 3D effects or gradient shading, not over the top but typical of BI systems today - and the other group an equivalent set of charts - very plain in their design (think the design principles of Tufte or Few). As we hypothesised, when asked questions about the data that required accurate analysis - the group that got the plain charts did much better than the group that got the sexier charts. Neat to see that this study and article that Trevor Clarke wrote for Computerworld after a chat with me has stirred up some debate.

Most BI systems provide users with a slicer dicer style interface in the hope that they will explore the information our systems give them access to - finding the "number" that solves their problem - just like in the demo's the vendors give. Sadly for that dream, in another study, we showed that a lot of people can't use pivot tables.


I think we have a problem. A serious one. Nigel Pendse thinks (based on his global survey) that the mean usage rate of a BI system is around 7 to 8%. That seems a little low to me - but its been a while since we collected that kind of data, and with web technologies (esp. in the last couple of years) meaning that more people have access to BI systems, the proportion using them might have fallen - so while 7-8% seems low, his data collection is good, I've got to think that he must be close to right. Even if he's not, the best take up rates we have seen in our case study work aren't very good.


In my presentations and writing I've been arguing that one main reason for this low usage rate is the interface we provide. I think that most BI systems have very similar interfaces (with few exceptions, the character of the main offerings from the BI vendors is identical). Those interfaces suit a few people, but not many. We need a major re-focus on the role and place of the interface if we are going to get better rates of usage of BI systems.


Now, there are a couple of underlying assumptions to those assertions. The first is that its desirable for lots of people to have access to a BI system, and to use that in their daily work. That may not be true. There have been studies that show that using decision support systems actually decreases decision performance - so the people using BI systems might actually be doing their organisations and themselves harm. I'm happy for the moment to believe in the value of BI systems, and leave the research on that one to other people.


The second assumption that I have been thinking about - is that people use BI system to help make decisions. What if that's not true? What if people use these systems for some other purpose. It could be (as would be predicted by cognitive artefacts like the confirmation bias) that our systems are to build a case to convince ourselves and others that a decision already made is correct?


If that's the case, no wonder we are putting 3d funnel charts in our systems - in response to user demand. The aim of the display of data would be to impress people, show them how clever we are, how the data supports our strategy (better obscure the data a bit just in-case it doesn't). Im exaggerating the case a little (of course), but it is interesting to think about - and not to hard I think to design a research instrument that can be used to go out and find out what BI systems are actually being used for: decision making or decision justification?


POD
:-( back to marking exam papers.







Get More from the original blog...