The devil is in the detail

One of the common questions in Business Intelligence is "what level of detailed information should we store?" (this is often called granularity). The received wisdom is always to keep it as low-level (detailed) as possible. As a rule of thumb this is great. We can attest to the user frustration when data is summarised and details are discarded that would help give meaning to the summarised data.

However, as with all rules of thumb, it can be somewhat simplistic. For example, what is the lowest level? If you wish to see a history of changes then that is a very low level of detail, and correspondingly a large volume of data. In addition, detailed data may be useless e.g. do we care about every click on a web page?

Read next »