A ray of light on Dark Data

In the 21st century of our Information Age the winner is the one who owns information and is capable to get sense out of it. Our data provides us with better understanding of a market, customers and competitors. This is why we gather all available information and this is why we care about Big Data, Master Data and Data Governance nowadays.

But there is one thing missing.

In our systems we store huge amounts of information that we do not use and sometimes even don’t know about. For example, mobile geo-location data can reveal traffic patterns, which may help in business planning or in targeted marketing campaigns.

In fact, there may be millions of dollars hidden somewhere in our data warehouses and mainframes. Having data collected, processed and stored we don’t manage to monetise it. This data is known as Dark Data. If you take a challenge to analyse Dark Data, the return on investment could be enormous.

If Big Data is the new Oil, then Dark Data is Shale Gas. And just as drilling for gas-rich shale means diving deeper into the Earth’s crust, process of extracting Dark Data requires big efforts and comprehensive approach.

What is Dark Data?

  • It’s data, which has been collected, stored and is available, but not leveraged by business;
  • It’s data, which has been collected, but not processed or analysed because of high costs of analysis, security restrictions or whatever;
  • It’s data, which exists, but has not been collected.

They say:

Dark Data is information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes. Gartner, Inc.

Up to 90 percent of big data is Dark Data. IDC
Dark-Data-ChartSource : HP/Syncsort

Dark Data you may get from:

  • Websites log files which can give clues to visitor’s behaviour that can indicate a consumer sentiment;
  • Customer demographics (especially relevant to Telecoms);
  • Caller’s time zone information which may be found in call centre logs;
  • Information from social networks (Facebook, Twitter);
  • How customers use products or services;
  • How satisfied customers are with support services.

Operating such information is tempting, but to get advantage of it requires efforts. Collecting hidden data from different, often obsolete, data sources may require significant financial costs. Furthermore, use and storage of such information entails certain legal and regulatory risks. For example, disclosure of data protected by NDA (patient records, credit card details), contained in some Dark Data collections, can involve legal and financial liability.

The situation is actually better than it may seem. Dark Data, unlike dark matter, can be brought to light and help to achieve potential ROI. Benefits that companies may get from Dark Data are in getting a much deeper understanding of their customers, which eventually results in additional revenue.

But this is only for those, who are eager to take a risk of dealing with it.