The 6 capabilities that drive future business value from Staggeringly Enormous Data

I recently participated in a panel interview on SkyNews of “some of the brightest minds in business technology”, including futurist Mark Pesce, commentator Brad Howarth, and myself. It’s just come out on the web and having now been able to watch it I think it turned out to be a very interesting discussion.

To see the video click on the image below.

After covering some of the major trends of 2012 towards the end of the panel we turned out attention to what was coming in the year ahead. When asked what I thought needed to be on the business agenda, my response was Big Data, or as I more accurately described it, Staggeringly Enormous Data.

As I wrote in an article on Governance as opportunity, research by MIT’s Eric Brynjolfsson showed 5% higher productivity from organizations that do ‘data-driven decision making’.

The key issue is the capabilities that organizations will need to get value from Big Data, and how to develop those capabilities.

I covered some of these points in my opening keynote at the Implementing Information Infrastructure Symposium, which was also written up in Computerworld under the title Big data to dictate the future of IT infrastructure.

Sis of the most critical capabilities for creating business value from Big Data are:

Consistently capturing relevant data. Many organizations already have a vast amount of data, and their immediate pressing issue is to extract value from that. However sustainable competitive advantage can only be gained from capturing the broadest range of potential valuable data. There does need to be a simple analysis of the potential value versus cost of gathering new categories of data, however as storage costs slide more data domains become viable. Emergent patterns and resultant value cannot always be anticipated so the key is developing consistent, efficient gathering of data that could drive better business decisions or operations today or down the track.

Adding metadata. Most organizations have vastly more unstructured than structured data, and to extract value from their current trove requires a preliminary exercise of structuring and tagging that data that can be expensive and time-consuming. Capabilities in adding metadata to existing data are important, with automated tagging and the use of crowdsourcing two of the most promising domains. However future capabilities must be focused on adding metadata at source. Effective tagging as data is gathered will pave the way for far broader and deeper applications of the data.

Infrastructure and architecture. Clearly plenty of computing “iron” is required to store and manage massive amounts of data. More important is the architecture, particularly in using, where appropriate, distributed storage and processing across often external as well as internal systems.

Data analytics. Demand for talented data scientists outweighs supply. Some organizations are fortunate to be able to attract and retain outstanding data analysts. However others are able to draw on external talent by leveraging data science competitions or defining specific projects. However what is less often recognized is that most data scientists are good at answering questions but not necessarily asking the right questions. As such data analytics capabilities must be linked to identifying where business value can or might be created.

Executive focus. Unless the top executive team recognizes the potential value of big data it will not allocate the requisite resources. More importantly, it will not spend the time to explore the questions and the possibilities that could lead to business value. You cannot necessarily predict where the greatest value in data analysis lies, so there must be the willingness to search. Ultimately, executives need to allow the outputs from their data pool to shape their decision making, and to understand when and where data and intuition have their place.

Communication of data analysis. The link between data analytics and executive focus comes largely from effective communication. Data visualization is a primary tool, which requires software but also softer skills in using visual representation to relate data to business value and decisions. There must be a rich two-way communication from those who make business decisions, be it at strategic or front-line levels, and those who can serve them with the fruits of Staggeringly Enormous Data.