Reality mining, pervasive data capture, and how Big Data can create value
On Tuesday I gave the opening keynote on The Future of Information Infrastructure at the Implementing Information Infrastructure Symposium.
CIO magazine did a nice article titled IIIS: Big Data driving new trends which reviews my keynote and the one immediately after from Steve Duplessie, one of the world’s top analysts on data and storage. It says:
Speaking at the event, co-hosted by Storage Networking Industry Association A/NZ and Computerworld Australia, strategy advisor, author and futurist, Ross Dawson, said “reality mining” — the gathering of data based on the activities of people in a given environment — was a major trend to emerge out of, and contributor to, Big Data.
“If you look at an office environment there is an extraordinary amount of data to look at. For example, what gestures people are making, where are they looking, what conversations are they having, how much are they smiling when they speak to each other?” he said.
“You can literally get terabytes of data out of just a few hours of this. That data is being collected to drive productivity; to design new ways to enhance collaboration and create value inside organisations.”
The state of the art work in the field has been done at MIT’s Reality Mining and Human Dynamics programs, however they have just been scratching at the surface of the potential. The potential to understand at a deep level what supports positive, constructive human interaction in an organizational context is compelling.
Clearly there are sensitivities in the minutiae of people’s activities through the day being captured, and it will be fascinating to see what the social and regulatory response to this is. It could go either way, but there is a fair chance that some office environments move to pervasive data capture, and using that to support productivity.
Dawson said an added emphasis for this type of data collection was research suggesting that organisations which used data-based decision making had been able to achieve 5 to 6 per cent greater productivity.
“That requires this kind of reality mining,” he said. “This requires all that data within the organisation to be captured and used effectively.”
In fact the MIT research on data-driven decision-making and productivity suggests that kind of productivity increment is already available. There is now the possibility of pushing that productivity gap even higher, supporting the trend to increasing divergence of organizational performance.
To enable effective data capture organisations would also have to transition away from the mass collection of unstructured data to a process where all data was tagged at the point of capture.
“The vast majority of the information we have is unstructured… so what is critical is tagging data at its source,” he said. “In this explosion of data… the key thing is to add structure to it by tagging at the source as the cost of doing it after vastly more.”
Dawson said video was a good example of this, where not only the speakers in the video, but the themes and even emotions of the speakers could be tagged and used later in reality mining exercises.
Such tagging work would likely be carried out by machines but also by services such as Amazon’s Mechanical Turk which offers an on-demand workforce to “build human intelligence” directly into applications.
The issue of shifting unstructured to structured data is central in creating value from the vast amount of information we store. Tagging data at source is critical, as this is by far the lowest cost way to add metadata. However we can increasingly use analytics to structure data, for example in mining video for not just words said, but emotion and more.
With the security of the public Cloud being a major inhibitor to adoption, Dawson said Cloud ratings agencies would likely emerge that would rate the security of Cloud providers in a similar to financial ratings agencies Standard & Poor’s and Moody’s.
“This goes alongside the ‘reputation economy’. This decade is the one where reputation is driving almost everything we do — reputation of individuals, people’s influence on social media and the like,” he said.
“So for companies, being in the Cloud doesn’t automatically mean you are reputable. We need the creation of organisations like S&P and Moody’s rate institutions in the Cloud.”
As well as Cloud ratings agencies, companies which offered services such as Cloud-based distributed data bases, and Cloud brokers — organisations trading storage and processing power on spot markets — will emerge.
While the cloud is exploding, led by the reputation of established leaders, we must differentiate between the quality of cloud providers. Part of that will be done by newly-emerging rating organizations, part will be done by the crowd in establishing reputation measures. Just as the financial ratings agencies are not always right, the cloud ratings won’t always predict failures. Yet they will help clients to sort through a proliferation of vendors to make choices.
Definitely read the full article for great insights from Steve Duplessie’s keynote as well.
