In part 1 of this series, we disambiguated the terms Big Data and Analytics. This post focuses on the forces driving Big Data in the evolution of Big Data and Analytics.
[Part 1] [Part 2] [Part 3] [Part 4]Big Data and Analytics has quietly crept up on many of us. Only a few decades ago, texts held in the United States Library of Congress and the AT&T global telephone number index was our reference point for big data.
Big Data today is a much more complex combination of dynamic streaming data with large static data sets. Big Data is now text, audio, images, video and richly tagged and indexed amalgamations of content stored in many analog and digital formats. Big Data presents a rich opportunity to create value created by social, political, economic and industry drivers.
Democratization Of Computing Power And Infrastructure
Formerly, an expensive class of computing called High Performance Computing (HPC) was only available to academia and large institutions. Today, globally distributed multi-blade servers with multi-core 64-bit CPU architectures are common. Disk storage is a low-cost commodity below $75 per terabyte. Open source operating systems (OS) like Linux make these resources available – at a reasonable cost.
Consumers use cloud and client computing on billions of smartphones and tablets. Developers deliver consumer, commercial and industrial applications on these mobile devices. The shift to mobile platforms has changed our perception of software, and its maintenance. Infrastructure-centric changes are driving rapid entrepreneurial innovation in a race to satisfy consumers. A new class of easy-to-use applications that capture and create data at alarming rates all contribute to the explosion of Big Data.
Big Data And Analytics – From Active Creation To Passive Capture
Only a few years ago, data had to be actively created and was in the hands of the few. Consumer data was captured on clipboards by survey-takers. Expensive software applications on desktop computers required training. Data was hoarded in isolated silos. Most of all, it required time to create.
Today, data is increasingly passively captured and is in the hands of the many. IDC reported in the Digital Universe Study of September 2011 that “70% of information is created by individuals.” Largely due to the proliferation of smart phones and mobile devices. GPS tracking applications and social media tools such as Facebook, Twitter, Foursquare, and Google+ have flipped the ratio of actively created data in favor of passively captured data. Our mobile phones produce a constant streaming record of our personal activity including text messages, email, voicemail, call records, use of our camera.
Constant Real-time Data
In addition to smart phones generating passively captured consumer data, an entire new category of data-generating sensors has been deployed. These sensors are generating massive amounts of Big Data in real-time.
Sensors track almost everything today, like FedEx packages from origin to destination. Our automobiles recording maintenance events in embedded sensors and have GPS systems both constantly interfacing with Big Data as we drive. Sensors track efficiency of goods through the supply chain for manufacturers and retail giants like Amazon. These devices generate a stream of location-centric time-stamped data forming interesting real-time fact records.
Pervasive Proliferation of Photos, Audio And Video
Historically, we had not fully grasped the potential of digitally recorded photos, audio and video as data sources – they lived in a data silo. Images, audio and video were (and to some degree still are) stifled by commercial production industries through varying degrees of digital rights management.
Two things have put the lion’s share of image, audio and video creation in the hands of the consumer. Integration of cameras, audio and video recording technology in smartphones has given us the tools. Social media and the proliferation of sharing and re-sharing content has changed our behavior. Social applications like Instagram, YouTube, Vimeo, Facebook and Twitter have catalyzed this shift accelerating the evolution of Big Data and Analytics.
Looking Ahead
Each of these drivers have social, political, economic, and industry –centric elements that have created a perfect convergence of transformative large-scale digital, social and cultural change. This kind of change is transformational, and irreversible.
The biggest change may be the volume ratio between past and present data our mobile apps analytically consider. This in turn will drive individual decisions for personal, business political and economic issues. Governments and institutions will be revolutionized by this ratio flip. Rearward-looking analysis will become less interesting and increasingly less relevant to the present and future. An entire set of value-metrics will enable us to leverage future technical and social evolution in Big Data, and the resulting Analytic value we extract from it.
Image credits: IDC from the EMC2 sponsored 2011 Digital Universe Study,
Featured Image: metaroll via photopin cc
