What is Big Data?

What is Big Data

The growth in volume of big data is huge and is coming from everywhere, every second of the day. Systems and devices including computers, smart phones, appliances and equipment generate and build upon the existing massive data sets.
 

But what is big data? Big data is a broad term for both structured and unstructured data sets so large and complex that traditional data processing applications and systems cannot adequately handle them. Big data often powers predictive analytics. Analysis of data sets are used to find new correlations to identify business trends, prevent diseases, combat crime and much more. 
 

Industry analyst, Doug Laney, defined Big Data in terms of the three Vs:

• Volume: Terabytes, records, transactions, tables and files
• Velocity: Batch, near time, real time and streams
• Variety: Structured, unstructured and semi structured

Apache Hadoop (aka Hadoop) is open source software used for distributed storage and processing of big data. Hadoop has been packaged and integrated in large distributions (aka distros) by companies such as Cloudera, Hortonworks, MAPR and Pivotal to run big data workloads.

Roadblocks to Successful Big Data Projects

Companies often encounter roadblocks when implementing big data projects. These can include budget constraints, lack of IT expertise and risk of platform lock-in.

Budget Constraints

Budget constraints and cost are the top reasons why many companies are shying away from deploying big data, according to a study performed by Deloitte. It can be hard to justify investing in new IT infrastructure to process large amounts of data, especially if the business does not yet havean immediate business case.

IT Expertise

Processing big data workloads is different than processing typical enterprise application workloads. Big data workloads are processed in parallel, instead of sequentially. IT typically prioritizes business critical workloads and schedules lower priority jobs in batches at night or when there is excess capacity. With big data analytics, many use cases must be run in real-time for live analysis and reaction. This forces IT to change data center policies and learn new tools to create, manage and monitor these new workloads.

Platform Lock-In

Companies need to choose the right type of infrastructure to run their applications and data. Procuring hardware takes time. Going into the cloud may be great for a proof of concept, but it carries the risk of platform lock-in, comes with security concerns and incurs tremendous cost at scale. Companies must also decide which Hadoop distribution to select, with Cloudera, Hortonworks, MAPR, and Pivotal all offering competing (and incompatible) architectures. There are many decisions that, once made, make it difficult for a company to pivot later so many companies just delay having the big data conversation.