The growth in volume of big data is huge and is coming from everywhere, every second of the day. Systems and devices including computers, smart phones, appliances and equipment generate and build upon the existing massive data sets.
But what is big data? Big data is a broad term for both structured and unstructured data sets so large and complex that traditional data processing applications and systems cannot adequately handle them. Big data often powers predictive analytics. Analysis of data sets are used to find new correlations to identify business trends, prevent diseases, combat crime and much more.
Industry analyst, Doug Laney, defined Big Data in terms of the three Vs:
Apache Hadoop (aka Hadoop) is open source software used for distributed storage and processing of big data. Hadoop has been packaged and integrated in large distributions (aka distros) by companies such as Cloudera, Hortonworks, MAPR and Pivotal to run big data workloads.
Companies often encounter roadblocks when implementing big data projects. These can include budget constraints, lack of IT expertise and risk of platform lock-in.
Budget constraints and cost are the top reasons why many companies are shying away from deploying big data, according to a study performed by Deloitte. It can be hard to justify investing in new IT infrastructure to process large amounts of data, especially if the business does not yet havean immediate business case.
Processing big data workloads is different than processing typical enterprise application workloads. Big data workloads are processed in parallel, instead of sequentially. IT typically prioritizes business critical workloads and schedules lower priority jobs in batches at night or when there is excess capacity. With big data analytics, many use cases must be run in real-time for live analysis and reaction. This forces IT to change data center policies and learn new tools to create, manage and monitor these new workloads.
Companies need to choose the right type of infrastructure to run their applications and data. Procuring hardware takes time. Going into the cloud may be great for a proof of concept, but it carries the risk of platform lock-in, comes with security concerns and incurs tremendous cost at scale. Companies must also decide which Hadoop distribution to select, with Cloudera, Hortonworks, MAPR, and Pivotal all offering competing (and incompatible) architectures. There are many decisions that, once made, make it difficult for a company to pivot later so many companies just delay having the big data conversation.
The role of infrastructure, whether it’s physical or virtual, is to support applications. This includes traditional business critical applications as well as modern cloud, mobile and big data applications.
Virtualizing big data applications like Hadoop offers a lot of benefits that cannot be obtained on physical infrastructure or in the cloud. Simplifying the management of your big data infrastructure gets faster time to results, making it more cost effective. VMware is the best platform for big data just as it is for traditional applications.
Simplify operations and maintenance of your big data infrastructure.
Decrease CapEx cost through cluster consolidation. Decrease OpEx through automation and simple workflows.
Get your infrastructure on demand so you can rapidly deliver business value.
Pool your resources and increase server utilization. Automation of workload mobility adds to process efficiencies.
Ensure control and compliance of your sensitive data.