Big Data Analytics

From Strategic Planning to Enterprise Integration with Tools, Techniques, NoSQL, and Graph

David Loshin

• Publisher: Morgan Kaufmann
• Year: 2013
• ISBN:
  9780124186644
• Topic: Computer Science
• Subtopic: Databases
• Index: Computer Science

Big Data Analytics will assist managers in providing an overview of the drivers for introducing big data technology into the organization and for understanding the types of business problems best suited to big data analytics solutions, understanding the value drivers and benefits, strategic planning, developing a pilot, and eventually planning to integrate back into production within the enterprise.

• Guides the reader in assessing the opportunities and value proposition
• Overview of big data hardware and software architectures
• Presents a variety of technologies and how they fit into the big data ecosystem

Chapter 1

Market and Business Drivers for Big Data Analytics

We consider the market conditions that have enabled broad acceptance of big data analytics, including commoditization of hardware and software, increased data volumes, growing variation in types of data assets for analysis, different methods for data delivery, and increased expectations for real-time integration of analytical results into operational processes. We examine some contrasting approaches in adopting high performance capabilities, such as simplified execution and application development models, scalable storage, alternative data management schemes, and the alternative hardware and software appliance models (as well as cloud-based or utility models) for instituting the big data platform. We suggest criteria for evaluation, including feasibility, reasonability, value, integrability, and sustainability.

Keywords

Big data; high velocity; analytics; massive data volume; big data business drivers; data growth; real-time integration; data delivery; lowered barrier to entry; application development; platform; data management; massive parallelism; MPP

1.1 Separating the Big Data Reality from Hype

There are few technology phenomena that have taken both the technical and the mainstream media by storm than “big data.” From the analyst communities to the front pages of the most respected sources of journalism, the world seems to be awash in big data projects, activities, analyses, and so on. However, as with many technology fads, there is some murkiness in its definition, which lends to confusion, uncertainty, and doubt when attempting to understand how the methodologies can benefit the organization.

Therefore, it is best to begin with a definition of big data. The analyst firm Gartner can be credited with the most-frequently used (and perhaps, somewhat abused) definition:

Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.¹

For the most part, in popularizing the big data concept, the analyst community and the media have seemed to latch onto the alliteration that appears at the beginning of the definition, hyperfocusing on what is referred to as the “3 Vs—volume, velocity, and variety.” Others have built upon that meme to inject additional Vs such as “value” or “variability,” intended to capitalize on an apparent improvement to the definition.

The ubiquity of the Vs definition notwithstanding, it is worth noting that the origin of the concept is not new, but was provided by (at the time Meta Group, now Gartner) analyst Doug Laney in a research note from 2001 about “3-D Data Management,” in which he noted:

While enterprises struggle to consolidate systems and collapse redundant databases to enable greater operational, analytical, and collaborative consistencies, changing economic conditions have made this job more difficult. E-commerce, in particular, has exploded data management challenges along three dimensions: volumes, velocity and variety. In 2001/02, IT organizations must compile a variety of approaches to have at their disposal for dealing with each.²

The challenge with Gartner’s definition is twofold. First, the impact of truncating the definition to concentrate on the Vs effectively distils out two other critical components of the message:

1. “cost-effective innovative forms of information processing” (the means by which the benefit can be achieved);

2. “enhanced insight and decision-making” (the desired outcome).

The second is a bit subtler: the definition is not really a definition, but rather a description. People in an organization cannot use the definition to determine whether they are using big data solutions or even if they have problems that need a big data solution. The same issue impedes the ability to convey a value proposition because of the difficulty in scoping what is intended to be designed, developed, and delivered and what the result really means to the organization.

Basically, it is necessary to look beyond what is essentially a marketing definition to understand the concept’s core intent as the first step in evaluating the value proposition. Big data is fundamentally about applying innovative and cost-effective techniques for solving existing and future business problems whose resource requirements (for data management space, computation resources, or immediate, in-memory representation needs) exceed the capabilities of traditional computing environments as currently configured within the enterprise. Another way of envisioning this is shown in Figure 1.1.

Figure 1.1 Cracking the big data nut.

To best understand the value that big data can bring to your organization, it is worth considering the market conditions that have enabled its apparently growing acceptance as a viable option to supplement the intertwining of operational and analytical business application in light of exploding data volumes. Over the course of this book, we hope to quantify some of the variables that are relevant in evaluating and making decisions about integrating big data as part of an enterprise information management architecture, focusing on topics such as:

• characterizing what is meant by “massive” data volumes;

• reviewing the relationship between the speed of data creation and delivery and the integration of analytics into real-time business processes;

• exploring reasons that the traditional data management framework cannot deal with owing to growing data variability;

• qualifying the quantifiable measures of value to the business;

• developing a strategic plan for integration;

• evaluating the technologies;

• designing, developing, and moving new applications into production.

Qualifying the business value is particularly important, especially when the forward-looking stakeholders in an organization need to effectively communicate the business value of embracing big data platforms, and correspondingly, big data analytics. For example, a business justification might show how incorporating a new analytics framework can be a competitive differentiator. Companies that develop customer upselling profiles based on limited data sampling face a disadvantage when compared to enterprises that create comprehensive customer models encompassing all the data about the customer intended to increase revenues while enhancing the customer experience.

Adopting a technology as a knee-jerk reaction to media buzz has a lowered chance of success than assessing how that technology can be leveraged along with the existing solution base as away of transforming the business. For that reason, before we begin to explore the details of big data technology, we must probe the depths of the business drivers and market conditions that make big data a viable alternative within the enterprise.

1.2 Understanding the Business Drivers

The story begins at the intersection of the need for agility and the demand for actionable insight as the proportion of signal to noise decreases. Decreasing “time to market” for decision-making enhancements to all types of business processes has become a critical competitive differentiator. However, the user demand for insight that is driven by ever-increasing data volumes must be understood in the context of organizational business drivers to help your organization appropriately adopt a coherent information strategy as a prelude to deploying big data technology.

Corporate business drivers may vary by industry as well as by company, but reviewing some existing trends for data creation, use, sharing, and the demand for analysis may reveal how evolving market conditions bring us to a point where adoption of big data can become a reality.

Business drivers are about agility in utilization and analysis of collections of datasets and streams to create value: increase revenues, decrease costs, improve the customer experience, reduce risks, and increase productivity. The data explosion bumps up against the requirement for capturing, managing, and analyzing information. Some key trends that drive the need for big data platforms include the following:

• Increased data volumes being captured and stored: According to the 2011 IDC Digital Universe Study, “In 2011, the amount of information created and replicated will surpass 1.8 zettabytes, … growing by a factor of 9 in just five years.”³ The scale of this growth surpasses the reasonable capacity of traditional relational database management systems, or even typical hardware configurations supporting file-based data access.

• Rapid acceleration of data growth: Just 1 year later, the 2012 IDC Digital Universe study (“The Digital Universe in 2020”) postulated, “From 2005 to 2020, the digital universe will grow by a factor of 300, from 130 exabytes to 40,000 exabytes, or 40 trillion gigabytes (more than 5,200 gigabytes for every man, woman, and child in 2020). From now until 2020, the digital universe will about double every two years.”⁴

• Increased data volumes pushed into the network: According to Cisco’s annual Visual Networking Index Forecast, by 2016, annual global IP traffic is forecast to be 1.3 zettabytes.⁵ This increase in network traffic is attributed to the increasing number of smartphones, tablets and other Internet-ready devices, the growing community of Internet users, the increased Internet bandwidth and speed offered by telecommunications carriers, and the proliferation of Wi-Fi availability and connectivity. More data being funneled into wider communication channels create pressure for capturing and managing that data in a timely and coherent manner.

• Growing variation in types of data assets for analysis: As opposed to the more traditional methods for capturing and organizing structured datasets, data scientists seek to take advantage of unstructured data accessed or acquired from a wide variety of sources. Some of these sources may reflect minimal elements of structure (such as Web activity logs or call detail records), while others are completely unstructured or even limited to specific formats (such as social media data that merges text, images, audio, and video content). To extract usable signal out of this noise, enterprises must enhance their existing structured data management approaches to accommodate semantic text and content-stream analytics.

• Alternate and unsynchronized methods for facilitating data delivery: In a structured environment, there are clear delineations of the discrete tasks for data acquisition or exchange, such as bulk file transfers via tape and disk storage systems, or via file t...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Foreword
• Preface
• Acknowledgments
• Chapter 1. Market and Business Drivers for Big Data Analytics
• Chapter 2. Business Problems Suited to Big Data Analytics
• Chapter 3. Achieving Organizational Alignment for Big Data Analytics
• Chapter 4. Developing a Strategy for Integrating Big Data Analytics into the Enterprise
• Chapter 5. Data Governance for Big Data Analytics: Considerations for Data Policies and Processes
• Chapter 6. Introduction to High-Performance Appliances for Big Data Management
• Chapter 7. Big Data Tools and Techniques
• Chapter 8. Developing Big Data Applications
• Chapter 9. NoSQL Data Management for Big Data
• Chapter 10. Using Graph Analytics for Big Data
• Chapter 11. Developing the Big Data Roadmap