1 Introduction to Big Data and Hadoop

Introduction to Big Data and Hadoop

Objectives

·         Identify the need for Big Data

·         Explain the concept of Big Data

·         Describe the basics of Hadoop

·         Explain the benefits of Hadoop

Introduction

Over 2.5 exabytes (2.5 billion gigabytes) of data is generated every day.

Following are some of the sources of the huge volume of data:

·         A typical, large stock exchange captures more than 1 TB of data everyday

·         There are around 5 billion mobile phones (including 1.75 billion smart phones) in the world

·         YouTube users upload more than 48 hours of video every minute.

·         Large social networks such as twitter and facebook capture more than 10 TB of data daily.

·         There are more than 30 million networked sensors in the world.

Types of Data

There are three types of data:

·         Structure Data: Data which is represented in a tabular format. E.g. Databases

·         Semi-structured Data: Data which does not have a formal data model. E.g. XML files

·         Un-structured Data: Data which does not have a predefined data model. E.g. Text files

Characteristics of Big Data

Big data has three characteristics: variety, velocity, and volume.

·         Variety: Variety encompasses managing the complexity of data in many different structures, ranging from relational data to logs and raw text.

·         Velocity: Velocity account from streaming of data and movement of large volume data at a high speed.

·         Volume: Volume denotes the scaling of data ranging from terabytes to zettabytes.

Appeal of Big Data Technology

Big Data Technology is appealing because of the following reasons:

·         It helps to manage and process a huge amount of data cost efficiently

·         It analyzes data in its native form, which may be unstructured, structured, or streaming.

·         It captures data from fast-happening events in real time.

·         It can handle failure of isolated nodes and tasks assigned to such nodes.

·         It can turn data into actionable insights.

Business Benefits of Big Data Technology

Following are the business benefits of implementing Big Data technology, with examples:

·         It can help organizations to create personalized products, gain insight into products that are profitable, and retain customers by solving their problems.
Example: Utilizing Big Data Analytics permits banks to study the money-saving patterns and practices of individual clients.

·         The Big Data analytic solutions support or automate cost cutting, bring greater efficiency of operations and the evaluation of historical trends.
Example: Using Big Data Analytics, banks keep track of their client’s geographical shopping locations.

·         By using Big Data predictive analysis techniques, organizations can provide an early warnings of a problem and enable preventive maintenance to avoid a potential outage
Example: A portable computer manufacturer has the capacity to assemble and break down information utilized as part of segment assembling. This information can help the producer to focus on satisfactory levels of heat, vibration, and different variables utilized.

·         Big Data offers a range of analytical techniques help organizations to develop new products and services.
Example: An application can analyze data to the most granular level, even to observe that the customers who bought a smart phone also bought memory cards or back covers.

Traditional IT Analytics Approach

The following are the requirements of the traditional IT analytics approach and the challenging factors:

Requirements:

·         The business team needs to define questions before IT development.

·         They need to define data sources and structures.

Challenging Factors:

·         The requirements are iterative and volatile.

·         The data sources keep changing.

In typical scenario of traditional IT systems development, the requirements are defined, followed by solution design and build. Once the solution is implemented, queries are executed. If there are new requirements or queries, the system is redesigned and rebuilt.

Define Requirements à Design Solution à Execute queries à Redesign and Rebuild for new requirements.

Approach for Big Data Solutions

Following are the requirements for using Big Data technology as a platform for discovery and exploration, and the challenges overcome by the same:

Requirements

·         The business team needs to define data sources

·         They need to establish the hypothesis

Challenges overcome by Big Data

·         The technology should enable explorative analysis.

·         Data systems and sources need to be integrated as required.

The steps illustrates how IT systems are built with the help if Big Data technology.

·         Initial Data Sources are identified

·         IT Team creates a platform for creative exploration of available data and content

·         The business teams determine the questions to ask and test hypothesis

·         Any new questions lead to addition of data sources and integration without the need to redesign or rebuild the platform.

Big Data Technology Capabilities

·         Understand and navigate Big Data sources

·         Manage and store huge volume of a variety of data

·         Process Data in reasonable time

·         Ingest data at a high speed

·         Analyze unstructured data

·         Bear faults and exceptions

Big Data Use Cases

The use cases of Big Data Hadoop are given below:

·         Automotive: Auto sensors reporting location, problems

·         Communication: Location based advertising

·         Consumer Packaged Goods: Sentiment analysis of what’s hot, customer service

·         Financial Services: Risk and portfolio analysis New Products

·         Education and Research: Experiment sensors analysis

·         High Technology/Industrial Mfg.: Mfg. quality Warranty analysis

·         Life Sciences: Clinical Trials Geonomics

·         Media/ Entertainment: Viewers/Advertising Effectiveness

·         Online Services/Social Media: People and career matching

·         Health Care: Patient sensors, monitoring, EHRs Quality of care

·         Oil and Gas: Drilling exploration sensor analysis

·         Retail: Consumer sentiments Optimized marketing

·         Travel and Transportation: Sensors analysis for optimal traffic flows. Customer sentiments

·         Utilities: Smart Meter analysis for network capability

·         Law Enforcement and Defense: Threat analysis – social media monitoring, photo analysis

Challenges of Big Data

Following are the challenges that need to be addressed by Big Data Technology:

·         Fault tolerance and handling the system uptime and downtime

o   Using commodity hardware for data storage and analysis

o   Maintaining a copy of the same data across clusters

·         Combining data accumulated from all systems

o   Analyzing data across different machines

o   Merging of data

Introduction to Hadoop

Following are the facts related to Hadoop and why it is required:

What is Hadoop?

·         A free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment.

·         Based on Google File System (GFS)

Why Hadoop?

·         Runs a number of applications on distributed system with thousands of nodes involving petabytes of data

·         Has a distributed file system, called Hadoop Distributed File System or HDFS which enables fast data transfer among the nodes.

·         Further it encompasses a distributed processing framework called MapReduce

Hadoop and Traditional RDBMS

Feature	RDBMS	Hadoop
Computing Model	·      Notion of transactions ·      Transaction is the unit of work ·      ACID properties, Concurrency control	·      Notion of jobs ·      Job is the unit of work ·      No concurrency control
Data Model	·      Structured data with known schema ·      Read/Write mode	·      Any data will fit in any format ·      (un)(semi)structured ·      Read Only Mode
Cost Model	·      Expensive Server	·      Cheap commodity machines
Fault Tolerance	·      Failures are rare ·      Recovery mechanisms	·      Failures are common over thousands of machines ·      Simple yet efficient fault tolerance

History and Milestones Of Hadoop

Hadoop Originated from the Nutch open source project on search engines and works over distributed network nodes.

Period	Milestone
2003 & 2004	Google released two papers which provided insight into their success. The google file system or GFS and MapReduce. Simplified Data processing on large clusters. The papers told the world how Google performed large scale data processing.
July 2005	Nutch used GFS to perform MapReduce operations
Feb 2006	Nutch started a Lucene sub project which led to the era of Hadoop
Apr 2007	Yahoo started using Hadoop on a 1000-node cluster
Jan 2008	Apache took over Hadoop and made it a top-level project
Jul 2008	A 4000-node cluster with Hadoop was tested by Apache. The performance of that cluster was surprisingly the fastest when compared to the other technologies implemented that year
May 2009	Hadoop Successfully sorted a petabyte of data in 17 hours
Dec 2011	Hadoop reached version 1.0

Hadoop Core Services and Components.

Major components of Hadoop are:

·         HDFS: HDFS runs on commodity machines which are low in cost and hardware. It is highly fault tolerant and efficient enough to process huge amount of data.

·         NameNode: Is the brain of the system. It stores the Metadata of the data blocks along with location of data blocks. If this NameNode crashes the entire system is dead.

·         Secondary NameNode: Is the replica of the Primary NameNode. This is used to ensure that even if the Primary NameNode crashes Hadoop system is not dead, but name space image on Secondary NameNode can be used to restart the system.

·         DataNode: Stores the blocks of data.

·         JobTracker: Schedules client jobs and creates Map or Reduce tasks and schedules them. It can run on the same machines as NameNode or different Node.

·         TaskTracker: Runs on DataNodes and its primary responsibility is to run the MapReduce tasks assigned by the name node.

	Master	Slave 1	Slave 2	…	Slave N
MapReduce	JobTracker
	TaskTracker	TaskTracker	TaskTracker		TaskTracker
HDFS	NameNode
	DataNode	DataNode	DataNode		DataNode

HDFS Architecture

HDFS architecture and be summarized as follows:

·         The NameNode is the master and DataNode are the slaves.

·         NameNode is the brain of the system, and is accessing client data. DataNode manages the storage of data.

·         The data is split into files of one or more blocks.

·         When a client needs a data, it first interacts with NameNode that holds the MetaData and replies back to client with location of the Data on DataNodes.

·         After this, client starts interactions with DataNode, till the time data requirement is completed.

Organizations Using Hadoop

The following table shows how various organizations use Hadoop:

Name of the Organization	Cluster Specifications	Uses
A9.com: Amazon	Clusters vary from 1 to 100 nodes	·    Amazon’s product search indices are built using this program ·    Processes millions of sessions daily for analysis
Yahoo	More than 100,000 CPUs in approximately 20,000 computers running Hadoop; biggest cluster has 2000 nodes {2 * 4 cpu boxes with 4 TB disk space}	·    To support research for ad systems and web search
AOL	Cluster size is 50 machines, Intel Xeon, dual processors, and dual core, each with 16GB RAM and 800 GB hard disk in total of 37 TB HDFS capacity	·    For variety of functions ranging from generating data to running advanced algorithms for performing behavioral analysis and targeting
Facebook	320-machines cluster with 2,560 cores and about 1.3 PB raw storage	·    Storing copies of internal logs and dimension data sources ·    Used as a source for reporting analytics and machine learning

Summary

·         Big Data relies on volume, velocity, and variety with respect to processing.

·         Data can be divided into three types – unstructured data, semi-structured data, and structured data.

·         Big Data technology understands and navigates big data sources, analyzes unstructured data, and ingests data at a high speed.

·         Hadoop is free, Java based programming framework that supports the processing of large data sets in a distributed computing environment

·         Hadoop originated from the Nutch open source project on search engines and works over distributed network nodes

The core services of Hadoop are NameNode, DataNode, JobTracker, TaskTracker, and Secondary NameNode.