Instant MapReduce Patterns - Hadoop Essentials How-to

Srinath Perera

• Publisher: Packt Publishing
• Year: 2013
• ISBN:
  9781782167709
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

In Detail

MapReduce is a technology that enables users to process large datasets and Hadoop is an implementation of MapReduce. We are beginning to see more and more data becoming available, and this hides many insights that might hold key to success or failure. However, MapReduce has the ability to analyze this data and write code to process it.

Instant MapReduce Patterns - Hadoop Essentials How-to is a concise introduction to Hadoop and programming with MapReduce. It is aimed to get you started and give you an overall feel for programming with Hadoop so that you will have a well-grounded foundation to understand and solve all of your MapReduce problems as needed.

Instant MapReduce Patterns - Hadoop Essentials How-to will start with the configuration of Hadoop before moving on to writing simple examples and discussing MapReduce programming patterns.

We will start simply by installing Hadoop and writing a word count program. After which, we will deal with the seven styles of MapReduce programs: analytics, set operations, cross correlation, search, graph, Joins, and clustering. For each case, you will learn the pattern and create a representative example program. The book also provides you with additional pointers to further enhance your Hadoop skills.

Approach

Filled with practical, step-by-step instructions and clear explanations for the most important and useful tasks. This is a Packt Instant How-to guide, which provides concise and clear recipes for getting started with Hadoop.

Who this book is for

This book is for big data enthusiasts and would-be Hadoop programmers. It is also meant for Java programmers who either have not worked with Hadoop at all, or who know Hadoop and MapReduce but are not sure how to deepen their understanding.

Instant MapReduce Patterns – Hadoop Essentials How-to

---

Instant MapReduce Patterns – Hadoop Essentials How-to

Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: May 2013

Production Reference: 1160513

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78216-770-9

www.packtpub.com

Credits

Author

Srinath Perera

Reviewer

Skanda Bhargav

Acquisition Editor

Kartikey Pandey

Commissioning Editors

Meeta Rajani

Llewellyn Rozario

Technical Editor

Worrell Lewis

Project Coordinator

Amey Sawant

Proofreader

Amy Guest

Graphics

Valentina D'silva

Production Coordinator

Prachali Bhiwandkar

Cover Work

Prachali Bhiwandkar

Cover Image

Nitesh Thakur

About the Author

Srinath Perera is a senior software architect at WSO2 Inc., where he overlooks the overall WSO2 platform architecture with the CTO. He also serves as a research scientist at Lanka Software Foundation and teaches as a visiting faculty at Department of Computer Science and Engineering, University of Moratuwa. He is a co-founder of Apache Axis2 open source project, and he has been involved with the Apache Web Service project since 2002 and is a member of Apache Software foundation and Apache Web Service project PMC. He is also a committer of Apache open source projects Axis, Axis2, and Geronimo.

He received his Ph.D. and M.Sc. in Computer Sciences from Indiana University, Bloomington, USA and received his Bachelor of Science in Computer Science and Engineering degree from the University of Moratuwa, Sri Lanka.

He has authored many technical and peer reviewed research articles, and more details can be found on his website. He is also a frequent speaker at technical venues.

He has worked with large-scale distributed systems for a long time. He closely works with Big Data technologies like Hadoop and Cassandra daily. He also teaches a parallel programming graduate class at University of Moratuwa, which is primarily based on Hadoop.

I would like to thank my wife Miyuru, my son Dimath, and my parents, whose never-ending support keeps me going. I would also like to thank Sanjiva from WSO2 who encourages us to make our mark even though projects like these are not in the job description. Finally, I would like to thank my colleges at WSO2 for the ideas and companionship that has shaped the book in many ways.

About the Reviewer

Skanda Bhargav is an Engineering graduate from VTU, Belgaum in Karnataka, India. He did his majors in Computer Science Engineering. He is currently employed with a MNC based out of Bangalore. Skanda is a Cloudera-certified developer in Apache Hadoop. His interests are Big Data and Hadoop.

I would like to thank my family for their immense support and faith in me throughout my learning stage. My friends have brought the confidence in me to a level that makes me bring out the best out of myself. I am happy that God has blessed me with such wonderful people around me without which this work might not have been the success that it is today.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <[email protected]> for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why Subscribe?

• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

Although there are many resources available on the Web for Hadoop, most stop at the surface or provide a solution for a specific problem. Instant MapReduce Patterns – Hadoop Essentials How-to is a concise introduction to Hadoop and programming with MapReduce. It is aimed to get you started and give an overall feel to programming with Hadoop so that you will have a solid foundation to dig deep into each type of MapReduce problem, as needed.

What this book covers

Writing a word count application using Java (Simple) describes how to write a word count program using Java, without MapReduce. We will use this to compare and contrast against the MapReduce model.

Writing a word count application with MapReduce and running it (Simple) explains how to write the word count using MapReduce and how to run it using the Hadoop local mode.

Installing Hadoop in a distributed setup and running a word count application (Simple) describes how to install Hadoop in a distributed setup and run the above Wordcount job in a distributed setup.

Writing a formatter (Intermediate) explains how to write a Hadoop data formatter to read the Amazon data format as a record instead of reading data line by line.

Analytics – drawing a frequency distribution with MapReduce (Intermediate) describes how to process Amazon data with MapReduce, generate data for a histogram, and plot it using gnuplot.

Relational operations – join two datasets with MapReduce (Advanced) describes how to join two datasets using MapReduce.

Set operations with MapReduce (Intermediate) describes how to process Amazon data and perform the set difference with MapReduce. Further, it will discuss how other set operations can also be implemented using similar methods.

Cross correlation with MapReduce (Intermediate) explains how to use MapReduce to count the number of times two items occur together (cross correlation).

Simple search with MapReduce (Intermediate) describes how to process Amazon data and implement a simple search using an inverted index.

Simple graph operations with MapReduce (Advanced) describes how to perform a graph traversal using MapReduce.

Kmeans with MapReduce (Advanced) describes how to cluster a dataset using the Kmeans algorithm. Clustering groups the data into several groups such that items in each group are similar and items in different groups are different according to some distan...

Table of contents

• Instant MapReduce Patterns – Hadoop Essentials How-to