About Course

• This course is a zoom-in, zoom-out, hands-on workout involving Hadoop, MapReduce and the art of thinking parallel. 

  Zoom-in, Zoom-Out:  This course is both broad and deep. It covers the individual components of Hadoop in great detail, and also gives you a higher level picture of how they interact with each other. 

  Hands-on workout involving Hadoop, MapReduce : This course will get you hands-on with Hadoop very early on.  You'll learn how to set up your own cluster using both VMs and the Cloud. All the major features of MapReduce are covered - including advanced topics like Total Sort and Secondary Sort. 

  The art of thinking parallel: MapReduce completely changed the way people thought about processing Big Data. Breaking down any problem into parallelizable units is an art. The examples in this course will train you to "think parallel". 

Topics covered in this course

• Curriculum

  Section 1: Introduction
  	Lecture 1	You, this course and Us	01:52
  Section 2: Why is Big Data a Big Deal
  	Lecture 2	The Big Data Paradigm	14:20
  	Lecture 3	Serial vs Distributed Computing	08:37
  	Lecture 4	What is Hadoop?	07:25
  	Lecture 5	HDFS or the Hadoop Distributed File System	11:00
  	Lecture 6	MapReduce Introduced	11:39
  	Lecture 7	YARN or Yet Another Resource Negotiator	04:00
  Section 3: Installing Hadoop in a Local Environment
  	Lecture 8	Hadoop Install Modes	08:32
  	Lecture 9	Setup a Virtual Linux Instance (For Windows users)	15:31
  	Lecture 10	Hadoop Standalone mode Install	09:33
  	Lecture 11	Hadoop Pseudo-Distributed mode Install	14:25
  Section 4: The MapReduce "Hello World"
  	Lecture 12	The basic philosophy underlying MapReduce	08:49
  	Lecture 13	MapReduce - Visualized And Explained	09:03
  	Lecture 14	MapReduce - Digging a little deeper at every step	10:21
  	Lecture 15	"Hello World" in MapReduce	10:29
  	Lecture 16	The Mapper	09:48
  	Lecture 17	The Reducer	07:46
  	Lecture 18	The Job	12:27
  Section 5: Run a MapReduce Job
  	Lecture 19	Get comfortable with HDFS	10:58
  	Lecture 20	Run your first MapReduce Job	14:30
  Section 6: Juicing your MapReduce - Combiners, Shuffle and Sort and The Streaming API
  	Lecture 21	Parallelize the reduce phase - use the Combiner	14:39
  	Lecture 22	Not all Reducers are Combiners	14:31
  	Lecture 23	How many mappers and reducers does your MapReduce have?	08:23
  	Lecture 24	Parallelizing reduce using Shuffle And Sort	14:55
  	Lecture 25	MapReduce is not limited to the Java language - Introducing the Streaming API	05:05
  	Lecture 26	Python for MapReduce	12:19
  Section 7: HDFS and Yarn
  	Lecture 27	HDFS - Protecting against data loss using replication	15:32
  	Lecture 28	HDFS - Name nodes and why they're critical	06:48
  	Lecture 29	HDFS - Checkpointing to backup name node information	11:10
  	Lecture 30	Yarn - Basic components	08:33
  	Lecture 31	Yarn - Submitting a job to Yarn	13:10
  	Lecture 32	Yarn - Plug in scheduling policies	14:21
  	Lecture 33	Yarn - Configure the scheduler	12:26
  Section 8: Setting up a Hadoop Cluster
  	Lecture 34	Manually configuring a Hadoop cluster (Linux VMs)	13:50
  	Lecture 35	Getting started with Amazon Web Servicies	06:25
  	Lecture 36	Start a Hadoop Cluster with Cloudera Manager on AWS	13:04
  Section 9: MapReduce Customizations For Finer Grained Control
  	Lecture 37	Setting up your MapReduce to accept command line arguments	13:47
  	Lecture 38	The Tool, ToolRunner and GenericOptionsParser	12:35
  	Lecture 39	Configuring properties of the Job object	10:41
  	Lecture 40	Customizing the Partitioner, Sort Comparator, and Group Comparator	15:16
  Section 10: The Inverted Index, Custom Data Types for Keys, Bigram Counts and Unit Tests!
  	Lecture 41	The heart of search engines - The Inverted Index	14:40
  	Lecture 42	Generating the inverted index using MapReduce	10:25
  	Lecture 43	Custom data types for keys - The Writable Interface	10:23
  	Lecture 44	Represent a Bigram using a WritableComparable	13:13
  	Lecture 45	MapReduce to count the Bigrams in input text	08:26
  	Lecture 46	Test your MapReduce job using MRUnit	13:41
  Section 11: Input and Output Formats and Customized Partitioning
  	Lecture 47	Introducing the File Input Format	12:48
  	Lecture 48	Text And Sequence File Formats	10:21
  	Lecture 49	Data partitioning using a custom partitioner	07:11
  	Lecture 50	Make the custom partitioner real in code	10:25
  	Lecture 51	Total Order Partitioning	10:10
  	Lecture 52	Input Sampling, Distribution, Partitioning and configuring these	09:04
  	Lecture 53	Secondary Sort	14:34
  Section 12: Recommendation Systems using Collaborative Filtering
  	Lecture 54	Introduction to Collaborative Filtering	07:25
  	Lecture 55	Friend recommendations using chained MR jobs	17:15
  	Lecture 56	Get common friends for every pair of users - the first MapReduce	14:50
  	Lecture 57	Top 10 friend recommendation for every user - the second MapReduce	13:46
  Section 13: Hadoop as a Database
  	Lecture 58	Structured data in Hadoop	14:08
  	Lecture 59	Running an SQL Select with MapReduce	15:31
  	Lecture 60	Running an SQL Group By with MapReduce	14:02
  	Lecture 61	A MapReduce Join - The Map Side	14:19
  	Lecture 62	A MapReduce Join - The Reduce Side	13:07
  	Lecture 63	A MapReduce Join - Sorting and Partitioning	08:49
  	Lecture 64	A MapReduce Join - Putting it all together	13:46
  Section 14: K-Means Clustering
  	Lecture 65	What is K-Means Clustering?	14:04
  	Lecture 66	A MapReduce job for K-Means Clustering	16:33
  	Lecture 67	K-Means Clustering - Measuring the distance between points	13:52
  	Lecture 68	K-Means Clustering - Custom Writables for Input/Output	08:26
  	Lecture 69	K-Means Clustering - Configuring the Job	10:49
  	Lecture 70	K-Means Clustering - The Mapper and Reducer	11:23
  	Lecture 71	K-Means Clustering : The Iterative MapReduce Job	03:39

What included in this Course?

• Lot's of cool stuff ..

  • Using MapReduce to 

     

    • Recommend friends in a Social Networking site: Generate Top 10 friend recommendations using a Collaborative filtering algorithm. 
    • Build an Inverted Index for Search Engines: Use MapReduce to parallelize the humongous task of building an inverted index for a search engine. 
    • Generate Bigrams from text: Generate bigrams and compute their frequency distribution in a corpus of text. 

   

  • Build your Hadoop cluster: 

     

    • Install Hadoop in Standalone, Pseudo-Distributed and Fully Distributed modes 
    • Set up a hadoop cluster using Linux VMs.
    • Set up a cloud Hadoop cluster on AWS with Cloudera Manager.
    • Understand HDFS, MapReduce and YARN and their interaction 

   

  • Customize your MapReduce Jobs: 

     

    • Chain multiple MR jobs together
    • Write your own Customized Partitioner
    • Total Sort : Globally sort a large amount of data by sampling input files
    • Secondary sorting 
    • Unit tests with MR Unit
    • Integrate with Python using the Hadoop Streaming API

   

   All the basics: 

  • MapReduce : Mapper, Reducer, Sort/Merge, Partitioning, Shuffle and Sort
  • HDFS & YARN: Namenode, Datanode, Resource manager, Node manager, the anatomy of a MapReduce application, YARN Scheduling, Configuring HDFS and YARN to performance tune your cluster. 
  • Develop advanced MapReduce applications to process BigData
  • Master the art of "thinking parallel" - how to break up a task into Map/Reduce transformations
  • Self-sufficiently set up their own mini-Hadoop cluster whether it's a single node, a physical cluster or in the cloud.
  • Use Hadoop + MapReduce to solve a wide variety of problems : from NLP to Inverted Indices to Recommendations
  • Understand HDFS, MapReduce and YARN and how they interact with each other
  • Understand the basics of performance tuning and managing your own cluster

Who should buy this course?

• • Yep! Analysts who want to leverage the power of HDFS where traditional databases don't cut it anymore
  • Yep! Engineers who want to develop complex distributed computing applications to process lot's of data
  • Yep! Data Scientists who want to add MapReduce to their bag of tricks for processing data
  • You'll need an IDE where you can write Java code or open the source code that's shared. IntelliJ and Eclipse are both great options.
  • You'll need some background in Object-Oriented Programming, preferably in Java. All the source code is in Java and we dive right in without going into Objects, Classes etc
  • A bit of exposure to Linux/Unix shells would be helpful, but it won't be a blocker