Pig For Wrangling Big Data

About Course

• Prerequisites: Working with Pig requires some basic knowledge of the SQL query language, a brief understanding of the Hadoop eco-system and MapReduce 

  Taught by a team which includes 2 Stanford-educated, ex-Googlers  and 2 ex-Flipkart Lead Analysts. This team has decades of practical experience in working with large-scale data processing jobs. 

  Pig is aptly named, it is omnivorous, will consume any data that you throw at it and bring home the bacon!

  omnivorous: Pig works with unstructured data. It has many operations which are very SQL-like but Pig can perform these operations on data sets which have no fixed schema. Pig is great at wrestling data into a form which is clean and can be stored in a data warehouse for reporting and analysis.

  bring home the bacon: Pig allows you to transform data in a way that makes is structured, predictable and useful, ready for consumption.

Topics covered in this course

• Section 1: You, This Course and Us
  	Lecture 1	You, This Course and Us Preview	01:46
  Section 2: Where does Pig fit in?
  	Lecture 2	Pig and the Hadoop ecosystem Preview	09:37
  	Lecture 3	Install and set up	08:50
  	Lecture 4	How does Pig compare with Hive?	10:15
  	Lecture 5	Pig Latin as a data flow language	06:17
  	Lecture 6	Pig with HBase	05:18
  Section 3: Pig Basics
  	Lecture 7	Operating modes, running a Pig script, the Grunt shell Preview	09:52
  	Lecture 8	Loading data and creating our first relation	08:45
  	Lecture 9	Scalar data types Preview	09:55
  	Lecture 10	Complex data types - The Tuple, Bag and Map	13:45
  	Lecture 11	Partial schema specification for relations	10:00
  	Lecture 12	Displaying and storing relations - The dump and store commands	03:54
  Section 4: Pig Operations And Data Transformations
  	Lecture 13	Selecting fields from a relation Preview	10:22
  	Lecture 14	Built-in functions	05:08
  	Lecture 15	Evaluation functions	10:31
  	Lecture 16	Using the distinct, limit and order by keywords	05:04
  	Lecture 17	Filtering records based on a predicate	11:01
  Section 5: Advanced Data Transformations
  	Lecture 18	Group by and aggregate transformations	12:12
  	Lecture 19	Combining datasets using Join	16:19
  	Lecture 20	Concatenating datasets using Union Preview	04:32
  	Lecture 21	Generating multiple records by flattening complex fields	05:24
  	Lecture 22	Using Co-Group, Semi-Join and Sampling records	09:26
  	Lecture 23	The nested Foreach command	13:47
  	Lecture 24	Debug Pig scripts using Explain and Illustrate	12:55
  Section 6: Optimizing Data Transformations
  	Lecture 25	Parallelize operations using the Parallel keyword Preview	08:02
  	Lecture 26	Join Optimizations: Multiple relations join, large and small relation join	10:34
  	Lecture 27	Join Optimizations: Skew join and sort-merge join	08:51
  	Lecture 28	Common sense optimizations	05:24
  Section 7: A real-world example
  	Lecture 29	Parsing server logs	07:55
  	Lecture 30	Summarizing error logs	08:47
  Section 8: Installing Hadoop in a Local Environment
  	Lecture 31	Hadoop Install Modes	08:32
  	Lecture 32	Setup a Virtual Linux Instance (For Windows users)	15:31
  	Lecture 33	Hadoop Standalone mode Install	09:33
  	Lecture 34	Hadoop Pseudo-Distributed mode Install	14:25

What you get from this course?

• Pig Basics: Scalar and Complex data types (Bags, Maps, Tuples), basic transformations such as Filter, Foreach, Load, Dump, Store, Distinct, Limit, Order by and other built-in functions.

  Advanced Data Transformations and Optimizations: The mind-bending Nested Foreach, Joins and their optimizations using "parallel", "merge", "replicated" and other keywords, Co-groups and Semi-joins, debugging using Explain and Illustrate commands

  Real-world example: Clean up server logs using Pig

  Work with unstructured data to extract information, transform it and store it in a usable form

  Write intermediate level Pig scripts to munge data

  Optimize Pig operations which work on large data sets

Who should buy this course?

• • A basic understanding of SQL and working with data
  • A basic understanding of the Hadoop eco-system and MapReduce tasks
  • Analysts who want to wrangle large, unstructured data into shape
  • Engineers who want to parse and extract useful information from large datasets