MapReduce
Outline
Why MapReduce?
An Example Program
Map? Reduce?
Outline
MapReduce vs Hadoop
GFS/HDFS
Outline
Major Components
Key Notes
Basic Concepts
Data Flow
Input Splitter
Mapper
Reducer
Combiner
Output Committer
Partitioner (Shuffler)
Master
Master Cont.
Writables
Outline
Mapper Code
Reducer Code
Combiner Code
That’s it!
Workflows
Handling Data By Type
Handling Data By Type

Master

Responsible for scheduling & managing jobs Scheduled computation should be close to the data if possible Bandwidth is expensive! (and slow) This relies on a Distributed File System (GFS / HDFS)! If a task fails to report progress (such as reading input, writing output, etc), crashes, the machine goes down, etc, it is assumed to be stuck, and is killed, and the step is re-launched (with the same input) The Master is handled by the framework, no user code is necessary