How it works
An input format divides a dataset into logical splits and converts each split into records for a map task. Splits define units of work, not necessarily files or storage blocks. With distributed filesystems, a scheduler can place computation near the data. With object storage, workers generally read over the network.
Each mapper emits intermediate key-value pairs. A partitioner selects a reducer partition for each key, usually with a hash. The system buffers and sorts map output, then spills it to local disk as necessary. Thus, the entire dataset does not need to fit in memory.
During the shuffle, reducers fetch their partitions from map tasks and merge the sorted data. The framework groups values by key before it calls the reduce function. A combiner can safely aggregate local data to reduce shuffle traffic. Correctness must not depend on whether the combiner runs.
Reducers process each key and its values, then write output through an output format and commit mechanism. Reducers can fetch completed map output while other maps still run. Final reduction requires the relevant map outputs. Computations with multiple stages usually require multiple jobs. These jobs write intermediate data between stages.
A coordinator tracks task attempts and schedules retries when workers fail. Classic Hadoop MapReduce usually stores intermediate map output on each worker's local disk. If a worker fails, the system may need to repeat its completed maps. Hadoop MapReduce on YARN uses an ApplicationMaster to coordinate the job. YARN allocates containers.
What it gives you in a design
- Distributed aggregation
- Groups of values with the same key simplify counts, sums, grouped statistics, and similar batch computations. Define the intermediate key carefully. It determines both correctness and work distribution.
- Disk-backed processing
- The system sorts, spills, and merges data so jobs can process more data than worker memory can hold. The trade-off is substantial disk and network I/O, especially across multiple stages.
- Combiners
- Local partial aggregation can reduce shuffle volume when the operation supports safe repeated combination. For an average, combine sum-and-count pairs instead of partial averages.
- Custom partitioning and sorting
- Partitioners control which reducer receives a key. Sort and group rules control how records reach reduce calls. Range partitioning supports a global sort. Composite keys support secondary-sort patterns.
- Task retries and speculative execution
- The system can repeat failed attempts. Speculative execution can start duplicate attempts for slow tasks. This helps with slow workers. It does not help a reducer whose input is disproportionately large.
- Replayable batch computation
- The system can recompute retained input after a logic change or use it to rebuild derived datasets. Reproducibility still requires versioned inputs, deterministic logic, and controlled external dependencies.
Scaling, replication and failure
Map parallelism largely follows the number of input splits. The configuration sets reducer parallelism separately. More reducers can distribute work. They also increase shuffle connections, scheduler overhead, and output file counts. Estimate input bytes, intermediate expansion, shuffle bytes, and the largest partition before you choose capacity.
Partitioning balances keys, but not necessarily bytes or computation cost. Ordinary hash partitioning still sends a hot key to one reducer. For decomposable aggregates, salt the key, compute partial results, and run a second aggregation. For a sort, sample the key distribution and choose range boundaries carefully.
The underlying filesystem or object store usually provides durability. The system does not usually replicate every task's intermediate state. The system can recompute lost intermediate output from durable input. Storage replication, object-store guarantees, and coordinator recovery behavior depend on the deployment.
Task retries do not guarantee exactly-once external side effects in general. Use the framework's supported output committer for file output. Use idempotent writes or deduplication for external sinks. Publish a completed dataset version through a manifest or metadata pointer. This prevents readers from seeing a batch before the system completes it.
Limits and defaults to know
Many of these values are configurable. Check the current documentation before you use an exact value.
| Default reducer count | Hadoop MapReduce documents 1 as the default for mapreduce.job.reduces. Configure it for the workload. Map-only jobs can use 0 reducers. |
|---|---|
| Task attempt defaults | Hadoop MapReduce sets mapreduce.map.maxattempts and mapreduce.reduce.maxattempts to 4 attempts each by default. You can configure both. |
| Combiner execution guarantee | A combiner can run zero, one, or multiple times. The result must remain correct in all cases. |
| Order guarantee | The system sorts reducer input by key within each partition. Ordinary hash partitioning does not put multiple reducer outputs in global order. The system does not automatically sort records again after reducers emit them. |
| Single-key parallelism | Under normal group rules, one reduce call processes all values for one key. More reducers alone do not divide that key's work. |
Use it when
- The design needs large historical aggregations or periodic reports. The time to complete a batch is acceptable.
- You need to rebuild indexes or derived datasets from durable raw input.
- The computation naturally uses independent record transformations, then groups records by key.
- You need a distributed external sort that limits worker memory use.
- An existing Hadoop environment makes operational reuse more valuable than a newer engine.
Avoid it when
- The result must update continuously with low latency, as with live alerts or real-time counters.
- The workload repeatedly reuses the same working set across iterative steps. Each step writes data to disk at a high cost.
- You need interactive queries or synchronous computation on the request path.
- The job mainly produces non-idempotent external side effects that cannot safely tolerate retries or speculative attempts.
What interviewers ask about it
When MapReduce is on your whiteboard, expect these follow-up questions. Each answer gives the core of a strong answer.
- What will probably cause most of a MapReduce job's cost?
- Check input scans, intermediate data growth, shuffle traffic, disk spills, and the slowest reducer. Filter early and use combiners where valid. Estimate intermediate bytes. Do not assume that input size alone predicts cost.
- How would you handle one extremely popular key?
- More reducers do not help because the key still belongs to one reduce group. If you can divide the computation, distribute it across salted keys. Merge partial results in a second stage. Otherwise, discuss a specialized algorithm or the bottleneck that you cannot avoid.
- What happens when a worker fails?
- The framework retries affected tasks within the configured attempt limits. Maps may need to run again if a worker held local map outputs that reducers still need. The configured recovery and commit mechanisms handle durable output that tasks successfully committed.
- Do task retries guarantee exactly-once output?
- No: a task's code can execute more than once. Supported file committers can select successful task output. Arbitrary database writes or API calls need their own idempotency or deduplication strategy.
- How do you produce a dataset with a global sort order?
- Sample the input to select ordered range boundaries. Partition by those ranges and sort within each partition. Ensure that the reducer preserves key order and readers read files in range order. Hash partitioning orders data only within each partition.
- When is a combiner correct?
- It must preserve the final answer if it runs repeatedly or not at all. It usually uses associative and commutative aggregation with compatible intermediate types. Sum and count work directly. Averages require both sum and count, not just a partial average.
Common mistakes
- You place MapReduce on a user request path but ignore scheduler delays, shuffle delays, and the time to complete a batch.
- You assume more reducers solve hot keys or guarantee partitions of equal size.
- You assume combiners always run or use mathematically invalid partial aggregation.
- You claim retries guarantee exactly-once behavior for arbitrary side effects. Or you assume output files become visible as one atomic dataset.
Alternatives
- Apache Spark
- Choose it for batch pipelines with multiple stages, iterative algorithms, and workloads that benefit from data reuse in memory. You must still explicitly handle skew, shuffle cost, and sink semantics.
- Apache Flink
- Choose it for continuous, stateful computation with event-time windows and low-latency updates, not periodic jobs over the full dataset.
- Distributed SQL engines
- Choose engines such as Trino or a cloud data warehouse for SQL-oriented analysis. An optimizer and declarative queries reduce custom code.
Where it is used
MapReduce is a good fit for these interview questions. Each line tells you why.
- Design a system to sort huge datasetsMapReduce directly illustrates range partitioning, external sorts, shuffle-and-merge computation, and recovery after worker failure.
- Design an ad click aggregatorBatch jobs can recompute historical click totals. They can also reconcile live aggregates against durable input without duplicates.
- Design Google AnalyticsOffline aggregation over retained events supports historical reports and backfills. It keeps batch computation off the collection path.
- Design Facebook post searchMapReduce can build a baseline inverted index from a large corpus. A separate incremental pipeline then keeps the index current.
- Design an A/B testing platformBatch jobs can join exposure and outcome data. They can recompute experiment results after corrections to attribution or data quality.
Common questions
What is MapReduce?
MapReduce is a model that processes large datasets across workers. Map functions emit key-value pairs. The framework groups intermediate values by key. Reduce functions produce results from those groups.
What is the difference between MapReduce and Hadoop?
MapReduce is a model for programs and their execution. Hadoop is an ecosystem that includes an implementation of that model. Hadoop MapReduce commonly runs on YARN and uses HDFS or another supported storage system.
What is the difference between MapReduce and Spark?
Classic Hadoop MapReduce processes data through map and reduce jobs. It often writes data between jobs. Spark supports more general execution graphs and can cache reused datasets in memory. Thus, Spark better suits many workloads with repeated steps or multiple stages. Both can spill to disk. Both can have costly shuffles and skew.