How it works
A Spark application has one driver and multiple executor processes. The driver builds execution plans, schedules work, and tracks progress. Executors run tasks and hold cached data and shuffle files. A cluster manager, such as Kubernetes, YARN, or Spark Standalone, allocates resources. The driver decides which application tasks to run.
Spark evaluates transformations lazily. An action, such as a dataset write or result collection, starts execution. For SQL and DataFrames, the Catalyst optimizer improves a logical plan and selects a physical plan. Spark also improves execution at runtime, with features such as code generation where supported. Adaptive Query Execution can change parts of the SQL plan with runtime statistics.
Spark divides work into jobs, stages, and tasks. Spark combines operations that process partitions independently within a stage. Operations that redistribute data create shuffle boundaries between stages. Examples include many joins, groups, and global sorts. Within a stage, a task generally processes one partition.
A shuffle redistributes records across executors by keys or ranges. It uses network bandwidth and local disk. Sorts or aggregation can spill data to disk when memory is insufficient. Spark can cache datasets for reuse in memory or on disk, but this is optional. Spark does not require the full dataset to fit in memory.
Structured Streaming runs incremental queries with the Spark SQL engine, normally as micro-batches. Stateful operators retain information across batches, such as per-key aggregates or join state. Durable checkpoints record progress and state for recovery. End-to-end correctness also requires replayable sources and a compatible sink.
What it gives you in a design
- Distributed SQL and DataFrames
- Express filters, joins, aggregations, and transformations without manual schedules for distributed tasks. Built-in expressions give the optimizer more information than opaque user-defined functions.
- Large joins and aggregations
- Spark supports shuffle joins, broadcast joins, and other strategies. A broadcast of a truly small input can avoid a shuffle of a large input. The broadcast data must fit safely in executor memory.
- Adaptive Query Execution
- For supported SQL queries, Spark can combine shuffle partitions, change join strategies, and split skewed join partitions with runtime statistics. This helps when estimates are inaccurate. You must still understand data distribution.
- Reusable intermediate datasets
- A stored intermediate result that costs much to compute can avoid repeated scans and computation across actions. Choose a storage level carefully. Release cached data when you no longer need it.
- Stateful stream processing
- Structured Streaming supports event-time windows, watermarks, aggregations, and supported stream joins. The design must limit retained state. It must also define what happens to events that arrive too late.
- Storage and format integration
- Spark uses suitable connectors to read and write systems such as object storage, distributed filesystems, and Kafka. Columnar formats such as Parquet support column pruning and predicate pushdown. The table format and catalog provide transactional table behavior, not Spark alone.
Scaling, replication and failure
Partitions determine parallelism, not just the executor count. Too few partitions leave cores idle and create large tasks. Too many partitions add schedule overhead and small outputs. Input partitioning, shuffle partitioning, and the output file layout are separate choices. Explain each choice instead of one partition count for the full pipeline.
Skew often determines completion time. A popular key can send an unequal share of work to one task while other executors finish early. Depending on the operation, use partial aggregation, key salting with a later merge, broadcast joins, or adaptive skew handling. More workers alone cannot divide an indivisible task for a hot key.
Spark recovers from many executor failures through task retries and recomputation of lost partitions from lineage. Cached data is not a durable source of truth. Lost shuffle files can require Spark to repeat earlier work. Driver failure usually requires an external restart mechanism. A Structured Streaming query can resume from a compatible durable checkpoint.
The source, storage layer, and sink primarily provide replication and transaction guarantees. Task retries or speculative tasks can repeat external side effects. Thus, arbitrary database calls inside tasks are not exactly-once. Use supported commit protocols, transactional table writes, or idempotent sink operations. Plan spare capacity so stream lag can decrease after an outage.
Limits and defaults to know
Many of these values are configurable. Check the current documentation before you use an exact value.
| SQL shuffle partitions | spark.sql.shuffle.partitions defaults to 200 for SQL joins and aggregations. You can change it. Adaptive Query Execution can combine partitions after a shuffle for supported queries. |
|---|---|
| Automatic broadcast join threshold | spark.sql.autoBroadcastJoinThreshold defaults to 10 MiB. You can change it. It depends on available size statistics. It does not guarantee that broadcast data fits safely in memory. |
| File scan partition size | spark.sql.files.maxPartitionBytes defaults to 128 MiB for supported file-based sources. It guides plans for input partitions, not the output file size. Actual splits depend on file formats and other settings. |
| Task failure tolerance | spark.task.maxFailures defaults to 4 continuous failures of a specific task before Spark stops the job. You can change it. More retries do not correct deterministic failures or unsafe side effects. |
| Default stream trigger | In default micro-batch mode, Structured Streaming starts each batch as soon as the previous batch completes, unless you specify a trigger interval. This gives no fixed end-to-end latency guarantee. |
Use it when
- The design needs repeated ETL or backfills over large datasets in object storage or a data lake.
- Large joins, sorts, or aggregations exceed the data that one machine can practically process at once.
- The same team needs SQL-based batch processes and stream transformations that prioritize throughput.
- Historical recomputation is important to correct analytics, rebuild aggregates, or produce datasets to train models.
- The system can publish results to a separate database or index for application reads with low latency.
Avoid it when
- The component must serve individual API requests with strict tail-latency requirements or make transactional record updates.
- The dataset fits easily in one database or process. A distributed task schedule would add unnecessary complexity.
- The main requirement is consistently very low latency for event processes. Micro-batch schedules or large state checkpoints would use too much of the allowed time.
- The job mostly sends independent external API calls with strict limits per call and side-effect controls. A durable worker queue is usually simpler.
What interviewers ask about it
When Spark is on your whiteboard, expect these follow-up questions. Each answer gives the core of a strong answer.
- Where does Spark fit in the system, and what serves the results?
- Place Spark between durable inputs and derived outputs. Inputs can include Kafka or object storage. Outputs can include an analytics table or a database that serves data. Keep interactive requests outside the Spark execution path unless a separate query design meets the latency requirement.
- Why is this job slow even with more executors?
- Inspect the execution plan and stage metrics for skew, too few partitions, shuffle volume, spills, and slow input reads. If one task dominates a stage, more executors will not help until the partitioning or algorithm changes.
- How do you prevent a large join from using all memory?
- Select required input columns and filter inputs early. Verify statistics. Do not broadcast an input only because its compressed file size seems small. Choose a suitable join strategy and allow enough shuffle parallelism. Account for skew and multiple tasks that run at the same time and share executor memory.
- Does Structured Streaming guarantee exactly-once results?
- Spark can provide exactly-once processing with replayable sources, durable checkpoints, deterministic computation, and a sink with the required commit semantics. These conditions do not make arbitrary external writes exactly-once. foreachBatch uses at-least-once processing by default. Use its batch identifier or another stable key to make writes idempotent.
- How do watermarks affect late events and state size?
- A watermark tracks event-time progress with an allowed lateness delay. It lets supported operators eventually discard old state. Spark does not drop events within the configured delay as too late. Spark may or may not process older events. The operator and output mode determine exact state eviction and output behavior.
- What happens when a stream query cannot match the input rate?
- Monitor source lag, batch duration, state size, and sink throughput. Identify whether compute, skew, input reads, or output writes limit performance. Source-specific admission controls can limit input per batch, but they do not add capacity. Plan retention and enough spare throughput to process delayed data.
Common mistakes
- The claim that Spark is an in-memory database or requires all data to fit in RAM is incorrect. This compute engine can scan external storage, spill to disk, and recompute partitions.
- The claim that more executors always make a job faster ignores partition count, skew, shuffle costs, and source or sink bottlenecks.
- The claim that checkpoints make all external writes exactly-once is incorrect. Recovery can replay work. Arbitrary side effects still need idempotency or transaction coordination.
- A large dataset collection on the driver or conversion to a local dataframe without a size limit is a mistake. Distributed executor capacity does not protect the driver's memory.
Alternatives
- Apache Flink
- Prefer it for continuous, low-latency stateful event processes with detailed event-time and state control needs. Compare operational complexity and connector guarantees. Do not choose only by latency labels.
- Cloud data warehouse
- Prefer a managed warehouse for mainly SQL analytics when fewer cluster operations, concurrent users, and warehouse-managed tables matter more than custom distributed computation.
- Trino
- Prefer it for interactive distributed SQL over data lakes and federated sources when queries matter more than general-purpose data pipelines.
- Single-node SQL or dataframe engine
- Prefer tools such as DuckDB or Polars when one machine can process the workload in parallel within the required time. This avoids distributed task schedules, shuffle, and cluster operations.
Where it is used
Spark is a good fit for these interview questions. Each line tells you why.
- Design a system to sort huge datasetsSpark illustrates range partitioning, shuffle, external sorting, skew handling, and recomputation after executor failures.
- Design an ad click aggregatorStructured Streaming can build event-time aggregates. Batch jobs reconcile live counts with durable event history.
- Design Google AnalyticsSpark suits large event transformations, sessionization, historical backfills, and partitioned analytics tables.
- Design a top K trending systemSpark can compute popularity aggregates for time windows and merge candidates from each partition. It then publishes rankings to a store that serves requests.
- Design an A/B testing platformSpark supports large exposure-to-outcome joins and repeatable experiment analysis over historical events.
- Design a metrics monitoring system like DatadogSpark can produce rollups and historical recomputations, with explicit trade-offs for aggregation state and alert latency.
Common questions
What are the main components of Apache Spark architecture?
The main runtime components are the driver, executors, and cluster manager. The driver plans and schedules jobs. Executors process tasks for each partition. The cluster manager allocates resources. Storage systems remain separate from these compute components.
What is the difference between a job, stage, and task in Spark?
An action starts execution that can involve one or more jobs. Spark divides each job into stages, generally at shuffle boundaries. A stage runs tasks in parallel. Each task generally processes one partition.
What is the difference between Apache Spark and Hadoop?
Spark is a distributed processing engine. Hadoop commonly refers to an ecosystem that includes HDFS, YARN, and MapReduce. Unlike MapReduce, Spark can pipeline operations within stages and reuse cached intermediate data across computations. Spark can run on YARN and read HDFS, but it does not require either.