Spark cluster vs one machine

The problem

“The pipeline is slow, let’s move it to Spark” is a decision that usually gets made without a number attached to it.

The reasoning usally is: the job is CPU-bound, more machines means more CPUs, therefore more machines means faster. What that skips is that distribution is not free. You pay for JVM startup, for scheduling, for serialising data across a shuffle, and for an operational surface that a single Python process does not have. Those costs are fixed or near-fixed, while the benefit scales with the job. So there is a crossover point, and it is a number you can measure on your own hardware in an afternoon.

Setup

A trip-analytics pipeline over synthetic ride-hailing data, chosen because it contains one of each class of distributed operation: a filter (narrow), derived columns (narrow), a join against a 265-row dimension (broadcast, avoids a shuffle), a group-by (wide, shuffle), and a top-5-per-group window (shuffle plus sort). Read Parquet in, ~1 KB of aggregates out — the funnel shape that makes distribution worth considering at all.

Three implementations of the identical pipeline:

T2 is the rung that makes the comparison honest. T1 to T2 is vertical scaling; T2 to T3 is horizontal. T2 is also a distributed system in miniature — map each partition to a partial aggregate, reduce the partials — which is exactly what T3 does with the process boundary moved across machines.

T3 Spark setup

Benchmark results

Scaling gains plateau

Adding Spark workers does not keep buying scalability. The 4-node speedup goes 0.80x at 10M rows, 2.19x at 30M, 2.24x at 300M. It saturates around 2.2x — 56% efficiency against the host’s own 3.94x ceiling — and 300M rows buys essentially nothing over 30M. The plateau is identical whether or not the data fits in page cache, so it is structural (shuffle and coordination), not disk.

Same code, same total cores, only the job size changed. JVM startup, scheduling and plan construction are a fatal overhead at 10M and nearly amortised by 300M.

4 processes in one machine beat 4 one-process machines

The steady-state coordination tax is 1.55x — 1.04M rows/s/core in one box against 0.67M rows/s/core on the cluster — so on this workload the cluster needs about 6 nodes to break even with the single node with 4 processes.

If your data fits on one machine, put it on one machine — but the penalty for being wrong about that shrinks fast as the job grows.

Increase Spark worker task slots DO NOT increase capacity

Putting N task slots on a single Spark worker, with shuffle partitions held at 16 so slot count is the only variable, at 10M rows:

Going 1 to 10 slots on one woker is monotonically worse, ending 13% slower, with run-to-run ranges tight enough (27.4–28.8s at 10 slots) that it is signal rather than noise. A slot is a thread inside one executor JVM; ten threads on one core timeslice the same silicon while adding context switches, ten sets of task bookkeeping, and ten concurrent shuffle writers competing for one 1400 MB heap.

Implementation

Python, pandas, PyArrow, Apache Spark 3.5, Docker Compose, matplotlib. No cloud account, one laptop, nodes pinned to individual cores in the compose file.

Source code

— Sep 6, 2026