Running Apache Spark experimen... Note

Running Apache Spark experiments in my sleep (and on a plane)

Netflix's member-sessionizer, a critical data pipeline, processes billions of events to create user viewing sessions. Standard debugging approaches failed at Netflix's production scale, as issues like Out-Of-Memory errors only appeared with large, unsampled datasets due to data skew. The author's breakthrough was shifting focus from speeding up single runs to enabling multiple concurrent experiments. This was achieved by leveraging Dataflow's branch-driven development to isolate each test variant.By parameterizing the scheduler and assigning each experiment its own branch and output table, multiple tests could run simultaneously. This drastically reduced total execution time from hours per test to the duration of the longest single job. An AI agent was employed to manage the operational complexity of running many parallel experiments, automating configuration updates, job triggering, and metric polling. This freed the author from tedious manual checks, allowing for immediate comparison of results.Persistent tmux sessions on Netflix's cloud development environment proved crucial for preventing multi-hour Spark jobs from being interrupted by network drops or laptop closures. The core performance bottlenecks were identified as the collect_list operator buffering entire event bags in memory and inefficient data structures. Specifically, changing session IDs to UUID strings and using Scala Options for fields significantly increased memory usage. Additionally, bloated string attributes added unnecessary weight to each event.The fixes involved unboxing primitive fields from Options and slimming down event bags by removing unused attributes. Session-invariant metadata was hoisted to the session header. Two counterintuitive findings highlighted the complexities of Spark memory management: narrowing rows could trigger faster OOMs by forcing broadcast joins, and Adaptive Query Execution silently accumulated large amounts of data in memory through broadcast hash joins. Tuning AQE's broadcast threshold resolved this.A disciplined logging system tracked each experiment, its configuration, and key metrics, creating a single source of truth and forcing isolated variable changes. A mistake was made by testing a session capping fix before the Spark Window stage, which buffered memory before truncation. The final corrected fix moved the truncation upstream to prevent memory bloat. While these tuning efforts solved immediate problems, the fundamental data shape of aggregating entire sessions into single rows poses a future scaling challenge.
CdXz5zHNQW_imDTAsvHQS.png