Apache Spark PySpark job using a Python threading API function taking hours instead of minutes

Use the Databricks Spark connector and ensure your cluster configuration is optimized for the workload.

Written by John Benninghoff

Last published at: January 10th, 2025

Problem

Your Apache Spark PySpark job using the following Python threading API function ThreadPoolExecutor() takes over an hour to complete, instead of minutes.

 

with ThreadPoolExecutor(max_workers=MAX_THREAD_NUM) as executor:
    executor.map(thread_process_partition, cid_partitions)

 

Cause

When using Python threads, the driver node becomes overwhelmed, leading to inefficient task distribution and underutilization of worker nodes. 

 

Solution

  • Use the Databricks Spark connector instead of threading. For more information, review the Connect to external systems (AWS | Azure | GCP) documentation. 
  • Ensure that the cluster configuration is optimized for the workload. This includes adjusting the number of worker nodes and their specifications to match the job requirements. For cluster sizing guidance, review the Compute configuration recommendations (AWS | Azure | GCP) documentation.

 

Preventative measures

  • Implement best practices for Spark job optimization, such as caching intermediate results, using broadcast variables, and avoiding shuffles where possible. For more information, review the Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads.
  • Monitor and adjust the job's execution plan, using the Spark UI to identify and address any bottlenecks. Refer to the Debugging with the Apache Spark UI (AWS | Azure | GCP) documentation for details.