Problem

When using a cluster with Apache Spark Connect to run code that invokes temporary views in a loop, the row value assignments do not reflect the expected output. 

Example

In code with a RANGE of 2, you expect row one to have the value 0, and row two to have the value 1. Instead, both rows have the value 1. 

RANGE = 2
df_temp_view = None
for i in range(RANGE):
df = spark.sql(f"select {i} as iterator")
df.createOrReplaceTempView("temp_view")
if df_temp_view is None:
df_temp_view = spark.sql("select * from temp_view")
else:
df_temp_view = df_temp_view.union(spark.sql("select * from temp_view"))

df_temp_view.display()

Cause

Temporary views in Spark Connect are analyzed lazily. This means any changes to the temporary view are not validated until the view is called, including filters and transformations. 

Because the temporary view is recreated on each iteration, at the moment the Spark action is called Spark analyzes the latest version of the view, producing two rows with the same value. 

Solution

Use DataFrames directly instead of temporary views. For more information, refer to the Tutorial: Load and transform data using Apache Spark DataFrames (AWS | Azure | GCP) documentation.

If you prefer to continue using temporary views in this context, apply unique names to each temporary view.