Error running parameterized SQL queries in Databricks Connect with VS Code

Pass the SQL parameter in Databricks Connect using string interpolation.

Written by manoj.hegde

Last published at: November 22nd, 2024

Problem

When running parameterized SQL queries from an interactive session in Databricks Connect with Visual Studio Code (VS Code), you receive an error message despite running the same queries successfully from a notebook. 

Example

spark.sql("SELECT * FROM range(10) WHERE id > {bound1} AND id < {bound2}", bound1=7, bound2=9).show()

 

TypeError: SparkSession.sql() got an unexpected keyword argument '<variable>'

 

Cause

The SparkSession.sql() method in Databricks Connect does not support passing parameters directly as keyword arguments. 

 

Solution

Use string interpolation to pass parameters into the SQL query. 

 

  1. Define the parameters separately. 
bound1 = 7
bound2 = 9

 

  1. Use an f-string to interpolate the parameters into the SQL query. 
spark.sql(f"SELECT * FROM range(10) WHERE id > {bound1} AND id < {bound2}").show()