Filter condition in the for each task type not filtering correctly

Use param = :Param instead.

Written by nikhil.jain

Last published at: January 20th, 2025

Problem

When you pass param = "${param}" where ${param} is meant to dynamically reference values from job parameters, you notice the filter condition in the for each task type does not filter records as expected. 

 

Cause

The ${param} syntax doesn’t correctly parse or apply the filter condition within the for each  task type, resulting in an empty result set. This is a known limitation. 

 

Solution

In Databricks Runtime 15.2 or above, or in SQL warehouse, use the :Param syntax in your filter condition instead. For example, param = :Param. 

 

Example

The following is a modified query using the :Param syntax. The accompanying screenshot demonstrates the modified query with place = :Place. 

 

select * from <table-name> where <for-each-task-type-param> = :<for-each-task-type-param>

 

 

For more information, review the Work with query parameters (AWS | Azure | GCP) documentation.