Error when attempting to use 'stream_read_table' function from Sparklyr in 14.3 LTS ML

Install Sparklyr version 1.8.6 in the cluster's libraries, restart the cluster, and rerun the function to ensure the error is fixed.

Written by Amruth Ashoka

Last published at: January 7th, 2025

Problem

When attempting to use the stream_read_table function from the Sparklyr package version 1.8.6 in Databricks Runtime 14.3 LTS ML, you receive an error message. 

Error in stream_read_table(sc, 'table_name'): could not find function 'stream_read_table'.

 

Cause

Databricks Runtime 14.3 LTS ML includes Sparklyr version 1.8.1 by default, but the stream_read_table function requires version 1.8.6.

 

Solution

  1. Install Sparklyr version 1.8.6. Navigate to the cluster's library settings and add the Sparklyr package. This action will replace the default version, 1.8.1.
  2. Restart the cluster to implement the updated changes.
  3. Re-execute the stream_read_table function to verify the error is resolved.

Alternatively, update Databricks Runtime to 16.0 or above, which uses Sparklyr 1.8.6 by default. 

 

Important

Generally, changing package versions can potentially cause conflicts with other internal dependency changes. In this solution, Sparklyr 1.8.6 is a patch update, so no conflicts are anticipated. However as a precaution, you can test the version change in a development environment before deploying to production.