Apache Spark DStream is not supported

DStreams are not supported in Databricks. Migrate from DStream API to Structured Streaming.

Written by Adam Pavlacka

Last published at: May 17th, 2022

Problem

You are attempting to use a Spark Discretized Stream (DStream) in a Databricks streaming job, but the job is failing.

Cause

DStreams and the DStream API are not supported by Databricks.

Solution

Instead of using Spark DStream, you should migrate to Structured Streaming.

Review the Databricks Structured Streaming in production (AWS | Azure | GCP) documentation for more information.