Append output is not supported without a watermark

Append output mode is not supported on aggregated DataFrames without a watermark....

Apache Spark DStream is not supported

DStreams are not supported in Databricks. Migrate from DStream API to Structured Streaming....

Streaming with File Sink: Problems with recovery if you change checkpoint or output directories

Learn how to resolve issues that occur with recovery if you change checkpoint or output directories when streaming with File Sink....

Get the path of files consumed by Auto Loader

Get the path and filename of all files consumed by Auto Loader and write them out as a new column....

How to set up Apache Kafka on Databricks

Learn how to set up Apache Kafka on Databricks....

Handling partition column values while using an SQS queue as a streaming source

...

How to restart a structured streaming query from last written offset

Learn how to restart a structured streaming query from the last written offset....

How to switch a SNS streaming job to a new SQS queue

...

Kafka error: No resolvable bootstrap urls

A 'No resolvable bootstrap urls' error occurs when you try to read or write data to a Kafka stream....

readStream() is not whitelisted error when running a query

readStream() is not whitelisted error on clusters that have table access control enabled....

Checkpoint files not being deleted when using display()

Learn how to prevent display(streamingDF) checkpoint files from using a large amount of storage....

Checkpoint files not being deleted when using foreachBatch()

Learn how to prevent foreachBatch() checkpoint files from using a large amount of storage....

Conflicting directory structures error

You should use distinct paths in the storage location, otherwise conflicting directory structures may result in an error....

RocksDB fails to acquire a lock

When using RocksDB as a state store, you may need to increase the acquire timeout in the SQL config....

Stream XML files using an auto-loader

Stream XML files on Databricks by combining the auto-loading features of the Spark batch API with the OSS library Spark-XML....

Streaming job using Kinesis connector fails

A streaming job writing to a Kinesis sink fails with out of memory error because HTTP clients are not getting terminated....

Streaming job gets stuck writing to checkpoint

Streaming job appears to be stuck even though no error is thrown. You are using DBFS for checkpoint storage, but it has filled up....

Explicit path to data or a defined schema required for Auto loader

If you do not specify an explicit path to your data or define your data schema, you get an IllegalArgumentException error when you start an Auto loader job....

Optimize streaming transactions with .trigger

Use .trigger to define the storage update interval. A higher value reduces the number of storage transactions....

Structured streaming jobs slow down on every 10th batch

Automatic compaction of the metadata folder can slow down structured streaming jobs....

Get last modification time for all files in Auto Loader and batch jobs

Define a UDF to list all files in the path and return the last modification time for each one....

Stream to stream join failure

Avoid using a memory sink when running streaming queries with stream to stream join....

Offset reprocessing issues in streaming queries with a Kafka source

Resolve Kafka offset reprocessing issues in Structured Streaming by using a new checkpoint directory....

Autoloader job fails with a URISyntaxException error due to invalid characters in filenames

When using Directory listing mode you should not process files with colons in the filename. ...

Auto Loader streaming job failure with schema inference error

To selectively read a specific type of file using Auto Loader, use the pathGlobFilter option....

Auto Loader streaming query failure with unknownFieldException error

Use schema evolution to avoid streaming query failures when new columns are added to your data....

Casting string to date/timestamp in DLT pipeline does not throw an error

Configure the Delta Live Tables pipeline to enforce ANSI SQL compliance. ...

Incorrect numInputRows values even with row limits set per batch

Use endOffset instead of latestOffset to calculate the number of records read. ...

Auto Loader fails to pick up new files when using directory listing mode

Use file notification mode or disable incremental listing....

Incorrect input record count in Apache Spark streaming application logs/micro-batch metrics

Optimize actions on the DataFrame within the foreachBatch function. ...

Upgrading to 14.3 LTS gives the error "com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException"

Choose to configure either manually or via schema evolution....

Streaming application missing data from a Delta table when writing to a given destination

Restart a streaming query on a new checkpoint folder with startingVersion option pointing to the next Delta version (X+1)....

DatabricksS3LoggingException errors while attempting to read a binary file in an Apache Spark structured streaming job

Disable the delta format check and ensure correct IAM role setup. ...

How to efficiently manage state store files in Apache Spark streaming applications

Control the lifecycle of state store files using streaming configurations...

Auto Loader failures with java.io.FileNotFoundException for SST and log files

Use a separate checkpoint folder outside of the Delta directory....

Auto Loader job fails saying metadata file is missing from checkpoint directory

Recover or recreate the missing metadata file, and then ensure the checkpoint location is not in a bucket with a lifecycle policy enabled. ...

Stateful Structured Streaming jobs fail after making changes to stateful operations

Avoid changes to stateful operations between restarts....

Structured Streaming job fails with a Streaming Query Exception when a schema changes in the source table

Enable schema tracking and set allowSourceColumnRenameAndDrop to true....

Duplicates appearing in Auto Loader with file notification feature despite set backfill interval

Remove the allowOverwrites configuration or implement a deduplicate logic....

writeStream/readStream leads to an error when the schema contains “NullType”

Replace NullType with a literal None value and then cast it to StringType. ...

Resumed streaming job fails after pause with StreamingQueryException error

Avoid pausing streaming jobs for longer than the delta.logRetentionDuration value, or restart the stream with a new checkpoint location....

DLT pipeline is very slow when using Auto Loader and a Glob filter

Configure Auto Loader with file notification mode....

Getting an InconsistentReadException error after updating to Databricks Runtime 13.3 LTS or above

Disable file status caching to reduce the time between file status checks. ...

Using glob patterns for directory filtering impacting Auto Loader performance

Use a more specific root path to reduce the scope of the initial scan....

[CONCURRENT_QUERY] Error on Auto Loader job

Set the Auto Loader job to be configured to run in "continuous" mode instead of "available now" mode....

Structured Streaming workflow reading data from CDC is failing

Set spark.databricks.streaming.stateStore.stateSchemaCheck.ignoreNullCompatibility to true....

Error when trying to run an Auto Loader job that uses cloudFiles

Set spark.databricks.cloudFiles.checkSourceChanged to false....

Stateful Streaming query failing with SparkSecurityException error after restarting on a shared cluster or serverless

Switch the cluster access mode....

Receiving com.databricks.sql.io.FileReadException: Error while reading file on streaming queries

Ensure the delta.deletedFileRetentionDuration value is longer than the time it takes your query to complete....

Auto Loader (file notification mode) fails to identify new files from the cloud queue service

Ensure that messages in the cloud queue are of the expected format....

Unable to use fields with qualifiers in the DLT Apply Changes API

Ensure that qualifiers are extracted prior to reference in the Apply Changes definition....

Streaming job reading from Kinesis using EFO mode fails with ResourceInUseException error

Set maxFetchDuration to 5 seconds....

NoSuchMethodError import failure of Protobuf-java when trying to run Apache Spark workflow

Shade the Protobuf class in the JAR....

Streaming job failing with error "org.rocksdb.RocksDBException: Too many open files"

Make changes to Apache Spark configurations applicable to Auto Loader, state store, or both....

Structured Streaming does not process batch size reduction after a failed transaction

Delete the uncommitted offset file from the checkpoint location and restart the stream....

Timeout error when integrating Kafka with Apache Spark Structured Streaming

Upgrade Kafka brokers....

How to retrieve DLT pipeline details using Python and the Databricks API

Use the provided code....

Streaming job failing with “Job terminated with exception” error

Restart the stream with a new checkpoint, or for DLT pipelines do a full refresh....