Streaming

These articles can help you with Structured Streaming and Spark Streaming (the legacy Apache Spark streaming feature).

• Append output is not supported without a watermark
• Apache Spark DStream is not supported
• Streaming with File Sink: Problems with recovery if you change checkpoint or output directories
• How to set up Apache Kafka on Databricks
• Handling partition column values while using an SQS queue as a streaming source
• How to restart a structured streaming query from last written offset
• How to switch a SNS streaming job to a new SQS queue
• Checkpoint files not being deleted when using display()
• Checkpoint files not being deleted when using foreachBatch()