Streaming

These articles can help you with Structured Streaming and Spark Streaming (the legacy Apache Spark streaming feature).

• Append output is not supported without a watermark
• Apache Spark DStream is not supported
• Streaming with File Sink: Problems with recovery if you change checkpoint or output directories
• Get the path of files consumed by Auto Loader
• How to set up Apache Kafka on Databricks
• Handling partition column values while using an SQS queue as a streaming source
• How to restart a structured streaming query from last written offset
• How to switch a SNS streaming job to a new SQS queue
• Kafka error: No resolvable bootstrap urls
• readStream() is not whitelisted error when running a query
• Checkpoint files not being deleted when using display()
• Checkpoint files not being deleted when using foreachBatch()
• Conflicting directory structures error
• RocksDB fails to acquire a lock
• Stream XML files using an auto-loader
• Streaming job using Kinesis connector fails
• Streaming job gets stuck writing to checkpoint