Streaming
These articles can help you with Structured Streaming and Spark Streaming (the legacy Spark streaming feature).
- Apache Spark DStream is not supported
- Streaming with File Sink: Problems with recovery if you change checkpoint or output directories
- How to set up Apache Kafka on Databricks
- Handling partition column values while using an SQS queue as a streaming source
- How to restart a structured streaming query from last written offset
- How to switch a SNS streaming job to a new SQS queue
- Checkpoint files not being deleted when using
display()
- Checkpoint files not being deleted when using
foreachBatch()