Delete your streaming query checkpoint and restart
Problem
Your job fails with a Delta table <value> doesn't exist. Please delete your streaming query checkpoint and restart.
error message.
Cause
Two different streaming sources are configured to use the same checkpoint directory. This is not supported.
For example, assume streaming query A streams data from Delta table A, and uses the directory /checkpoint/A
as a checkpoint.
If streaming query B streams data from Delta table B, but attempts to use the directory /checkpoint/A
as a checkpoint, the reservoirId
of the Delta tables doesn’t match and the query fails with an exception.
Note
A similar issue can occur with S3-SQS if you attempt to share the checkpoint directory. This is because S3-SQS uses an internal Delta table to maintain the event messages.