Delete your streaming query checkpoint and restart

Problem

Your job fails with a Delta table <value> doesn't exist. Please delete your streaming query checkpoint and restart. error message.

Cause

Two different streaming sources are configured to use the same checkpoint directory. This is not supported.

For example, assume streaming query A streams data from Delta table A, and uses the directory /checkpoint/A as a checkpoint.

If streaming query B streams data from Delta table B, but attempts to use the directory /checkpoint/A as a checkpoint, the reservoirId of the Delta tables doesn’t match and the query fails with an exception.

Note

A similar issue can occur with S3-SQS if you attempt to share the checkpoint directory. This is because S3-SQS uses an internal Delta table to maintain the event messages.

Solution

You should not share checkpoint directories between different streaming queries.

Use a new checkpoint directory for every new streaming query.