Delete your streaming query checkpoint and restart

Delta table doesn't exist. Please delete your streaming query checkpoint and restart.

Written by Adam Pavlacka

Last published at: May 10th, 2022

Problem

Your job fails with a Delta table <value> doesn't exist. Please delete your streaming query checkpoint and restart. error message.

Cause

Two different streaming sources are configured to use the same checkpoint directory. This is not supported.

For example, assume streaming query A streams data from Delta table A, and uses the directory /checkpoint/A as a checkpoint.

If streaming query B streams data from Delta table B, but attempts to use the directory /checkpoint/A as a checkpoint, the reservoirId of the Delta tables doesn’t match and the query fails with an exception.

AWS

Delete

Info

A similar issue can occur with S3-SQS if you attempt to share the checkpoint directory. This is because S3-SQS uses an internal Delta table to maintain the event messages.

Delete

Azure

Delete

Info

A similar issue can occur with ABS-AQS if you attempt to share the checkpoint directory. This is because ABS-AQS uses an internal Delta table to maintain the event messages.

Delete

Solution

You should not share checkpoint directories between different streaming queries.

Use a new checkpoint directory for every new streaming query.