Unclear how to control micro-batch size on a streaming table in Delta Live Tables (DLT)

Problem  You want to c ontrol the micro-batch size on a streaming table, which is created in the same Delta Live Tables (DLT) pipeline using rate limiters, but it is not clear how to achieve this in DLT. Cause  The dlt.readStream() function in Delta Live Tables (DLT) does not directly support the rate limit configuration maxBytesPerTrigger option.  ...

0 min reading time

Incorrect input record count in Apache Spark streaming application logs/micro-batch metrics

Problem When observing the logs of Spark Streaming applications, you notice the metric  numInputRows   in the micro batch metrics or number of input records,  event.progress.numInputRows , logged using the  StreamingQueryListener does not match the expected count. This leads to confusion about the actual number of records being processed. Cause A di...

0 min reading time

Auto Loader does not pick up files for processing when uploading via an Azure function

Problem When you upload files to a source location using an Azure function in Auto Loader, Auto Loader does not pick up the files for processing. The files are also not available in the queue, which Auto Loader sets automatically. The process does work, however, with manual intervention.  Cause Auto Loader listens for the ‘ FlushWithClose ’ event to...

0 min reading time