Auto Loader fails to pick up new files when using directory listing mode

Problem You may encounter an issue where Auto Loader does not pick up new files in  directory listing mode ( AWS |  Azure |  GCP ) in scenarios where the source  cloudFiles file naming convention has changed. Cause This is related to the way lexical ordering works when using directory listing mode in Auto Loader. New files with different naming conv...

0 min reading time

Vector search index contains incorrect number of rows

Problem You find your vector search index does not contain the expected number of rows. Example You upload your data, housed in different spreadsheets, to a Unity Catalog Volume. The data is parsed out of each spreadsheet using LangChain and each individual record is then loaded to a Delta table with Apache Spark. Your Delta table contains 475 rows ...

0 min reading time

[ISOLATION_STARTUP_FAILURE] in shared clusters with Databricks Runtime 14.3 LTS - 15.4 LTS

Problem  While working in Databricks Runtime 14.3 LTS - 15.4 LTS, you may encounter an  ISOLATION_STARTUP_FAILURE error when using User-Defined Functions (UDFs) via Unity Catalog Lakeguard, in clusters with shared access mode.  Job aborted due to stage failure: Task 36 in stage 583.0 failed 4 times, most recent failure: Lost task 36.3 in stage 583.0...

2 min reading time