Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Identify duplicate data on append operations

...

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table

Delta Sharing has limits on the metadata size of a shared table. If you exceed these limits it generates an error....

Found duplicate columns error blocks creation of a Delta table

Duplicate column names are not allowed in Delta tables....

Hive-style partitions not found on Delta table after enabling column mapping mode

Delta Lake column mapping does not support Hive-style partitions....

Dropping and recreating Delta tables results in a DeltaVersionsNotContiguousException error

Instead of dropping and recreating Delta tables, use the CREATE OR REPLACE command....

"AnalysisException: Incompatible Format Detected" error when writing to OpenSearch

Make sure there is no _delta_log folder in your root directory....

AnalysisException error due to a schema mismatch

Modify the write command and set the mergeSchema property to true....

Timestamp change to underlying Apache Parquet/change data files while using Change Data Capture (CDC)

For timestamp-based queries, ensure that the original file timestamps are preserved during the migration process....

Unable to read Delta table with deletion vectors

Use the cluster with Databricks Runtime 12.2 LTS - 15.3 to query all deletion-vector-enabled Delta tables....

Unknown Apache Spark internal error when running Delta table queries

Reorganize the folder structure for the specific partition causing the problem. ...

Running OPTIMIZE on Delta tables causing ConcurrentDeleteDeleteException error

Disable AUTO OPTIMIZE....

DELTA_CLUSTERING_COLUMN_MISSING_STATS error when attempting to define liquid clustering for a delta table

Ensure that you have generated Delta statistics for the columns used as clustering keys....

INSERT operation fails while trying to execute multiple concurrent INSERT or MERGE operations to append data

Make sure the isolation levels are correctly set or refactor to remove conflicts....

You do not use deletion vectors, but see a file named deletion vector in your data path

It is an artifact of low shuffle merge and is removed on the next VACUUM run....

Databricks Runtime is not able to read data in a format other than Delta

Delete the transactional log folder or move it to a different location. ...

Running a Python UDF fails with permission error

Ask function owner to grant EXECUTE permission....

COUNT operation on a DataFrame returning zero or incorrect number of records

Schedule operations to run sequentially, save the DataFrame to a checkpoint, and/or use snapshot isolation....

Error [DELTA_CLUSTERING_SHOW_CREATE_TABLE_WITHOUT_CLUSTERING_COLUMNS] when running SHOW CREATE TABLE command

Upgrade to Databricks Runtime 15 or above or disable the liquid clustering table feature....

FileReadException error when trying to run streaming job reading from system tables

Increase job frequency or the maxVersionsPerRpc....

DeltaFileNotFoundException when reading a table

Ensure that Delta log files are not getting deleted prematurely....

Error FileNotFoundException while streaming job or reading Delta table even with ignoremissingfiles set

Use the FSCK repair command to synchronize the metadata with the actual data files....

Syntax error when running vacuum with USING INVENTORY command

Upgrade your Databricks Runtime to version 15.2 or above. ...

How to get the full size of a Delta table or partition

Use a Scala command and pass in either the root path or the path to the partition....

Error when trying to copy datasets to different regions using Delta Sharing

Split your intervals into separate columns for day, hour, minute and second; convert the interval to a string, or convert the entire interval to seconds and store as an integer. ...

ETL process fails to process a column and throws error Row group size has overflowed

Reduce the default row group size and increase the frequency of size checks....

InvalidSchemaException error when trying to insert data into a Delta table

Define a field type for any fields that use a StructType within a StructField....

Job fails with ExecutorLostFailure error due to excessive garbage collection (GC)

Broadcast the smaller table instead of the larger one....

Parquet table last modification retrieval returns NULL

List the files within the parquet table path and sort them by the modificationTime column. ...

Time travel SELECT query works on older dates even after VACUUM

This is expected behavior but you can also test that your VACUUM command ran successfully. ...

Unable to exclude columns from a table based on specific strings in the comments

Leverage DESCRIBE TABLE to retrieve and filter on the metadata which includes comments....

Delta table as a streaming source returns error DELTA_FILE_NOT_FOUND_DETAILED even though no user or lifecycle rule has deleted files

Consume the source table from scratch, consume from a specific version, or load data using a specific timestamp as filter....

Symlink format manifest fails when trying to enable liquid clustering on a table

Disable deletion vectors or avoid using Symlink format manifest, as appropriate for your context. ...

DataFrame to Ray Dataset conversions taking a long time to execute

Set spark.task.resource.gpu.amount to 0, modify num_cpus_worker_node, or enable Spark cluster auto-scaling....

Files restored from a Delta table archive are not recognized by Delta with archival support enabled

Temporarily disable archival support or increase the value for the table property delta.timeUntilArchived....

Job failing with DELTA_CHANGE_DATA_FILE_NOT_FOUND error

Use ignoreMissingFile config or a new checkpoint....

Column statistics missing when running ANALYZE TABLE COMPUTE STATISTICS after ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS

Only run ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS to ensure column statistics remain available. ...

Error “Job aborted due to stage failure” when trying to shallow clone a large table with significant DeltaLog entries

Increase ​​spark.driver.maxResultSize to 6 GB or greater....

Iceberg metadata not reflecting latest changes made to table, appearing out of sync

Add `MSCK REPAIR TABLE SYNC METADATA` to the end of the ingestion job, or run manually after each ingestion....

Recover a dropped table when a new table is created with the same name

Rename the existing table and then UNDROP the deleted table....

Find the number of files per partition in a Delta table

Retrieve the partition structure of a Delta table and display the number of data files per partition value....

Intermittent long-running OPTIMIZE command for liquid clustered table

Use Databricks Runtime 16.2 or above to run the OPTIMIZE command....

JOIN operation on masked column using masked values instead of unmasked ones

Create a materialized view or hash sensitive columns....

Automate VACUUM metrics logging for Delta table cleanup audits

Parse the operationMetrics from the Delta transaction log using DESCRIBE HISTORY, and store this information in a dedicated Delta audit table....

Track deleted files from VACUUM in Delta table history

Use DESCRIBE HISTORY to find VACUUM operations and check operationMetrics for deleted file count and size....

Casting BigNumeric values to Decimal(38, 38) fails with NUMERIC_VALUE_OUT_OF_RANGE error

Modify the data type to NUMERIC in BigQuery first....

Delta Sharing shared table appearing empty in the target Databricks workspace

Use the correct representation of the data type when adding partition filters....

PySpark merge operation with 'withSchemaEvolution' fails on serverless compute

Use a job or all-purpose compute in Databricks Runtime 15.4 LTS, or use SQL....

Photon memory issue while querying a table with a large number of columns

Use the variant datatype in your CREATE TABLE AS SELECT query, or disable Photon....

Error [DELTA_UNSUPPORTED_FEATURES_FOR_READ] when accessing a Delta table

Use Databricks Runtime 15.3 or above....