Databricks Knowledge Base

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Delta Lake (Azure)

These articles can help you with Delta Lake.

25 Articles in this category

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.

  • Home
  • All articles
  • Delta Lake (Azure)

A file referenced in the transaction log cannot be found

A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement....

Last updated: May 10th, 2022 by Adam Pavlacka

Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Last updated: May 10th, 2022 by mathan.pillai

Converting from Parquet to Delta Lake fails

Converting a file from Parquet to Delta Lake fails with a partition error when you have a subdirectory. Expecting 0 partition column(s), but found 1 partition column(s)...

Last updated: May 10th, 2022 by Jose Gonzalez

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

Last updated: May 10th, 2022 by Adam Pavlacka

Delete your streaming query checkpoint and restart

Delta table doesn't exist. Please delete your streaming query checkpoint and restart....

Last updated: May 10th, 2022 by Adam Pavlacka

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

Last updated: May 10th, 2022 by Adam Pavlacka

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Last updated: May 10th, 2022 by Adam Pavlacka

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

Last updated: May 10th, 2022 by Adam Pavlacka

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Last updated: May 10th, 2022 by Adam Pavlacka

Identify duplicate data on append operations

...

Last updated: May 10th, 2022 by chetan.kardekar

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Last updated: May 10th, 2022 by mathan.pillai

Delta Lake UPDATE query fails with IllegalState exception

Learn how to resolve an issue with Delta Lake UPDATE, DELETE, or MERGE queries that use Python UDFs....

Last updated: May 10th, 2022 by Adam Pavlacka

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Last updated: May 10th, 2022 by DD Sharma

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Last updated: October 7th, 2022 by DD Sharma

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Last updated: May 10th, 2022 by mathan.pillai

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Last updated: July 1st, 2022 by pratik.bhawsar

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Last updated: October 14th, 2022 by deepak.bhutada

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Last updated: October 26th, 2022 by shanmugavel.chandrakasu

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Last updated: December 2nd, 2022 by Rajeev kannan Thangaiah

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

Last updated: February 2nd, 2023 by priyanka.biswas

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

Last updated: February 3rd, 2023 by mathan.pillai

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Last updated: February 3rd, 2023 by mathan.pillai

Recover from a DELTA_LOG corruption error

Learn how to repair a Delta table that reports an IllegalStateException error when queried....

Last updated: February 17th, 2023 by gopinath.chandrasekaran

FileReadException when reading a Delta table

A FileReadException error occurs when you attempt to read from a Delta table. The underlying data has been deleted, or the storage blob was unmounted during a write....

Last updated: February 23rd, 2023 by Adam Pavlacka

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

Last updated: March 16th, 2023 by mounika.tarigopula


© Databricks 2022-2023. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Notice (Updated) | Terms of Use | Your Privacy Choices | Your California Privacy Rights Privacy Rights icon

Definition by Author

0
0