Databricks Knowledge Base

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Delta Lake (AWS)

These articles can help you with Delta Lake.

29 Articles in this category

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.

  • Home
  • All articles
  • Delta Lake (AWS)

A file referenced in the transaction log cannot be found

A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement....

Last updated: May 10th, 2022 by Adam Pavlacka

Cannot delete data using JDBC in Eclipse

Delete operations on a Delta table fail with a SparkJDBCDriver error when using JDBC in an Eclipse local environment....

Last updated: May 10th, 2022 by annapurna.hiriyur

Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Last updated: May 10th, 2022 by mathan.pillai

Converting from Parquet to Delta Lake fails

Converting a file from Parquet to Delta Lake fails with a partition error when you have a subdirectory. Expecting 0 partition column(s), but found 1 partition column(s)...

Last updated: May 10th, 2022 by Jose Gonzalez

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

Last updated: May 10th, 2022 by Adam Pavlacka

Delete your streaming query checkpoint and restart

Delta table doesn't exist. Please delete your streaming query checkpoint and restart....

Last updated: May 10th, 2022 by Adam Pavlacka

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

Last updated: May 10th, 2022 by Adam Pavlacka

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Last updated: May 10th, 2022 by Adam Pavlacka

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

Last updated: May 10th, 2022 by Adam Pavlacka

HIVE_CURSOR_ERROR when reading a table in Athena

When you try to read a table in Athena, the select query returns a HIVE_CURSOR_ERROR message....

Last updated: May 10th, 2022 by annapurna.hiriyur

Access denied when writing Delta Lake tables to S3

Learn how to resolve an access denied 403 Forbidden error when writing Delta Lake tables to S3....

Last updated: May 10th, 2022 by Adam Pavlacka

Delta Lake write job fails with java.lang.UnsupportedOperationException

Learn how to prevent java.lang.UnsupportedOperationException in Delta Lake write jobs....

Last updated: May 10th, 2022 by Adam Pavlacka

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Last updated: May 10th, 2022 by Adam Pavlacka

Identify duplicate data on append operations

...

Last updated: May 10th, 2022 by chetan.kardekar

Object lock error when writing Delta Lake tables to S3

Delta Lake does not support S3 buckets with object lock enabled. com.amazonaws.services.s3.model.AmazonS3Exception...

Last updated: May 10th, 2022 by ashritha.laxminarayana

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Last updated: May 10th, 2022 by mathan.pillai

Delta Lake UPDATE query fails with IllegalState exception

Learn how to resolve an issue with Delta Lake UPDATE, DELETE, or MERGE queries that use Python UDFs....

Last updated: May 10th, 2022 by Adam Pavlacka

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Last updated: May 10th, 2022 by DD Sharma

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Last updated: October 7th, 2022 by DD Sharma

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Last updated: May 10th, 2022 by mathan.pillai

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Last updated: July 1st, 2022 by pratik.bhawsar

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Last updated: October 14th, 2022 by deepak.bhutada

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Last updated: October 26th, 2022 by shanmugavel.chandrakasu

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Last updated: December 2nd, 2022 by Rajeev kannan Thangaiah

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

Last updated: February 2nd, 2023 by priyanka.biswas

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

Last updated: February 3rd, 2023 by mathan.pillai

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Last updated: February 3rd, 2023 by mathan.pillai

Recover from a DELTA_LOG corruption error

Learn how to repair a Delta table that reports an IllegalStateException error when queried....

Last updated: February 17th, 2023 by gopinath.chandrasekaran

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

Last updated: March 16th, 2023 by mounika.tarigopula


© Databricks 2022-2023. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Notice (Updated) | Terms of Use | Your Privacy Choices | Your California Privacy Rights Privacy Rights icon

Definition by Author

0
0