Problem

You attempt to shallow clone a large table with a significant number of DeltaLog entries using the following command. 

CREATE TABLE <table-name> SHALLOW CLONE <source-table-name>

You receive the following error message.  

"Job aborted due to stage failure: Total size of serialized results of <number> tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB."

Cause

The cloning process needs to collect entries on the driver before committing, which can result in a large amount of data overflowing in memory when the metadata is laid out. 

Overflowing memory happens when the SHALLOW CLONE operation involves an excessive amount of AddFile entries.

Solution

Set spark.driver.maxResultSize above 4 GB. Start with 6 GB, and if needed, increase further. Consider increasing driver instance size as well. 

For more information, refer to the Spark Configuration documentation.

Preventive measures

• Monitor the size of the metadata and transaction log, and the number of files in the tables you are cloning.
• Consider optimizing the tables themselves, such as by vacuuming and optimizing, to reduce the scale of the shallow clone task.

For more information, review the Diving Into Delta Lake: Unpacking The Transaction Log Databricks blog post.