Append to a DataFrame

Learn how to append to a DataFrame in Databricks....

How to improve performance with bucketing

Learn how to improve Databricks performance by using bucketing....

How to handle blob data contained in an XML file

Learn how to handle blob data contained in an XML file....

Simplify chained transformations

Learn how to simplify chained transformations on your DataFrame in Databricks....

Hive UDFs

Learn how to create and use a Hive UDF for Databricks....

Prevent duplicated columns when joining two DataFrames

Learn how to prevent duplicated columns when joining two DataFrames in Databricks....

Revoke all user privileges

Use a regex and a series of for loops to revoke all privileges for a single user....

How to handle corrupted Parquet files with different schema

Learn how to read Parquet files with a specific schema using Databricks....

No USAGE permission on database

User does not have USAGE permission on the database....

Nulls and empty strings in a partitioned column save as nulls

Learn why nulls and empty strings in a partitioned column save as nulls in Databricks....

Behavior of the randomSplit method

Learn about inconsistent behaviors when using the randomSplit method in Databricks....

Generate schema from case class

Learn how to generate a schema from a Scala case class....

How to specify skew hints in dataset and DataFrame-based join commands

Learn how to specify skew hints in Dataset and DataFrame-based join commands in Databricks....

How to update nested columns

Learn how to update nested columns in Databricks....

Incompatible schema in some files

Learn how to resolve incompatible schema in Parquet files with Databricks....

Unable to infer schema for ORC error

Apache Spark returns an error for ORC files if no schema is defined when reading from an empty directory or a base path with multiple subfolders....

User does not have permission SELECT on ANY File

Regular users cannot create tables without permission when access control is enabled....

Sync fails with [UPGRADE_NOT_SUPPORTED.HIVE_SERDE] Table is not eligible for upgrade from Hive Metastore to Unity Catalog

Convert your Hive SerDE tables to Delta format. ...

Parquet table counts not being reflected based on concurrent updates

Manually refresh the table in the notebook where the count was initially taken....

Empty string values convert to NULL values when saving a table as CSV or text-based file format

Use Delta as the target format for CSV files or other text-based data formats....

'CREATE OR REPLACE' SQL error in a Delta table

Correct the job schedule to ensure that only one query is executed at a time for a specific table....

Increased wait times between micro-batches in Auto Loader

Use file notification mode instead of the directory listing method....

Addressing performance issues with over-partitioned Delta tables

Implement liquid clustering for improved performance....

Count of corrupt_records returns zero in serverless

Collect records in an array and get the count of the array instead....

Error INVALID_TEMP_OBJ_REFERENCE when trying to create a view

Persist the temporary object to a location, then create your view....

Error when trying to parse XML in a shared mode cluster using the from_xml()function

Define an XML schema in a Data Definition Language (DDL) string first. ...

Error when trying to create a distributed Ray dataset using from_spark() function

Set spark.databricks.pyspark.dataFrameChunk.enabled to true....

INVALID_PARAMETER_VALUE error when trying to access a table or view with fine-grained access control

Upgrade the cluster's Databricks Runtime version to a version newer than 15.4 and use a single user access mode cluster....

Other users can see root (main) folder despite not having access permissions

Permanently delete items in the Trash and create separate folders for shared files or objects. ...

Execution error when trying to mount a storage account

Unmount the nested path and mount each storage account separately at distinct mount points....

Resolve Spark directory structure conflicts

Fixing java.lang.AssertionError: assertion failed: Conflicting directory structures detected...

Column values assigning in the order they are passed into Row() as arguments, not to the column name indicated

Create the DataFrame from a list of dictionaries or use the row.toDict() method....

Insufficient privileges error when querying views in a Unity Catalog metastore

Conduct a traceback of the permission tree for the view table and grant access where needed....

Getting error com.univocity.parsers.common.TextParsingException when parsing data

Make sure the delimiter used in your read operation matches the format of your input files....

ALTER TABLE command only reordering on metadata level

Use CREATE OR REPLACE TABLE or INSERT OVERWRITE TABLE instead....

Error when creating a DataFrame with values of different data types

Set spark.sql.pyspark.inferNestedDictAsStruct.enabled in the same notebook in a preceding cell....