SSO server redirects to original URL, not to vanity Databricks URL

SSO configuration should use the vanity URL as the base target URL....

SSO SAML authentication error with PingFederate

...

Find your workspace ID

Learn how to find your Databricks workspace ID in the web UI as well as via a notebook command....

Failed to add user error due to email or username already existing with a different case

You should ensure casing for usernames is consistent across all accounts and providers in your system....

Cannot access Databricks secrets when using a "No isolation shared" cluster

You cannot use dbutils.secrets.get() when admin protection for No isolation shared clusters is enabled in your account....

DNS resolution fails for a newly created Databricks workspace

Users should wait for 10 minutes before re-creating a workspace with the same name....

Reactivate a user that has been disabled with AAD at the Account and Workspace level

Using a default account API call, activate the user first at the account level, then the workspace level....

Slack alarm notifications test fails with “invalid_token”

Verify that the notification destination URL is still active, check the expiration status of the destination token, or create a new webhook in the destination. ...

AWS services fail with No region provided error

AWS services fail with a Java "No region provided" error in Databricks Runtime 7.0 and above....

Troubleshooting Amazon Redshift connection problems

Learn how to troubleshoot Amazon Redshift connection problems....

Vulnerability scan shows vulnerabilities in Databricks EC2 instances

Learn how to resolve vulnerabilities in long-running Databricks EC2 instances....

Configure custom DNS settings using dnsmasq

Learn how to configure custom DNS settings using dnsmasq....

Unable to load AWS credentials

...

Access denied when writing logs to an S3 bucket

Learn how to resolve an access denied error while writing logs to an S3 bucket....

S3 part number must be between 1 and 10000 inclusive

Learn how to resolve the S3 part number must be between 1 and 10000 inclusive error....

How to analyze user interface performance issues

Learn how to troubleshoot Databricks user interface performance issues....

Unable to mount Azure Data Lake Storage Gen1 account

Learn how to resolve errors that occur when mounting Azure Data Lake Storage Gen1 to Databricks....

Failed credential validation checks error with Terraform

You get a 'Failed credential validation checks' error message when using Terraform to deploy a Databricks workspace in AWS....

Deployment name prefix not defined in Terraform

You must have a deployment_name defined in Databricks, before using it to deploy workspaces with Terraform....

Job fails with input/output error when displaying a data frame

Check your NSG and port settings. ...

Workspace creation failure when using Terraform

Check SCPs and make adjustments. ...

Clusters terminating with error AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE (CLIENT_ERROR)

Increase the size of your subnet to allow for more IP addresses or free up some of the IP addresses currently in use....

Configure Simba ODBC driver with a proxy in Windows

How to configure the Simba ODBC driver to connect through a proxy server when using Windows....

Enable OpenJSSE and TLS 1.3

Add OpenJSSE to allow the use of TLS 1.3 for encrypted data transmission....

How to calculate the number of cores in a cluster

Learn how to calculate the number of cores in a Databricks cluster....

Install a private PyPI repo

How to install libraries from private PyPI repositories....

IP access list update returns INVALID_STATE

Cannot update IP access list. INVALID_STATE error message....

Launch fails with Client.InternalError

Cluster launch fails with Client.InternalError on a E2 workspace due to IAM role KMS key policy....

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Cluster failed to launch

Learn how to resolve cluster launch failures....

Custom Docker image requires root

Custom Docker containers must be configured to start as the root user when used with Databricks....

Job fails due to cluster manager core instance request limit

Learn how to troubleshoot Databricks errors related to API rate limits....

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Failed to expand the EBS volume

Learn how to set the correct permissions on AWS EC2 to enable EBS volume expansion....

EBS leaked volumes

Learn how to handle leaked EBS volumes that are not automatically deleted....

Log delivery fails with AssumeRole

You get an access denied error when you try to use `AssumeRole` to send cluster logs to a S3 bucket in another account....

Multi-part upload failure

Learn how to resolve a multi-part upload failure....

Persist Apache Spark CSV metrics to a DBFS location

Persist Spark CSV metrics to a sink in a DBFS location....

Replay Apache Spark events in a cluster

Use a single node cluster to replay another cluster's event log in the Spark UI....

S3 connection fails with "No role specified and no roles available"

Databricks Utilities (dbutils) fails with a role error when IAM Role Passthrough and an Instance Profile are both enabled....

Set Apache Hadoop core-site.xml properties

Set Apache Hadoop core-site.xml properties in a Databricks cluster....

Set executor log level

Learn how to set the log levels on Databricks executors....

Set instance_profile_arn as optional with a cluster policy

Use a cluster policy to set the AWS attribute instance_profile_arn as optional....

Apache Spark job doesn’t start

Learn how to troubleshoot a Databricks Spark job that won't start....

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

Unexpected cluster termination

Learn how to troubleshoot a Databricks cluster that stopped unexpectedly....

How to configure single-core executors to run JNI libraries

Learn how to configure single-core executors to run JNI libraries on Databricks....

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Apache Spark UI shows less than total node memory

Learn what to do when the Spark UI shows less memory than is actually available on the node....

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Enable retries in init script

Add a retry function to your init script....

Cannot read audit logs due to duplicate columns

Case-sensitive parameter names cause a duplicate column error when reading audit logs....

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Use iptables to access the EC2 metadata server

You must setup custom iptables rules in order to access the EC2 metadata server. You cannot access it by default....

Run a custom Databricks Runtime on your cluster

Configure your cluster to run a custom Databricks Runtime image via the UI or API....

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Use audit logs to identify who deleted a cluster

You can use audit logs to identify who deleted a cluster configuration....

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

R commands fail on custom Docker cluster

R version 4.2.0 changed the way Renviron.site is initialized, so you must set an environment variable when using custom Docker clusters....

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Legacy global init script migration notebook

Easily migrate your legacy global init scripts to the current global init script framework....

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

VcpuLimitExceeded error when creating a GPU ML cluster

You need to increase the vCPU capacity in your AWS account....

Shorten cluster provisioning time by using Docker containers

Learn how to speed up cluster provisioning by using Docker container services...

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Failure to start AWS clusters with c5.2xlarge instance type

Contact AWS support if you cannot start clusters with the c5.2xlarge instance type....

Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files....

Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Embed system.exit code in your application to shutdown the Java virtual machine with exit code 0....

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Modify your log4j2 configuration file directly within the Databricks environment. ...

High-cached memory in clusters but no active applications submitted

Clear cache, restart cluster, and add executors to manage memory....

[ISOLATION_STARTUP_FAILURE] in shared clusters with Databricks Runtime 14.3 LTS - 15.4 LTS

Leverage an assigned cluster or revert to a custom Databricks Runtime release version as interim solutions....

UDF functions not supported error in Unity Catalog

Rewrite UDFs in Python or SQL, otherwise use single user clusters where Java UDFs are supported....

Ganglia metrics not appearing in historical metrics snapshots list

Configure the collection period, or update Databricks Runtime to at least 13.0 to use Compute metrics instead. ...

Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes. ...

"Security Daemon Registration Exception" error when trying to create a compute cluster in AWS workspaces

Ensure you have correctly created and set up a VPC endpoint, regional STS endpoints, and your network allows communication. ...

Init script stored on a volume fails to execute on cluster start

Init scripts created on Windows systems and uploaded to Unity Catalog volumes have CRLF as a newline which needs to be converted to LF before the cluster can process it. ...

Databricks API last_activity_time attribute shows incorrect timestamp

Use the cluster auto-termination feature to manage cluster termination based on inactivity....

BROADCAST_VARIABLE_NOT_LOADED or JVM_ATTRIBUTE_NOT_SUPPORTED errors when using broadcast variables in a shared access mode cluster

Use a single-user cluster or pass a variable into a function as a state instead. ...

404 error when installing krb5-user module

Manually remove the var directory path to refresh the cached data. ...

Cluster startup failure while running proxy-configured init script with other init scripts

Modify the proxy init script to bypass local addresses....

Init scripts failing with unexpected end of file error

Remove the special characters from the init script....

Unable to access the hive_metastore schema

Ensure all clusters use the same Hive metastore version and Apache Spark configurations are set. ...

Job fails while installing ODBC Driver 18 for SQL Server using an init script

Add msodbcsql18 to the LD_LIBRARY_PATH then append LD_LIBRARY_PATH path to /etc/environment....

403 AccessDenied error when writing cluster delivery logs

Use the clusters API to apply KMS encryption for cluster logs. ...

Append to a DataFrame

Learn how to append to a DataFrame in Databricks....

How to improve performance with bucketing

Learn how to improve Databricks performance by using bucketing....

How to handle blob data contained in an XML file

Learn how to handle blob data contained in an XML file....

Simplify chained transformations

Learn how to simplify chained transformations on your DataFrame in Databricks....

How to dump tables in CSV, JSON, XML, text, or HTML format

Learn how to output tables from Databricks in CSV, JSON, XML, text, or HTML format....

Hive UDFs

Learn how to create and use a Hive UDF for Databricks....

Prevent duplicated columns when joining two DataFrames

Learn how to prevent duplicated columns when joining two DataFrames in Databricks....

Revoke all user privileges

Use a regex and a series of for loops to revoke all privileges for a single user....

How to handle corrupted Parquet files with different schema

Learn how to read Parquet files with a specific schema using Databricks....

No USAGE permission on database

User does not have USAGE permission on the database....

Nulls and empty strings in a partitioned column save as nulls

Learn why nulls and empty strings in a partitioned column save as nulls in Databricks....

Behavior of the randomSplit method

Learn about inconsistent behaviors when using the randomSplit method in Databricks....

Job fails when using Spark-Avro to write decimal values to AWS Redshift

Learn how to resolve job failures when writing decimal values to AWS Redshift with Spark-Avro....

Generate schema from case class

Learn how to generate a schema from a Scala case class....

How to specify skew hints in dataset and DataFrame-based join commands

Learn how to specify skew hints in Dataset and DataFrame-based join commands in Databricks....

How to update nested columns

Learn how to update nested columns in Databricks....

Incompatible schema in some files

Learn how to resolve incompatible schema in Parquet files with Databricks....

Access denied when writing to an S3 bucket using RDD

Learn how to resolve an access denied error when writing to an S3 bucket using RDD....

Invalid timestamp when loading data into Amazon Redshift

Learn how to resolve an invalid timestamp error when loading data into AWS Redshift....

Unable to infer schema for ORC error

Apache Spark returns an error for ORC files if no schema is defined when reading from an empty directory or a base path with multiple subfolders....

Object ownership is getting changed on dropping and recreating tables

Use TRUNCATE or REPLACE for tables and ALTER VIEW for views instead of dropping and recreating them....

User does not have permission SELECT on ANY File

Regular users cannot create tables without permission when access control is enabled....

Sync fails with [UPGRADE_NOT_SUPPORTED.HIVE_SERDE] Table is not eligible for upgrade from Hive Metastore to Unity Catalog

Convert your Hive SerDE tables to Delta format. ...

Parquet table counts not being reflected based on concurrent updates

Manually refresh the table in the notebook where the count was initially taken....

Empty string values convert to NULL values when saving a table as CSV or text-based file format

Use Delta as the target format for CSV files or other text-based data formats....

'CREATE OR REPLACE' SQL error in a Delta table

Correct the job schedule to ensure that only one query is executed at a time for a specific table....

Increased wait times between micro-batches in Auto Loader

Use file notification mode instead of the directory listing method....

Addressing performance issues with over-partitioned Delta tables

Implement liquid clustering for improved performance....

Count of corrupt_records returns zero in serverless

Collect records in an array and get the count of the array instead....

Create tables on JSON datasets

Create tables on JSON datasets; requires SerDe JAR....

Delete table when underlying S3 bucket is deleted

Do not delete the contents of a S3 bucket before dropping a table that stores data in the bucket....

Failure when mounting or accessing Azure Blob storage

Learn how to resolve a failure when mounting or accessing Azure Blob storage from Databricks....

Unable to read files and list directories in a WASB filesystem

Learn how to interpret errors that occur when accessing WASB append blob types in Databricks....

Optimize read performance from JDBC data sources

Learn how to optimize performance when reading from JDBC data sources in Databricks....

Troubleshooting JDBC/ODBC access to Azure Data Lake Storage Gen2

Learn how to troubleshoot JDBC and ODBC access to Azure Data Lake Storage Gen2 from Databricks....

CosmosDB-Spark connector library conflict

Learn how to resolve conflicts that arise when using the CosmosDB-Spark connector library with Databricks....

Failure to detect encoding in JSON

Learn how to resolve a failure to detect encoding of input JSON files when using BOM with Databricks....

Inconsistent timestamp results with JDBC applications

Timestamp records are inconsistent with JDBC applications when daylight saving time adjustments are made....

Kafka client terminated with OffsetOutOfRangeException

Kafka client is terminated with `OffsetOutOfRangeException` when trying to fetch messages...

Apache Spark JDBC datasource query option doesn’t work for Oracle database

Learn how to resolve an error that occurs when using the Apache Spark JDBC datasource to connect to Oracle Database from Databricks....

Accessing Redshift fails with NullPointerException

Learn how to resolve a `NullPointerException` error that occurs when you read a Redshift table....

Redshift JDBC driver conflict issue

Learn how to resolve a Redshift JDBC SQLDriverWrapper driver conflict....

ABFS client hangs if incorrect client ID or wrong path used

Trying to access an Azure Blob File System (ABFS) path results in a hung command when using Azure Data Lake Storage Gen2 (ADLS)....

Recursive references in Avro schema are not allowed

Apache Avro data sources cannot have recursive references in the schema when used with Spark....

SQL access control error when using Snowflake as a data source

Snowflake does not officially support schema as an option; you must use sfschema....

Apache Spark reading .gzip files from S3 instead of decompressed data

Rename the files in S3 from .gzip to .gz....

Column drift when reading multiple delimited files

Ensure that all files being processed together have the same schema....

NullPointerException when reading shapefiles from cloud storage on a Mosaic and GDAL enabled cluster

Zip the entire shapefile and upload to a Unity Catalog volume or DBFS storage. ...

MULTIPLE_XML_DATA_SOURCE error while working with XML data

Remove the external XML library from the cluster. ...

Databricks jobs using AWS Glue Data Catalog failing due to inability to reach cluster drivers

Ensure the Databricks cluster's IAM role has necessary permissions to access AWS Glue Data Catalog, update the IAM policy, and restart the cluster....

ALTER TABLE (drop partition) error in Unity Catalog external tables

For CSV, JSON, ORC, or data formats, use partition metadata logging. ...

Schema mismatch issue while reading parquet files

Fix the file schema or read the files separately. ...

Reading a CSV file in DROPMALFORMED still includes malformed rows in the result

...

Cannot see ingested data loaded from an external ORC table

Use the same Hive interface to ingest and read your Delta table. ...

Security Bulletin: Databricks JDBC Driver Vulnerability Advisory - [CVE-2024-49194]

Restart any long running clusters and update your JDBC driver to the latest version....

Error when creating a Delta table using the UI and external data in Delta format

Create the table using a notebook instead. ...

Cannot access objects written by Databricks from outside Databricks

Learn how to resolve a HeadObject operation error and access objects written by Databricks from outside Databricks....

Cannot read Databricks objects stored in the DBFS root directory

Learn what to do when you cannot read Databricks objects stored in the DBFS root directory....

How to calculate the Databricks file system (DBFS) S3 API call cost

Learn how to calculate the Databricks file system (DBFS) S3 API call cost....

How to specify the DBFS path

Learn how to specify the DBFS path in Apache Spark, Bash, DBUtils, Python, and Scala....

Operation not supported during append

...

Parallelize filesystem operations

Parallelize Apache Spark filesystem operations with DBUtils and Hadoop FileUtil; emulate DistCp....

S3 connection reset error

Apache Spark job fails with S3 connection reset error....

Upload large files using DBFS API 2.0 and PowerShell

Use PowerShell and the DBFS API to upload large files to your Databricks workspace....

FileReadException on DBFS mounted filesystem

Use dbutils.fs.refreshMounts() to refresh mount points before referencing a DBFS path in your Spark job....

Directory view in the workspace UI does not match the result obtained using the dbutils command

Use the file:/ prefix when accessing workspace files via dbutils....

Error while calling o430.mount and the executor has failed to refresh mounts

Use the refreshMounts command....

Download files from DBFS with the web browser

Make sure the DBFS File Browser is enabled if you want to download files from DBFS via the web....

Null column values display as NaN

Null column values correctly display as NaN in Databricks SQL....

Retrieve queries owned by a disabled user

How to retrieve queries owned by a disabled user in Databricks SQL....

Job timeout when connecting to a SQL endpoint over JDBC

Increase the SocketTimeout value in the JDBC connection URL to prevent thread requests from timing out....

Slowness when fetching results in Databricks SQL

Ensure that cloud fetch is enabled for best performance when using ODBC/JDBC to fetch results....

ZORDER results in "Hilbert indexing can only be used on 9 or fewer columns" error

OPTIMIZE ZORDER BY command has a hard limit of nine columns....

Cannot customize Apache Spark config in Databricks SQL warehouse

You can only configure a limited set of global Spark properties when using a SQL warehouse....

SQL warehouse launch fails to start with "PERMISSION_DENIED"

Ensure the necessary permissions are provided to the SQL warehouse owner...

Update the Databricks SQL warehouse owner

Learn how to use the API to transfer ownership of a SQL warehouse to a new owner...

Unable to use dynamic variable passing to create a function in Databricks SQL Warehouse

Use a Python UDF in a notebook to dynamically pass the table name as a variable, then access the function in a notebook or DBSQL. ...

Error running parameterized SQL queries in Databricks Connect with VS Code

Pass the SQL parameter in Databricks Connect using string interpolation. ...

Divide by zero arithmetic exception when running the OPTIMIZE command on a Delta table

Use the REORG TABLE command and then OPTIMIZE to clean up soft-deleted records....

Databricks SQL ANSI enablement guide

Learn how to enable ANSI SQL for your existing Databricks workloads....

spark.rpc.message.maxSize error after running DESCRIBE HISTORY in SQL warehouse

You must limit or prune the query when using DESCRIBE HISTORY on large history logs....

DeltaInvariantViolationException: Exceeds char/varchar type length limitation error when writing a Delta table

...

Redshift queries using Lakehouse Federation taking longer than expected

Make JDBC fetchsize adjustments or use the COPY UNLOAD method....

Apache Spark session is null in DBConnect

A `sparkSession is null while trying to executeCollectResult` error message occurs when using DBConnect....

Databricks Connect reports version error with Databricks Runtime 6.4

...

Failed to create process error with Databricks CLI in Windows

Databricks CLI may not work correctly in Windows if your Python path has a space in it....

GeoSpark undefined function error with DBConnect

Use GeoSpark code with a DBConnect session....

Get Apache Spark config in DBConnect

Use a REST API call and DBConnect to get the Apache Spark configuration for your cluster....

How to Sort S3 files By Modification Time in Databricks Notebooks

...

Invalid Access Token error when running jobs with Airflow

Learn what to do when you receive an Invalid Access Token error when using Databricks jobs with Airflow....

ProtoSerializer stack overflow error in DBConnect

A stack overflow error in DBConnect indicates that you need to allocate more memory on the local PC....

Use tcpdump to create pcap files

Analyze network traffic between nodes on a specific cluster by using tcpdump to create pcap files....

Terraform registry does not have a provider error

You cannot install the Databricks Terraform provider if the required_providers block is not defined in your modules....

Databricks Connect job fails after a Databricks Runtime update

Use the most recent version of Databricks Connect that matches your Databricks Runtime version to avoid an error....

Databricks-sql-python package fails with self-signed certificate errors and code _ssl.c:1006

Make sure your certificate is valid and updated....

Paths behave differently on Git folders and workspace folders

Git folders can reference the project root, while workspace folders reference the current working directory....

A file referenced in the transaction log cannot be found

A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement....

Cannot delete data using JDBC in Eclipse

Delete operations on a Delta table fail with a SparkJDBCDriver error when using JDBC in an Eclipse local environment....

Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Converting from Parquet to Delta Lake fails

Converting a file from Parquet to Delta Lake fails with a partition error when you have a subdirectory. Expecting 0 partition column(s), but found 1 partition column(s)...

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

Delete your streaming query checkpoint and restart

Delta table doesn't exist. Please delete your streaming query checkpoint and restart....

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

HIVE_CURSOR_ERROR when reading a table in Athena

When you try to read a table in Athena, the select query returns a HIVE_CURSOR_ERROR message....

Access denied when writing Delta Lake tables to S3

Learn how to resolve an access denied 403 Forbidden error when writing Delta Lake tables to S3....

Delta Lake write job fails with java.lang.UnsupportedOperationException

Learn how to prevent java.lang.UnsupportedOperationException in Delta Lake write jobs....

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Identify duplicate data on append operations

...

Object lock error when writing Delta Lake tables to S3

Delta Lake does not support S3 buckets with object lock enabled. com.amazonaws.services.s3.model.AmazonS3Exception...

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Delta Lake UPDATE query fails with IllegalState exception

Learn how to resolve an issue with Delta Lake UPDATE, DELETE, or MERGE queries that use Python UDFs....

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Recover from a DELTA_LOG corruption error

Learn how to repair a Delta table that reports an IllegalStateException error when queried....

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table

Delta Sharing has limits on the metadata size of a shared table. If you exceed these limits it generates an error....

Found duplicate columns error blocks creation of a Delta table

Duplicate column names are not allowed in Delta tables....

Hive-style partitions not found on Delta table after enabling column mapping mode

Delta Lake column mapping does not support Hive-style partitions....

Dropping and recreating Delta tables results in a DeltaVersionsNotContiguousException error

Instead of dropping and recreating Delta tables, use the CREATE OR REPLACE command....

The delta.retentionDurationCheck property is not recognized when using serverless compute

Use VACUUM or table properties to handle retention instead. ...

"AnalysisException: Incompatible Format Detected" error when writing to OpenSearch

Make sure there is no _delta_log folder in your root directory....

AnalysisException error due to a schema mismatch

Modify the write command and set the mergeSchema property to true....

Timestamp change to underlying Apache Parquet/change data files while using Change Data Capture (CDC)

For timestamp-based queries, ensure that the original file timestamps are preserved during the migration process....

Unable to read Delta table with deletion vectors

Use the cluster with Databricks Runtime 12.2 LTS - 15.3 to query all deletion-vector-enabled Delta tables....

VACUUM operations not performing even after enabling predictive optimization

Create a new deletion operation to trigger VACUUM after enabling predictive optimization....

Unknown Apache Spark internal error when running Delta table queries

Reorganize the folder structure for the specific partition causing the problem. ...

Streaming job fails with a DELTA_MERGE_MATERIALIZE_SOURCE_FAILED_REPEATEDLY error

Disable source data materialization during MERGE statement optimization....

Running OPTIMIZE on Delta tables causing ConcurrentDeleteDeleteException error

...

DELTA_CLUSTERING_COLUMN_MISSING_STATS error when attempting to define liquid clustering for a delta table

Ensure that you have generated Delta statistics for the columns used as clustering keys....

INSERT operation fails while trying to execute multiple concurrent INSERT or MERGE operations to append data

Make sure the isolation levels are correctly set or refactor to remove conflicts....

You do not use deletion vectors, but see a file named deletion vector in your data path

It is an artifact of low shuffle merge and is removed on the next VACUUM run....

Databricks Runtime is not able to read data in a format other than Delta

Delete the transactional log folder or move it to a different location. ...

Distinguish active and dead jobs

Learn how to distinguish between active and dead Databricks jobs....

Spark job fails with Driver is temporarily unavailable

Job failure due to Driver being unavailable or unresponsive....

How to delete all jobs using the REST API

Learn how to delete all Databricks jobs using the REST API....

Job cluster limits on notebook output

Job clusters have a maximum notebook output size of 20 MB. If the output is larger, it results in an error....

Job fails, but Apache Spark tasks finish

Your job fails, but all of the Apache Spark tasks have completed successfully. You are using spark.stop() or System.exit(0) in your code....

Job fails due to job rate limit

Learn how to resolve Databricks job failures due to job rate limits....

Create table in overwrite mode fails when interrupted

Learn how to troubleshoot failures that occur when you rerun an Apache Spark write operation by cancelling the currently running job....

Apache Spark Jobs hang due to non-deterministic custom UDF

Learn what to do when your Apache Spark job hangs due to a non-deterministic custom UDF....

Apache Spark job fails with Failed to parse byte string

Apache Spark job fails with a Failed to parse byte string error....

Apache Spark UI shows wrong number of jobs

Apache Spark UI shows the wrong number of active jobs....

Apache Spark job fails with a Connection pool shut down error

Apache Spark job fails with a java.lang.IllegalStateException: Connection pool shut down error....

Job fails with atypical errors message

Job run is throttled and fails due to observing atypical errors message....

Apache Spark job fails with maxResultSize exception

Learn what to do when an Apache Spark job fails with a maxResultSize exception....

Databricks job fails because library is not installed

Learn how to prevent Databricks jobs from failing due to uninstalled libraries....

Job failure due to Azure Data Lake Storage (ADLS) CREATE limits

Learn what to do when your Databricks job fails due to Azure Data Lake Storage CREATE limits....

Job fails with invalid access token

Jobs that run more than 48 hours fail with invalid access token error when the dbutils token expires....

How to ensure idempotency for jobs

Learn how to ensure that jobs submitted through the Databricks REST API aren't duplicated if there is a retry after a request times out....

Streaming job has degraded performance

Streaming job has poor performance after stopping and restarting from same checkpoint....

Task deserialization time is high

Configure cluster-installed libraries to install on executors at cluster launch vs executor launch to speed up your job task runs....

Pass arguments to a notebook as a list

Use a JSON file to temporarily store arguments that you want to use in your notebook....

Uncommitted files causing data duplication

Partially uncommitted files from a failed write can result in apparent data duplication. Adjust VACUUM settings to resolve the issue....

Multi-task workflows using incorrect parameter values

If parallel tasks running on the same cluster use Scala companion objects the wrong values can be used due to sharing a single class in the JVM....

Job fails with Spark Shuffle FetchFailedException error

Disable the default Spark Shuffle service to work around a FetchFailedException error....

Users unable to view job results when using remote Git source

Databricks does not manage permission for remote repos, so you must sync changes with a local notebook so non-admin users can view results....

Single scheduled job tries to run multiple times

Ensure your cron syntax is correct when scheduling jobs. A wildcard in the wrong space can produce unexpected results....

Add custom tags to a Delta Live Tables pipeline

Manually edit the JSON configuration file to add custom tags....

Update notification settings for jobs with the Jobs API

You can use the Jobs API to add email notifications to some, or all, of the jobs in your workspace....

Stop all scheduled jobs

Use the included sample code to stop all of your scheduled jobs in the workspace....

Cluster terminates automatically even if operation or command is still executing

Consider using Databricks Jobs for long-running operations. ...

Permissions error when trying to run job clusters

Ensure that the service principal has the 'Service Principal User' role....

Idle clusters causing inefficient resource use and increased costs

Set file arrival triggers....

Error when trying to create more new jobs than the limit quota

Confirm the amount of jobs you have in your workspace, then identify and delete the jobs you do not need....

Databricks cannot access a notebook in GitHub

Check your file type or GitHub credentials and permissions....

Unable to pass a param string value of more than 65,535 characters in a workflow using a JAR in a job

Pass the param using a text file in Workspace FileSystem (WSFS). ...

Trigger a job as a specific user with "Run As"

Use the UI or API to run a job as a specific user....

Need to see job creator when investigating a job but can only see service principal

Leverage the API to retrieve job creator details....

'Cluster does not support jobs workload' error during notebook or job run

Use a cluster policy that allows the dbutils.notebooks.run API, or run the code directly within a notebook to avoid the API. ...

Getting NullPointerException when using dbutils.secrets.get in jar jobs

Include the necessary dependencies for dbutils....

Increase the number of tasks per stage

Learn how to increase the number of tasks per stage when using the spark-xml package with Databricks....

Maximum execution context or notebook attachment limit reached

Learn what to do when the maximum execution context or notebook attachment limit is reached in Databricks....

Serialized task is too large

Learn what to do when a serialized task is too large in Databricks....

Members of a Gmail group email not receiving notifications

Allow external entities to email the group inbox. ...

Apache Spark jobs failing due to stage failure when using spot instances in a cluster

Use on-demand nodes instead of spot instances. ...

Cannot import module in egg library

The module in the egg library cannot be imported. Easy install, Python....

Cannot import TabularPrediction from AutoGluon

Cannot import TabularPrediction from AutoGluon v0.0.14 due to a namespace collision. Upgrade to AutoGluon v0.0.15....

Latest PyStan fails to install on Databricks Runtime 6.4

PyStan 3 doesn't install on Databricks Runtime 6.4 ES....

Library unavailability causing job failures

Learn how to resolve Databricks job failures caused by unavailable libraries....

How to correctly update a Maven library in Databricks

Learn how to correctly update a Maven library in Databricks....

Init script fails to download Maven JAR

Cluster init script fails to download a Maven JAR when trying to install a library....

Install package using previous CRAN snapshot

Avoid a package install error by installing from an earlier CRAN snapshot....

Install PyGraphViz

Install PyGraphViz with all required dependencies....

Install Turbodbc via init script

Install Turbodbc and its dependencies, libboost-all-dev, unixodbc-dev, and python-dev, with an init script....

Cannot uninstall library from UI

Learn what to do when you can't uninstall a library using the Databricks user interface....

Error when installing Cartopy on a cluster

Cartopy installation fails if libgeos and libproj are not installed....

Error when installing pyodbc on a cluster

Learn how to troubleshoot an error when installing pyodbc on a Databricks cluster....

Libraries fail with dependency exception

Learn why notebook-scoped libraries trigger an Apache Spark dependency exception; return a requirement cannot be satisfied error....

Libraries failing due to transient Maven issue

Library resolution failed. Cannot download some libraries due to transient Maven issue....

New job fails when adding a library from DBFS or S3

New jobs fail when adding a library from DBFS or S3 storage. Error Uncaught TypeError Cannot read property 'concat' of undefined...

Reading .xlsx files with xlrd fails

xlrd no longer supports .xlsx files. Use openpyxl to read .xlsx files....

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath....

Python command fails with AssertionError: wrong color format

Resolve a wrong color format AssertionError caused by nbconvert when a Python command fails....

PyPMML fails with Could not find py4j jar error

...

TensorFlow fails to import

TensorFlow fails to import if you have an incompatible version of protobuf installed on your cluster....

Verify the version of Log4j on your cluster

Verify the version of Log4j installed on your cluster and upgrade if required....

Apache Spark jobs fail with Environment directory not found error

Spark jobs appear to time out after you install a library because security rules are preventing workers from resolving the Python executable path....

Copy installed libraries from one cluster to another

Copy libraries from a source cluster to a target cluster with a custom Python script....

Failed to install Elasticsearch via Maven

If library dependencies are already installed, it can result in a library installation failure....

Cluster fails to start with InvalidGroup.NotFound error

If the network security group policy is not correctly configured your clusters will fail to start....

Add libraries to a job cluster to reduce idle time

How to add libraries to a job cluster and reduce idle time in Databricks...

PyArrow hotfix breaking change

PyArrow versions 0.14 - 14.0.0 contain a security vulnerability....

OpenSSL SSL_connect: SSL_ERROR_SYSCALL error

Use a cluster-scoped init script to install necessary SSL certificates to resolve a SSL_ERROR_SYSCALL error....

Notebook cells fail to run with "Failure Starting repl." and Pandas "check_dependencies" errors

Ensure you do not have a dependency mismatch with the NumPy and/or Pandas versions installed on your cluster....

Virtualenv creation failure due to setuptools >= 71.0.0

Pin setuptools version 70.3.0....

Libraries failing with owner or network errors on Databricks Runtime 13.3 LTS - current (15.3)

Manually adjust your custom index URL. ...

Maven Libraries Start Failing with Timed-Out Errors When Updating to Databricks Runtime 11.3 LTS - 15.3 (current)

Whitelist Maven Central and the new Maven repo....

Paths behave differently on Git folders and workspace folders

Git folders can reference the project root, while workspace folders reference the current working directory....

GDAL library installation

Troubleshooting GDAL init script issues...

Installing lme4 fails with a Matrix version error

Upgrade the version of Matrix to 1.6.2 or above before installing lme4....

User not found error while trying to install a library on a shared cluster

Change cluster ownership and reconfigure all libraries under the new owner....

Init script to set up Dask library fails and cluster won’t start

Modify the initialization script to include a validation check for the required environment variables first....

Conda fails to download packages from Anaconda

Conda fails to download packages with PackagesNotFoundError when you try to install packages from Anaconda....

Download artifacts from MLflow

How to download artifacts from MLflow to local storage....

How to extract feature information for tree-based Apache SparkML pipeline models

Learn how to extract feature information for tree-based ML pipeline models in Databricks....

Fitting an Apache SparkML model throws error

Learn how to resolve errors thrown by Databricks when fitting a SparkML model or pipeline....

H2O.ai Sparkling Water cluster not reachable

H2O.ai Sparkling Water cluster not reachable if the version of the Sparkling Water package does not match the version of Spark used on your cluster....

How to perform group K-fold cross validation with Apache Spark

Learn how to perform group K-fold cross validation with Apache Spark on Databricks....

Error when importing OneHotEncoderEstimator

You get an error message when trying to import OneHotEncoderEstimator....

MLflow project fails to access an Apache Hive table

Resolve "Table or view not found" error when an MLflow project fails to access an Apache Hive table....

How to speed up cross-validation

Learn how to improve cross-validation performance in SparkML with Databricks....

Hyperopt fails with maxNumConcurrentTasks error

Do NOT install Hyperopt on a Databricks Runtime for Machine Learning cluster....

Incorrect results when using documents as inputs

Your model does not return expected results when documents are input using TfidfVectorizer. JSON array...

Errors when accessing MLflow artifacts without using the MLflow client

Resolve errors when attempting to access MLflow artifacts without using the MLflow client...

Experiment warning when custom artifact storage location is used

Resolve experiment warnings when a custom artifact storage location is used instead of the MLflow managed location....

Experiment warning when legacy artifact storage location is used

Resolve experiment warnings when a legacy artifact storage location is used instead of the MLflow managed location....

KNN model using pyfunc returns ModuleNotFoundError or FileNotFoundError

Predictions using pyfunc on a KNN model returns a ModuleNotFoundError or FileNotFoundError....

OSError when accessing MLflow experiment artifacts

Resolve an `OSError` when trying to access, download, or log MLflow experiment artifacts....

PERMISSION_DENIED error when accessing MLflow experiment artifact

Resolve a PERMISSION_DENIED error when trying to access MLflow experiment artifacts....

Python commands fail on Machine Learning clusters

Python commands are failing on Databricks Runtime for Machine Learning clusters. Conda....

Runs are not nested when SparkTrials is enabled in Hyperopt

When SparkTrials is enabled in Hyperopt, MLflow runs are not nested under the parent run....

MLflow 'invalid access token' error

Long running ML tasks require an access token with an extended lifetime to ensure the tasks complete before the token expires....

Unable to update a serving endpoint even with necessary permissions

Find the creating user in the workspace and re-add the permissions....

Vector search index contains incorrect number of rows

Ensure that the Delta table has a unique primary key....

ModuleNotFoundError: No module named 'packaging' when creating GPU Model Serving endpoint

Add pip_requirements during model logging. ...

Status FAILURE when attempting to update a Direct Vector Access Index

Ensure that the embedding_dimension parameter matches the length of the embedding field. ...

Table not available while creating AutoML experiment model

Migrate to Unified Compute (UC), with the option to use AutoML with the Python API in the meantime....

Failing API calls in MLflow because of Float64 column values

Explicitly log the input schema while creating the model....

Disable or restrict access to the foundation model APIs

Set rate limits to zero....

Trying to load an MLflow model using a Python script returns Py4JJavaError

Upgrade MLflow to version 2.15.0 or above. ...

TypeError with an unexpected keyword argument 'query_type' when attempting to perform hybrid similarity search using the databricks-vectorsearch package

Ensure you have the latest version of the databricks-vectorsearch package installed in your environment....

TABLE_ONLINE_VECTOR_INDEX_REPLICA does not support Lakehouse Federation

Use the Python SDK or the REST API for querying a vector search endpoint. ...

Error message when trying to send a JSON input data request to a model endpoint

Pass one more input parameter from the list of input columns in the endpoint’s model signature....

CUDA out of memory error message in GPU clusters

Change the GPU device used by your driver and/or worker nodes....

Slow model fitting when implementing Alternating Least Squares using Apache Spark PySpark

Override the block default to match the total cores available and consider using compute-optimized instances. ...

Vector search index queries with the `%` character not performing partial-string matches

When using the `LIKE` operator in vector search filters, specify the exact string you want to match...

Column name error when using Apache Spark Mlib feature transformers

When flattening the DataFrame, rename nested columns using an underscore instead of a dot....

Logging a model with MLflow in a PySpark pipeline throws a TempDir class assertion error

Upgrade your MLflow version to 2.16.0 or higher....

MLflow API 429 errors when transitioning models

Add retry logic with exponential backoff to avoid hitting the rate limit....

Model serving endpoint creation succeeds but deployment fails and error stack trace has message _ARRAY_API not found

Include NumPy as a pip dependency and specify the version range to be installed. ...

Autoscaling is slow with an external metastore

Improve autoscaling performance by only installing metastore jars to the driver....

Data too long for column error

If a column exceeds 4000 characters it is too big for the default datatype and returns an error....

Drop database without deletion

Use Hive commands to drop a database without deleting the underlying storage folder....

How to create table DDLs to import into an external metastore

Learn how to export all table metadata from Hive to an external metastore from Databricks....

Drop tables with corrupted metadata from the metastore

Learn how to drop tables that contain corrupted metadata from a metastore....

Error in CREATE TABLE with external Hive metastore

CREATE TABLE error with MySQL 8.0 in external Hive metastore due to charset....

AnalysisException when dropping table on Azure-backed metastore

Learn how to overcome an AnalysisException when dropping a table on an Azure-backed metastore....

How to troubleshoot several Apache Hive metastore problems

Learn how to troubleshoot Apache Hive metastore problems....

Listing table names

Learn how to list table names in Databricks....

How to set up an embedded Apache Hive metastore

Learn how to set up an embedded Apache Hive metastore with Databricks....

Japanese character support in external metastore

Use Japanese characters in tables in an external metastore....

Parquet timestamp requires Hive metastore 1.2 or above

Update the Hive metastore to version 1.2 or above to use TIMESTAMP with a Parquet table....

Failed to create query error when upgrading external metastore to Unity Catalog

The "Create query for upgrade" command only works when run on a warehouse in Data Explorer....

Databricks updates Glue Data Catalog during read/write operations

Upgrade to Unity Catalog. As an interim solution, turn off delta.catalog.update in Databricks....

Error while establishing user sessions to Oracle database through an external Hive metastore

Increase the session limit on your Oracle database or increase your pool size using an Apache Spark configuration....

How to explore Apache Spark metrics with Spark listeners

Learn how to explore Apache Spark metrics using Spark listeners with Databricks....

How to use Apache Spark metrics

Learn how to use Apache Spark metrics with Databricks....

Fetching the last access date/time of tables across all workspaces runs slowly and inefficiently

Databricks recommends using audit log system tables. ...

Access S3 with temporary session credentials

Extract IAM session credentials and use them to access S3 storage via S3A URI. Requires Databricks Runtime 8.3 and above....

Cannot use IAM roles with table ACL

You cannot obtain IAM credentials when table ACLs are enabled because access to the EC2 instance metadata service is blocked....

Enable s3cmd for notebooks

Use an init script to enable s3cmd for use in notebooks....

How to check if a spark property is modifiable in a notebook

Learn how to modify Spark properties in a Databricks notebook....

JSON reader parses values as null

When you read a JSON file, the Spark JSON reader returns null values instead of the actual data....

Common errors in notebooks

Learn about common errors from Databricks notebooks....

display() does not show microseconds correctly

Use show() to display timestamp values with microsecond precision. display() is limited to millisecond precision....

Error: Received command c on object id p0

You see the error message `INFO:py4j.java_gateway:Received command c on object id p0` after running Python code with imported libraries....

Failure when accessing or mounting storage

Do not mount storage to the root mount path....

Access notebooks owned by a deleted user

How to access Databricks notebooks owned by a deleted user....

Notebook autosave fails due to file size limits

Learn what to do when your Databricks notebook fails to autosave due to file size limits....

How to send email or SMS messages from Databricks notebooks

Learn how to send an email from your Databricks notebook....

Troubleshooting unresponsive Python notebooks or canceled commands

Learn how to troubleshoot unresponsive Python notebooks and cancelled commands in Databricks notebooks....

Update job permissions for multiple users

Use the job permissions API to update permissions for multiple users....

Service principal cannot create access token

You cannot create a token on behalf of a service principal with the API when token usage is disabled....

Generate browser HAR files

Learn how to record HAR files in your web browser. These are very useful when troubleshooting UI issues....

Recover deleted notebooks from the Trash

Deleted items can be recovered from the Trash for 30 days after deletion....

Get workspace configuration details

Display the complete configuration details for your Databricks workspace....

Iterate through all jobs in the workspace using Jobs API 2.1

Use the Jobs API 2.1 to iterate through and display a list of jobs in your workspace....

Too many execution contexts are open right now

Reduce the number of notebooks used to limit the number of execution contexts required for your job....

Generate a list of all workspace admins

Use the included sample code to generate a list of all workspace admins....

Forbidden error while accessing S3 data

...

Table creation fails with security exception

Learn what to do when table creation fails with a security exception....

Set an unlimited lifetime for service principal access token

Configure an extended or unlimited lifetime for a service principal access token....

Connection retries take a long time to fail

The default Apache Hadoop values for connection timeout and retry are high, reduce the values for quicker failures....

Search audit logs for connections from prohibited IP addresses

Use audit logs to review and validate connection attempts to your workspace....

"Unable to update Group Push mapping target" error when syncing Okta groups to workspace

You cannot SCIM sync users and groups directly to the workspace when identity federation is enabled....

Bulk update workflow permissions for a group

Use this sample code to update a single group's permissions for all the jobs in a workspace....

Restricting sensitive data in the workspace

Store your data in a way that makes permissions management easier....

CORS policy error when trying to run Databricks API from a browser-based application

Implement a server-server communication model. ...

Append output is not supported without a watermark

Append output mode is not supported on aggregated DataFrames without a watermark....

Apache Spark DStream is not supported

DStreams are not supported in Databricks. Migrate from DStream API to Structured Streaming....

Streaming with File Sink: Problems with recovery if you change checkpoint or output directories

Learn how to resolve issues that occur with recovery if you change checkpoint or output directories when streaming with File Sink....

Get the path of files consumed by Auto Loader

Get the path and filename of all files consumed by Auto Loader and write them out as a new column....

How to set up Apache Kafka on Databricks

Learn how to set up Apache Kafka on Databricks....

Handling partition column values while using an SQS queue as a streaming source

...

How to restart a structured streaming query from last written offset

Learn how to restart a structured streaming query from the last written offset....

How to switch a SNS streaming job to a new SQS queue

...

Kafka error: No resolvable bootstrap urls

A 'No resolvable bootstrap urls' error occurs when you try to read or write data to a Kafka stream....

readStream() is not whitelisted error when running a query

readStream() is not whitelisted error on clusters that have table access control enabled....

Checkpoint files not being deleted when using display()

Learn how to prevent display(streamingDF) checkpoint files from using a large amount of storage....

Checkpoint files not being deleted when using foreachBatch()

Learn how to prevent foreachBatch() checkpoint files from using a large amount of storage....

Conflicting directory structures error

You should use distinct paths in the storage location, otherwise conflicting directory structures may result in an error....

RocksDB fails to acquire a lock

When using RocksDB as a state store, you may need to increase the acquire timeout in the SQL config....

Stream XML files using an auto-loader

Stream XML files on Databricks by combining the auto-loading features of the Spark batch API with the OSS library Spark-XML....

Streaming job using Kinesis connector fails

A streaming job writing to a Kinesis sink fails with out of memory error because HTTP clients are not getting terminated....

Streaming job gets stuck writing to checkpoint

Streaming job appears to be stuck even though no error is thrown. You are using DBFS for checkpoint storage, but it has filled up....

Explicit path to data or a defined schema required for Auto loader

If you do not specify an explicit path to your data or define your data schema, you get an IllegalArgumentException error when you start an Auto loader job....

Optimize streaming transactions with .trigger

Use .trigger to define the storage update interval. A higher value reduces the number of storage transactions....

Structured streaming jobs slow down on every 10th batch

Automatic compaction of the metadata folder can slow down structured streaming jobs....

Get last modification time for all files in Auto Loader and batch jobs

Define a UDF to list all files in the path and return the last modification time for each one....

Stream to stream join failure

Avoid using a memory sink when running streaming queries with stream to stream join....

Offset reprocessing issues in streaming queries with a Kafka source

Resolve Kafka offset reprocessing issues in Structured Streaming by using a new checkpoint directory....

Autoloader job fails with a URISyntaxException error due to invalid characters in filenames

When using Directory listing mode you should not process files with colons in the filename. ...

Auto Loader streaming job failure with schema inference error

To selectively read a specific type of file using Auto Loader, use the pathGlobFilter option....

Auto Loader streaming query failure with unknownFieldException error

Use schema evolution to avoid streaming query failures when new columns are added to your data....

Casting string to date/timestamp in DLT pipeline does not throw an error

Configure the Delta Live Tables pipeline to enforce ANSI SQL compliance. ...

Incorrect numInputRows values even with row limits set per batch

Use endOffset instead of latestOffset to calculate the number of records read. ...

Auto Loader fails to pick up new files when using directory listing mode

Use file notification mode or disable incremental listing....

Incorrect input record count in Apache Spark streaming application logs/micro-batch metrics

Optimize actions on the DataFrame within the foreachBatch function. ...

Upgrading to 14.3 LTS gives the error "com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException"

Choose to configure either manually or via schema evolution....

Streaming application missing data from a Delta table when writing to a given destination

Restart a streaming query on a new checkpoint folder with startingVersion option pointing to the next Delta version (X+1)....

DatabricksS3LoggingException errors while attempting to read a binary file in an Apache Spark structured streaming job

Disable the delta format check and ensure correct IAM role setup. ...

How to efficiently manage state store files in Apache Spark streaming applications

Control the lifecycle of state store files using streaming configurations...

Auto Loader failures with java.io.FileNotFoundException for SST and log files

Use a separate checkpoint folder outside of the Delta directory....

Auto Loader job fails saying metadata file is missing from checkpoint directory

Recover or recreate the missing metadata file, and then ensure the checkpoint location is not in a bucket with a lifecycle policy enabled. ...

Stateful Structured Streaming jobs fail after making changes to stateful operations

Avoid changes to stateful operations between restarts....

Structured Streaming job fails with a Streaming Query Exception when a schema changes in the source table

Enable schema tracking and set allowSourceColumnRenameAndDrop to true....

Duplicates appearing in Auto Loader with file notification feature despite set backfill interval

Remove the allowOverwrites configuration or implement a deduplicate logic....

writeStream/readStream leads to an error when the schema contains “NullType”

Replace NullType with a literal None value and then cast it to StringType. ...

How to save Plotly files and display From DBFS

Learn how to save Plotly files and display them from DBFS....

AttributeError: ‘function’ object has no attribute

Using protected keywords from the DataFrame API as column names results in a function object has no attribute error message....

Convert Python datetime object to string

Display date and time values in a column, as a datetime object, and as a string....

Display file and directory timestamp details

Display file creation date and modification date using Python....

Install and compile Cython

Learn how to install and compile Cython with Databricks....

Reading large DBFS-mounted files using Python APIs

Learn how to resolve errors when reading large DBFS-mounted files using Python APIs....

Use the HDFS API to read files in Python

Learn how to read files directly by using the HDFS API in Python....

How to import a custom CA certificate

Learn how to import a custom CA certificate into your Databricks cluster for Python use....

Job remains idle before starting

Apache Spark jobs remain idle for a long time before starting....

List all workspace objects

List all Databricks workspace objects under a given path....

Load special characters with Spark-XML

Special characters are not rendering correctly. Use charset with Spark-XML....

Python commands fail on high concurrency clusters

Python commands fail on high concurrency clusters with Apache Spark process isolation and shared session enabled. WARN error message....

Cluster cancels Python command execution after installing Bokeh

Learn what to do when your Databricks cluster cancels Python command execution after you install Bokeh....

Cluster cancels Python command execution due to library conflict

Learn what to do when your Databricks cluster cancels Python command execution due to a library conflict....

Python command execution fails with AttributeError

Learn what to do when a Python command in your Databricks notebook fails with AttributeError....

Python REPL fails to start in Docker

Learn how to fix a Python virtualenv error that prevents REPL from starting in a Docker container...

How to run SQL queries from Python scripts

Learn how to run SQL queries using Python scripts....

Run C++ code in Python

Learn how to run C++ code in Python....

Python 2 sunset status

Learn about the sunset status of Python 2 in Databricks....

Job fails with Java IndexOutOfBoundsException error

When groupby() is used along with applyInPandas it generates an exception due to an arrow buffer limitation....

Job fails with NoSuchElementException error

NoSuchElementException errors can occur when using Apache Arrow....

Job fails with IndexOutOfBoundsException and ArrowBuf errors

When Groupby is used with applyinPandas it can result in Apache Arrow buffer size estimation errors....

Field name sorting changes in Apache Spark 3.x

Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically....

Job fails with "not enough memory to build the hash map" error

You should use adaptive query execution instead of explicit broadcast hints to perform joins on Databricks Runtime 11.3 LTS and above....

Create a DataFrame from a JSON string or Python dictionary

Create an Apache Spark DataFrame from a variable containing a JSON string or a Python dictionary....

Apache Spark driver stops and restarts while reading JSON file data in an S3 bucket

Use a predefined schema to read the files, or read folder contents as text instead. ...

Behavioral changes for the CHAR data type on Serverless

Pad your reads with spaces to match the declared length of the CHAR field or set the legacy charVarcharAsString config to true....

Column value errors when connecting from Apache Spark to Databricks using Spark JDBC

Use overriding quote identifiers in the JdbcDialect class and register them under JDBCDialects in Java or Python....

Trying to decode a protocol buffer and getting error [PROTOBUF_DEPENDENCY_NOT_FOUND]

Use the option --include_imports while creating the protobuf descriptor file, and then use this descriptor file in the from_protobuf() function....

Error java.io.FileNotFoundException when job attempts to read or write intermediary files

Cache the DataFrame before performing write operations and ensure you are using a compatible version of the com.crealytics.spark.excel library. ...

Using Pyspark testing library assertDataFrameEqual throws OutOfMemoryError

Verify schemas are equivalent, then either ensure sufficient driver memory or compare a subset of DataFrames. ...

Expensive transformation on DataFrame is recalculated even when cached

Understand how Apache Spark DataFrame caching works....

Change version of R (r-base)

Learn how to change the version of R on your Databricks cluster....

Fix the version of R packages

Learn how to fix the version of R packages....

How to parallelize R code with gapply

Learn how to parallelize R code using gapply....

How to parallelize R code with spark.lapply

Learn how to parallelize R code using spark.lapply....

How to persist and share code in RStudio

Learn how to share notebooks between Databricks and RStudio....

Install rJava and RJDBC libraries

Learn how to install rJava and RJDBC libraries on your Databricks cluster....

Rendering an R markdown file containing sparklyr code fails

Learn how to resolve failures when rendering an R markdown file containing sparklyr....

Resolving package or namespace loading error

Learn how to resolve package or namespace loading errors in a Databricks notebook....

RStudio server backend connection error

RStudio server backend connection error occurs if you exceed the maximum number of RBackends on your cluster....

Verify R packages installed via init script

Verify that R packages successfully installed via an init script. List all R packages that failed to install....

Apache Spark UI is not in sync with job

Status of Spark jobs gets out of sync with the Spark UI when events drop from the event queue before being processed....

Apache Spark job fails with Parquet column cannot be converted error

Parquet column cannot be converted error appears when you are reading decimal data in Parquet format and writing to a Delta table....

Cannot import timestamp_millis or unix_millis

Cannot use timestamp_millis or unix_millis directly with a DataFrame. You must first use selectExpr() or use SQL commands....

Cannot modify the value of an Apache Spark config

You cannot modify the value of a Spark config setting within a notebook. It must be set at the cluster level....

Convert flattened DataFrame to nested JSON

How to convert a flattened DataFrame to nested JSON using a nested case class....

Convert nested JSON to a flattened DataFrame

How to convert a flattened DataFrame to nested JSON using a nested case class....

Decimal$DecimalIsFractional assertion error

Using `round()` or casing a double to decimal results in a `Decimal$DecimalIsFractional` assertion error. java.lang.AssertionError assertion failed...

from_json returns null in Apache Spark 3.0

Spark 3.0 and above cannot parse JSON arrays as structs; from_json returns null....

Intermittent NullPointerException when AQE is enabled

When adaptive query execution (AQE) is enabled, and cluster scales down and loses shuffle data, you can get a `NullPointerException` error....

Manage the size of Delta tables

Recommendations that can help you manage the size of your Delta tables....

Trouble reading external JDBC tables after upgrading from Databricks Runtime 5.5

Fail to read external JDBC tables after upgrading from Databricks Runtime 5.5 to 6.0 and above....

Running C++ code in Scala

Learn how to run C++ code in Scala with this example notebook....

Select files using a pattern match

Use a glob pattern match to select specific files in a folder....

Multiple Apache Spark JAR jobs fail when run concurrently

Apache Spark JAR jobs failing with an AnalysisException error when run concurrently....

Write a DataFrame with missing columns to a Redshift table

Use `FILLRECORD` to populate missing columns when writing a DataFrame to a Redshift table....

Job fails with ExecutorLostFailure due to “Out of memory” error

Resolve executor failures where the root cause is due to the executor running out of memory.....

Job fails with ExecutorLostFailure because executor is busy

Resolve executor failures where the root cause is due to the executor being busy....

Understanding speculative execution

Learn how speculative execution works, how to identify it, and when you should use it....

Use custom classes and objects in a schema

You must define custom classes and objects inside a package if you want to use them in a notebook. ...

Jobs fails with a TimeoutException error

This error is usually caused by a Broadcast join that takes excessively long to complete....

Sort failed after writing partitioned data to parquet using PySpark on Databricks Runtime 13.3 LTS

Set the Apache Spark configuration to set the sorted data after writing partitioned data to parquet....

Reading Avro files with Structured Streaming using wildcards in the path fails with error ArrayIndexOutOfBoundsException

Add an option to enable recursively reading bulk Avro files using a wildcard path....

WithColumn operation when using in-loop slows performance

Use the select operator instead....

Broadcast join exceeds threshold, returns out of memory error

Resolve an Apache Spark OutOfMemorySparkException error that occurs when a table using BroadcastHashJoin exceeds the BroadcastJoinThreshold....

Cannot grow BufferHolder; exceeds size limitation

Cannot grow BufferHolder by size because the size after growing exceeds limitation; java.lang.IllegalArgumentException error....

Date functions only accept int values in Apache Spark 3.0

Date functions only accept int values in Apache Spark 3.0; fractional and string values return AnalysisException error....

Disable broadcast when query plan has BroadcastNestedLoopJoin

How to disable broadcast when the query plan has BroadcastNestedLoopJoin....

Duplicate columns in the metadata error

Spark job fails while processing a Delta table with org.apache.spark.sql.AnalysisException Found duplicate column(s) in the metadata error....

Generate unique increasing numeric values

Use Apache Spark functions to generate unique and increasing numbers in a column in a table in a file or DataFrame....

Error in SQL statement: AnalysisException: Table or view not found

Learn how to resolve the AnalysisException SQL error "Table or view not found"....

Error when downloading full results after join

If you have duplicate columns after a join, you will get an error when trying to download the full results....

Error when running MSCK REPAIR TABLE in parallel

Do not run `MSCK REPAIR` commands in parallel. It results in a read timed out or out of memory error message....

Find the size of a table

How to find the size of a table....

Inner join drops records in result

Avoid dropped records when performing an inner join....

Data is incorrect when read from Snowflake

Data read from Snowflake is incorrect when time zone value is not set correctly....

JDBC write fails with a PrimaryKeyViolation error

JDBC write to a SQL database fails with a `PrimaryKeyViolation` error or results in duplicate data...

Query does not skip header row on external table

External Hive tables do not skip the header row when queried from Spark SQL....

SHOW DATABASES command returns unexpected column name

Running the `SHOW DATABASES` command returns an unexpected column name....

Cannot view table SerDe properties

SHOW CREATE TABLE only returns the Apache Spark DDL. It does not show the SerDe properties....

Parsing post meridiem time (PM) with to_timestamp() returns null

When converting 12-hour time to 24-hour time with to_timestamp() the hours variable must be lowercase....

to_json() results in Cannot use null as map key error

You must filter or replace null values in your input data before using to_json()....

Set nullability when using SaveAsTable with Delta tables

Learn how to create a Delta table with the nullability of columns set to false....

Ensure consistency in statistics functions between Spark 3.0 and Spark 3.1 and above

Statistics functions in Databricks Runtime 7.3 LTS and below return NaN when a divide by zero occurs. Set a Spark config to return null instead....

Using datetime values in Spark 3.0 and above

How to correctly use datetime functions in Spark SQL with Databricks runtime 7.3 LTS and above....

ANSI compliant DECIMAL precision and scale

Learn how to enable ANSI compliant error messages when incorrect values are used for DECIMAL precision and scale....

Recreate LISTAGG functionality with Spark SQL

Use collect_list and concat_ws in Spark SQL to achieve the same functionality as LISTAGG on other platforms....

Decreased performance when using DELETE with a subquery on Databricks Runtime 10.4 LTS

Auto optimize should be disabled when you have a DELETE with a subquery where one side is small enough to be broadcast....

Automatic VACUUM on write does not work with non-Delta tables

Manually run VACUUM to clear uncommitted files from the entire table....

LEFT JOIN resulting in null values when joining timestamp column and date column

Cast the value of the timestamp column to date datatype when joining it with a column of 'date' datatype....

Handling case sensitivity issues in Delta Lake nested fields

Set a specific property in your Spark configuration to handle the case sensitivity of nested fields in Delta tables. ...

Error when creating a user, group, or service principal at the account level with Terraform

You must include your account_id in the Terraform Databricks provider block to manage users, groups, and service principals....

Identity federation is not enabled in workspaces created with Terraform

You need to assign a metastore when creating the workspace to enable identity federation....

Dependency error while deploying workspace using Terraform

Add “depends_on” clause to ensure identity federation is enabled before identity assignment....

Members not supported SCIM provisioning failure

You get a members or groups not supported error when trying to provision new users to your workspace via SCIM....

Cannot delete Unity Catalog metastore using Terraform

Set force_destroy in the databricks_metastore section of the Terraform configuration to delete a metastore and its catalog....

Permission denied error when creating external location

You must be a metastore admin or have the CREATE EXTERNAL LOCATION privilege....

Overlapping paths error when querying both Hive and Unity Catalog tables

If a Hive table and a Unity Catalog table both refer to the same external storage path you cannot query them in the same notebook cell....

Failed to create query error when upgrading external metastore to Unity Catalog

The "Create query for upgrade" command only works when run on a warehouse in Data Explorer....

SYNC command fails with a mismatched input error

Unity Catalog must be enabled on your cluster before you can use SYNC to migrate a legacy Hive table to Unity Catalog....

Unable to access Unity Catalog views

You cannot access a view if you do not have sufficient permissions on the underlying table....

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error

External tables cannot overlap with catalog/schema storage locations. They should be created in a subdirectory instead....

Cluster-scoped init script to Unity Catalog volume migration notebook

Easily migrate your cluster-scoped init scripts to Unity Catalog volumes....

List all available tables and their source formats in Unity Catalog

Use system.information_schema.tables to display available table names and their data source formats....

No way to restore dropped managed volumes

To reduce risk, always use external volumes for operations that require frequent schema changes. ...

Unsupported path error when creating an external table

Create the table in a different location that does not overlap with the external volume. ...

Azure Synapse Analytics (formerly SQL Data Warehouse) federated queries throwing "Invalid object name 'dbo.date'" error

Set the collation for the database in the dedicated SQL pool to "SQL_Latin1_General_CP1_CI_AS". ...

You get an Insufficient privileges on __databricks_internal catalog error when attempting to query DLT pipeline views

You should not access the __databricks_internal catalog directly. It is an internal catalog used for DLT pipeline materializations....

Failure to initialize configuration for storage account while registering a model serving model in Unity Catalog

An external location must be provisioned during the initial Unity Catalog configuration....

Find your metastore ID

Step-by-step instructions to find the metastore ID....

Access denied error when trying to read or write to an S3 bucket in a separate AWS account

Disable ACLs in the BucketOwnerEnforced setting in S3 Object Ownership, or ensure that access to the objects in the S3 bucket is permitted....

Unable to view, delete, or drop an external location in the UI or through commands, even with admin privileges

Use metastore admin rights to change ownership to yourself or the required user, and then manage. ...

Error [DELTA_UNSUPPORTED_TIME_TRAVEL_MULTIPLE_FORMATS] while creating VIEW

Use a three-level namespace to CREATE VIEW based on UC table version. ...

Circular reference error when trying to apply row filter function in Databricks SQL analytics

Remove the circular reference in the row filter configuration....

Comments not reflecting on Unity Catalog tables

Update the Apache Spark configuration to enable automatic metadata updates. ...

Job failing when trying to write dataset to external path

Clarify which table is using which location on a selected schema. ...

Cannot select a Databricks Runtime version when using a Delta Live Tables pipeline

Delta Live Tables do not allow you to directly configure the Databricks Runtime version....

Delta Live Tables job fails when using collect()

You should not use functions such as collect(), count(), toPandas(), save(), and saveAsTable() within the table and view function definitions....

Handling WARN Message: 'Could not turn on CDF for table (table-name)' in Delta Live Tables Pipeline

Exclude and then drop reserved columns. ...

Handling Data Duplication Issues with Databricks Autoloader and Delta Lake using replaceWhere

Ensure that the replaceWhere option is applied during the write operation...

Unclear how to control micro-batch size on a streaming table in Delta Live Tables (DLT)

Use the rate limiters along with the keyword LIVE....

Permissions issue when trying to access DLT-managed streaming tables and materialized views in Unity Catalog

Grant required Unity Catalog permissions and apply a compatible compute type. ...

applyInPandasWithState fails with a ModuleNotFoundError when used with Delta Live Tables

Define a function within the notebook instead of calling it directly....