Enable OpenJSSE and TLS 1.3

Add OpenJSSE to allow the use of TLS 1.3 for encrypted data transmission....

How to calculate the number of cores in a cluster

Learn how to calculate the number of cores in a Databricks cluster....

Install a private PyPI repo

How to install libraries from private PyPI repositories....

IP access list update returns INVALID_STATE

Cannot update IP access list. INVALID_STATE error message....

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Cluster failed to launch

Learn how to resolve cluster launch failures....

Custom Docker image requires root

Custom Docker containers must be configured to start as the root user when used with Databricks....

Job fails due to cluster manager core instance request limit

Learn how to troubleshoot Databricks errors related to API rate limits....

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Persist Apache Spark CSV metrics to a DBFS location

Persist Spark CSV metrics to a sink in a DBFS location....

Replay Apache Spark events in a cluster

Use a single node cluster to replay another cluster's event log in the Spark UI....

Set Apache Hadoop core-site.xml properties

Set Apache Hadoop core-site.xml properties in a Databricks cluster....

Set executor log level

Learn how to set the log levels on Databricks executors....

Apache Spark job doesn’t start

Learn how to troubleshoot a Databricks Spark job that won't start....

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

Unexpected cluster termination

Learn how to troubleshoot a Databricks cluster that stopped unexpectedly....

How to configure single-core executors to run JNI libraries

Learn how to configure single-core executors to run JNI libraries on Databricks....

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Apache Spark UI shows less than total node memory

Learn what to do when the Spark UI shows less memory than is actually available on the node....

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Enable retries in init script

Add a retry function to your init script....

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Run a custom Databricks Runtime on your cluster

Configure your cluster to run a custom Databricks Runtime image via the UI or API....

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Slow cluster launch and missing nodes

Learn how to resolve a "nodes could not be acquired" error when starting a Databricks ....

IP address limit prevents cluster creation

Learn how to fix a public IP address quota limit Cloud Provider Launch error when starting a Databricks cluster....

CPU core limit prevents cluster creation

Learn how to fix a CPU core quota limit Cloud Provider Launch error when starting a Databricks cluster....

Custom garbage collection prevents cluster launch

Using a custom garbage collection algorithm on Databricks Runtime 10.0 and above prevents the cluster from starting....

SSH to the cluster driver node

How to SSH to the Apache Spark cluster driver node in an Azure virtual network...

Adding a configuration setting overwrites all default spark.executor.extraJavaOptions settings

Learn how to resolve overwritten configuration settings in Databricks....

UnknownHostException on cluster launch

Troubleshoot an UnknownHostException on cluster launch. This is often a DNS configuration issue....

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

R commands fail on custom Docker cluster

R version 4.2.0 changed the way Renviron.site is initialized, so you must set an environment variable when using custom Docker clusters....

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Legacy global init script migration notebook

Easily migrate your legacy global init scripts to the current global init script framework....

Python kernel is unresponsive error message

Learn how to identify and troubleshoot the cause of an unresponsive Python kernel error....

Spark image download failure error message

Learn how to troubleshoot the Spark image download failure error message....

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

Shorten cluster provisioning time by using Docker containers

Learn how to speed up cluster provisioning by using Docker container services...

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files....

Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Embed system.exit code in your application to shutdown the Java virtual machine with exit code 0....

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Modify your log4j2 configuration file directly within the Databricks environment. ...

Ganglia metrics not appearing in historical metrics snapshots list

Configure the collection period, or update Databricks Runtime to at least 13.0 to use Compute metrics instead. ...

Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes. ...

Init script stored on a volume fails to execute on cluster start

Init scripts created on Windows systems and uploaded to Unity Catalog volumes have CRLF as a newline which needs to be converted to LF before the cluster can process it. ...

Databricks API last_activity_time attribute shows incorrect timestamp

Use the cluster auto-termination feature to manage cluster termination based on inactivity....

BROADCAST_VARIABLE_NOT_LOADED or JVM_ATTRIBUTE_NOT_SUPPORTED errors when using broadcast variables in a shared access mode cluster

Use a single-user cluster or pass a variable into a function as a state instead. ...

404 error when installing krb5-user module

Manually remove the var directory path to refresh the cached data. ...

Cluster startup failure while running proxy-configured init script with other init scripts

Modify the proxy init script to bypass local addresses....

Init scripts failing with unexpected end of file error

Remove the special characters from the init script....

Unable to access the hive_metastore schema

Ensure all clusters use the same Hive metastore version and Apache Spark configurations are set. ...

Job fails while installing ODBC Driver 18 for SQL Server using an init script

Add msodbcsql18 to the LD_LIBRARY_PATH then append LD_LIBRARY_PATH path to /etc/environment....

Error when trying to use Apache Spark’s Pyspark offset method on DataFrames with serverless compute

Use the limit method or the monotonically_increasing_id() function instead....

Jobs failing with schema conversion error: cannot convert Parquet type INT32 to Photon type long

Set spark.databricks.photon.scan.enabled to false....

Cannot access Apache SparkContext object using addPyFile

Leverage the addArtifact API instead....

Cluster fails to launch with a Bootstrap Timeout error

Verify allowlisting of necessary services and correct configuration of your VPC/VNet. ...

Cluster fails to launch with error, “user specified an invalid argument”

Change the cluster owner to an active user....

Job executions failing on clusters using Docker Container Services with MalformedInputException error

Specify the correct character encoding when reading the file and change the LANG settings. ...

Enabling Dynamic Allocation leads to NODES_LOST scenario

Enable Autoscaling when you create a Databricks cluster. ...

Clusters using Docker databricksruntime:latest tag are not starting

Replace latest tag with LTS for Databricks Docker image....

Jobs failing with BindException error after upgrading to Databricks Runtime 11.3 LTS or above

Change the ipywidgets default port to another available port....

Cluster fails to initialize after a Databricks Runtime upgrade

Check your init scripts and then your Apache Spark configurations....

Missing the audit log event of a cluster deletion

Pin your cluster and monitor usage....

Increased job execution time after migrating from all-purpose to job cluster

Increase the Hive client pool size in the job cluster configuration to match the previous all-purpose compute setting....

NODES_LOST error during cluster upsizing when Apache Spark dynamic allocation is enabled

Remove the spark.dynamicAllocation.enabled Spark config from the compute configuration....

Listing Hive metastore tables in Catalog Explorer failing with error getting schemas

Remove the Apache Spark configuration "spark.databricks.session.share true". ...

Change the minor version of Python in a cluster

Use an init script to install the desired versions of Python and pyenv....

Row value assignments not reflecting expected output in code that loops through temporary views

Avoid using temporary views with the same name when using loops in Spark Connect. ...

Unresolved column error when using Apache Spark Connect to run a query to create a temporary view

Use unique names for each temporary view. ...

scala.collection.immutable.HashMap$HashMap1 class leading to OOM error in driver

Change the webUI events location to RockDB....

Unable to attach init scripts to an interactive cluster

Ensure the cluster creator exists in the workspace and has the necessary group memberships....

How to restrict selection of specific Databricks runtimes in the compute creation UI

Implement a regular expression (regex) validation to ensure only certain selections are available....

Cannot select a compute policy for a DLT Pipeline

Ensure that you have permission to use the policy and are using the correct compute type....

How to restrict cluster creation to single-node only

Add a JSON configuration to your compute policy....

Workflows are failing with a 'Could not reach driver of the cluster' error

Use a larger driver instance or increase the REPL timeout....

Cannot create cluster: spark conf: 'spark.databricks.cluster.profile' is not allowed when choosing an access mode

Use the flag is_single_node to create single node compute....

Launch the web terminal on a DCS enabled cluster

Use ttyd to launch the web terminal with a custom Docker image on a cluster....

FileNotFoundError when trying to use Android Development Bridge (ADB) command line tool on a cluster

Explicitly install the ADB tool on your cluster and set in the correct system path....

Spark UI is empty for the job clusters after termination

For non-Spark tasks the Spark UI should be empty....

502 error when trying to access the Spark UI

Enable Spark UI to store its data on disk instead of in memory....

Pulumi fails to deploy workflows in serverless mode

Update the Pulumi provider, check for configuration errors, and remove the job cluster definition from the payload....

Unable to run interactive workloads using a dedicated (formerly single user) compute assigned to a service principal where you have the Service Principal User role

Use a dedicated compute linked to a user or group instead, or switch to standard compute to run interactive workloads....

Init script stored in volume fails with permission denied error

Ensure that the cluster owner has access to the volume and the init script....