Install a private PyPI repo

How to install libraries from private PyPI repositories....

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Set executor log level

Learn how to set the log levels on Databricks executors....

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Enable retries in init script

Add a retry function to your init script....

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Run a custom Databricks Runtime on your cluster

Configure your cluster to run a custom Databricks Runtime image via the UI or API....

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files....

Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Embed system.exit code in your application to shutdown the Java virtual machine with exit code 0....

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Modify your log4j2 configuration file directly within the Databricks environment. ...

Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes. ...

Cluster fails with a DRIVER_EVICTION error

Ensure that the driver instances are not running on preemptive nodes....

Init script stored on a volume fails to execute on cluster start

Init scripts created on Windows systems and uploaded to Unity Catalog volumes have CRLF as a newline which needs to be converted to LF before the cluster can process it. ...

Databricks API last_activity_time attribute shows incorrect timestamp

Use the cluster auto-termination feature to manage cluster termination based on inactivity....

BROADCAST_VARIABLE_NOT_LOADED or JVM_ATTRIBUTE_NOT_SUPPORTED errors when using broadcast variables in a shared access mode cluster

Use a single-user cluster or pass a variable into a function as a state instead. ...

404 error when installing krb5-user module

Manually remove the var directory path to refresh the cached data. ...

Cluster startup failure while running proxy-configured init script with other init scripts

Modify the proxy init script to bypass local addresses....

Init scripts failing with unexpected end of file error

Remove the special characters from the init script....

Unable to access the hive_metastore schema

Ensure all clusters use the same Hive metastore version and Apache Spark configurations are set. ...

Job fails while installing ODBC Driver 18 for SQL Server using an init script

Add msodbcsql18 to the LD_LIBRARY_PATH then append LD_LIBRARY_PATH path to /etc/environment....

Error when trying to use Apache Spark’s Pyspark offset method on DataFrames with serverless compute

Use the limit method or the monotonically_increasing_id() function instead....

Jobs failing with schema conversion error: cannot convert Parquet type INT32 to Photon type long

Set spark.databricks.photon.scan.enabled to false....

Cannot access Apache SparkContext object using addPyFile

Leverage the addArtifact API instead....

Cluster fails to launch with a Bootstrap Timeout error

Verify allowlisting of necessary services and correct configuration of your VPC/VNet. ...

Cluster fails to launch with error, “user specified an invalid argument”

Change the cluster owner to an active user....

Job executions failing on clusters using Docker Container Services with MalformedInputException error

Specify the correct character encoding when reading the file and change the LANG settings. ...

Enabling Dynamic Allocation leads to NODES_LOST scenario

Enable Autoscaling when you create a Databricks cluster. ...

Jobs failing with BindException error after upgrading to Databricks Runtime 11.3 LTS or above

Change the ipywidgets default port to another available port....

Cluster fails to initialize after a Databricks Runtime upgrade

Check your init scripts and then your Apache Spark configurations....

Receiving “no space left on device error” message when attempting to use Apache Spark

Switch to a VM instance with attached local storage or increase the size of the VM boot disk....

Missing the audit log event of a cluster deletion

Pin your cluster and monitor usage....

Increased job execution time after migrating from all-purpose to job cluster

Increase the Hive client pool size in the job cluster configuration to match the previous all-purpose compute setting....

NODES_LOST error during cluster upsizing when Apache Spark dynamic allocation is enabled

Remove the spark.dynamicAllocation.enabled Spark config from the compute configuration....

Listing Hive metastore tables in Catalog Explorer failing with error getting schemas

Remove the Apache Spark configuration "spark.databricks.session.share true". ...

Change the minor version of Python in a cluster

Use an init script to install the desired versions of Python and pyenv....

Row value assignments not reflecting expected output in code that loops through temporary views

Avoid using temporary views with the same name when using loops in Spark Connect. ...

Unresolved column error when using Apache Spark Connect to run a query to create a temporary view

Use unique names for each temporary view. ...

scala.collection.immutable.HashMap$HashMap1 class leading to OOM error in driver

Change the webUI events location to RockDB....

Unable to attach init scripts to an interactive cluster

Ensure the cluster creator exists in the workspace and has the necessary group memberships....

How to restrict selection of specific Databricks runtimes in the compute creation UI

Implement a regular expression (regex) validation to ensure only certain selections are available....

Cannot select a compute policy for a DLT Pipeline

Ensure that you have permission to use the policy and are using the correct compute type....

How to restrict cluster creation to single-node only

Add a JSON configuration to your compute policy....

Workflows are failing with a 'Could not reach driver of the cluster' error

Use a larger driver instance or increase the REPL timeout....

Cannot create cluster: spark conf: 'spark.databricks.cluster.profile' is not allowed when choosing an access mode

Use the flag is_single_node to create single node compute....

FileNotFoundError when trying to use Android Development Bridge (ADB) command line tool on a cluster

Explicitly install the ADB tool on your cluster and set in the correct system path....

Spark UI is empty for the job clusters after termination

For non-Spark tasks the Spark UI should be empty....

502 error when trying to access the Spark UI

Enable Spark UI to store its data on disk instead of in memory....

Pulumi fails to deploy workflows in serverless mode

Update the Pulumi provider, check for configuration errors, and remove the job cluster definition from the payload....

Unable to run interactive workloads using a dedicated (formerly single user) compute assigned to a service principal where you have the Service Principal User role

Use a dedicated compute linked to a user or group instead, or switch to standard compute to run interactive workloads....

Init script stored in volume fails with permission denied error

Ensure that the cluster owner has access to the volume and the init script....

Using Vertica Spark Connector to write a DataFrame to external database fails with [TABLE_OR_VIEW_NOT_FOUND] error

Switch to a non-Unity Catalog cluster....

XML file read executes slowly

Define the schema explicitly. ...

Performing count on Delta table using dedicated vs standard compute

Using a dedicated compute, you will be able to get the expected output by performing the count on the version of the Delta table from which the associated files are removed....

How to gather total cluster duration

Run a SQL query....

UnsupportedClassVersionError when running a Databricks job on Graviton machines

Specify an appropriate ARM architecture instead of AMD....

Cannot deploy the agent model with a compute resource assigned to a group

Use Dedicated access mode compute assigned to a user or service principal....

Cluster’s Apache Spark UI not appearing

Append the default Databricks daemon listener to your custom listener....

Unable to connect endpoints via non http/https port number when using standard (formerly shared) access mode

Use Databricks Runtime 12.2 LTS or above, or use an init script to open a required port....

How to install Python packages from a private PyPI repository on Databricks

Configure a cluster-wide index URL or install the package using a notebook cell....

Connection to external sources after configuring proxy settings is failing

Allowlist the required FQDNs, remove unnecessary Apache Spark configuration properties, check your proxy settings, and verify connectivity....

Job run fails with error message “Could not reach driver of cluster”

Increase the REPL launch timeout....

Standard (formerly shared) cluster not allowing use of Machine Learning Runtime

Use a dedicated compute with a group including the list of users....

Error: PERMISSION_DENIED when running notebooks with a group-assigned dedicated compute

Grant workspace access entitlement to the group....