Databricks Help Center

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Clusters (Azure)

These articles can help you manage your Apache Spark clusters.

95 Articles in this category

  • Home
  • All articles
  • Clusters (Azure)

Enable OpenJSSE and TLS 1.3

Add OpenJSSE to allow the use of TLS 1.3 for encrypted data transmission....

Last updated: March 2nd, 2022 by Adam Pavlacka

How to calculate the number of cores in a cluster

Learn how to calculate the number of cores in a Databricks cluster....

Last updated: March 31st, 2023 by Adam Pavlacka

Install a private PyPI repo

How to install libraries from private PyPI repositories....

Last updated: December 26th, 2023 by darshan.bargal

IP access list update returns INVALID_STATE

Cannot update IP access list. INVALID_STATE error message....

Last updated: March 4th, 2022 by Gobinath.Viswanathan

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Last updated: March 4th, 2022 by jordan.hicks

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Last updated: March 4th, 2022 by Gobinath.Viswanathan

Cluster failed to launch

Learn how to resolve cluster launch failures....

Last updated: March 4th, 2022 by Adam Pavlacka

Custom Docker image requires root

Custom Docker containers must be configured to start as the root user when used with Databricks....

Last updated: March 4th, 2022 by dayanand.devarapalli

Job fails due to cluster manager core instance request limit

Learn how to troubleshoot Databricks errors related to API rate limits....

Last updated: March 4th, 2022 by Adam Pavlacka

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Last updated: March 4th, 2022 by Adam Pavlacka

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Last updated: March 4th, 2022 by arvind.ravish

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Last updated: March 4th, 2022 by kavya.parag

Persist Apache Spark CSV metrics to a DBFS location

Persist Spark CSV metrics to a sink in a DBFS location....

Last updated: March 4th, 2022 by Adam Pavlacka

Replay Apache Spark events in a cluster

Use a single node cluster to replay another cluster's event log in the Spark UI....

Last updated: February 10th, 2023 by arjun.kaimaparambilrajan

Set Apache Hadoop core-site.xml properties

Set Apache Hadoop core-site.xml properties in a Databricks cluster....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Set executor log level

Learn how to set the log levels on Databricks executors....

Last updated: March 4th, 2022 by Adam Pavlacka

Apache Spark job doesn’t start

Learn how to troubleshoot a Databricks Spark job that won't start....

Last updated: March 4th, 2022 by Adam Pavlacka

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

Last updated: August 23rd, 2022 by navya.athiraram

Unexpected cluster termination

Learn how to troubleshoot a Databricks cluster that stopped unexpectedly....

Last updated: March 4th, 2022 by Adam Pavlacka

How to configure single-core executors to run JNI libraries

Learn how to configure single-core executors to run JNI libraries on Databricks....

Last updated: March 4th, 2022 by Adam Pavlacka

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Last updated: February 29th, 2024 by Adam Pavlacka

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Last updated: August 9th, 2024 by Adam Pavlacka

Apache Spark UI shows less than total node memory

Learn what to do when the Spark UI shows less memory than is actually available on the node....

Last updated: July 22nd, 2022 by Adam Pavlacka

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Last updated: December 8th, 2022 by xin.wang

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Last updated: December 8th, 2022 by xin.wang

Enable retries in init script

Add a retry function to your init script....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Last updated: September 13th, 2022 by prakash.jha

Run a custom Databricks Runtime on your cluster

Configure your cluster to run a custom Databricks Runtime image via the UI or API....

Last updated: September 11th, 2024 by rakesh.parija

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Last updated: October 31st, 2022 by harrison.schueler

Slow cluster launch and missing nodes

Learn how to resolve a "nodes could not be acquired" error when starting a Databricks ....

Last updated: December 8th, 2022 by Adam Pavlacka

IP address limit prevents cluster creation

Learn how to fix a public IP address quota limit Cloud Provider Launch error when starting a Databricks cluster....

Last updated: May 30th, 2023 by laila.haddad

CPU core limit prevents cluster creation

Learn how to fix a CPU core quota limit Cloud Provider Launch error when starting a Databricks cluster....

Last updated: December 8th, 2022 by Adam Pavlacka

Custom garbage collection prevents cluster launch

Using a custom garbage collection algorithm on Databricks Runtime 10.0 and above prevents the cluster from starting....

Last updated: December 8th, 2022 by harikrishnan.kunhumveettil

SSH to the cluster driver node

How to SSH to the Apache Spark cluster driver node in an Azure virtual network...

Last updated: March 15th, 2023 by xin.wang

Adding a configuration setting overwrites all default spark.executor.extraJavaOptions settings

Learn how to resolve overwritten configuration settings in Databricks....

Last updated: December 8th, 2022 by Adam Pavlacka

UnknownHostException on cluster launch

Troubleshoot an UnknownHostException on cluster launch. This is often a DNS configuration issue....

Last updated: December 8th, 2022 by arnab.saha

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Last updated: December 21st, 2022 by simran.arora

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

Last updated: December 21st, 2022 by simran.arora

R commands fail on custom Docker cluster

R version 4.2.0 changed the way Renviron.site is initialized, so you must set an environment variable when using custom Docker clusters....

Last updated: January 20th, 2023 by Atanu.Sarkar

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Last updated: March 17th, 2023 by vivian.wilfred

Legacy global init script migration notebook

Easily migrate your legacy global init scripts to the current global init script framework....

Last updated: August 28th, 2023 by Adam Pavlacka

Python kernel is unresponsive error message

Learn how to identify and troubleshoot the cause of an unresponsive Python kernel error....

Last updated: July 17th, 2023 by laila.haddad

Spark image download failure error message

Learn how to troubleshoot the Spark image download failure error message....

Last updated: April 17th, 2023 by laila.haddad

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Last updated: May 2nd, 2023 by Adam Pavlacka

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Last updated: February 27th, 2024 by Adam Pavlacka

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Last updated: July 17th, 2023 by simran.arora

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Last updated: November 30th, 2023 by Adam Pavlacka

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

Last updated: November 30th, 2023 by Adam Pavlacka

Shorten cluster provisioning time by using Docker containers

Learn how to speed up cluster provisioning by using Docker container services...

Last updated: November 30th, 2023 by Adam Pavlacka

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Last updated: March 26th, 2024 by Adam Pavlacka

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Last updated: December 4th, 2023 by Adam Pavlacka

Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files....

Last updated: February 5th, 2024 by Adam Pavlacka

Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Embed system.exit code in your application to shutdown the Java virtual machine with exit code 0....

Last updated: September 12th, 2024 by shubham.chhabra

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Modify your log4j2 configuration file directly within the Databricks environment. ...

Last updated: September 12th, 2024 by raahat.varma

Ganglia metrics not appearing in historical metrics snapshots list

Configure the collection period, or update Databricks Runtime to at least 13.0 to use Compute metrics instead. ...

Last updated: September 27th, 2024 by david.vega

Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes. ...

Last updated: September 27th, 2024 by walter.camacho

Init script stored on a volume fails to execute on cluster start

Init scripts created on Windows systems and uploaded to Unity Catalog volumes have CRLF as a newline which needs to be converted to LF before the cluster can process it. ...

Last updated: October 24th, 2024 by kunal.jadhav

Databricks API last_activity_time attribute shows incorrect timestamp

Use the cluster auto-termination feature to manage cluster termination based on inactivity....

Last updated: November 4th, 2024 by walter.camacho

BROADCAST_VARIABLE_NOT_LOADED or JVM_ATTRIBUTE_NOT_SUPPORTED errors when using broadcast variables in a shared access mode cluster

Use a single-user cluster or pass a variable into a function as a state instead. ...

Last updated: November 6th, 2024 by kaushal.vachhani

404 error when installing krb5-user module

Manually remove the var directory path to refresh the cached data. ...

Last updated: November 25th, 2024 by david.vega

Cluster startup failure while running proxy-configured init script with other init scripts

Modify the proxy init script to bypass local addresses....

Last updated: December 10th, 2024 by guruprasad.bn

Init scripts failing with unexpected end of file error

Remove the special characters from the init script....

Last updated: December 20th, 2024 by jeremy.ramirez

Unable to access the hive_metastore schema

Ensure all clusters use the same Hive metastore version and Apache Spark configurations are set. ...

Last updated: December 20th, 2024 by girish.sharma

Job fails while installing ODBC Driver 18 for SQL Server using an init script

Add msodbcsql18 to the LD_LIBRARY_PATH then append LD_LIBRARY_PATH path to /etc/environment....

Last updated: December 20th, 2024 by julian.campabadal

Error when trying to use Apache Spark’s Pyspark offset method on DataFrames with serverless compute

Use the limit method or the monotonically_increasing_id() function instead....

Last updated: December 23rd, 2024 by Tarun Sanjeev

Jobs failing with schema conversion error: cannot convert Parquet type INT32 to Photon type long

Set spark.databricks.photon.scan.enabled to false....

Last updated: January 16th, 2025 by Guilherme Leite

Cannot access Apache SparkContext object using addPyFile

Leverage the addArtifact API instead....

Last updated: January 17th, 2025 by Raghavan Vaidhyaraman

Cluster fails to launch with a Bootstrap Timeout error

Verify allowlisting of necessary services and correct configuration of your VPC/VNet. ...

Last updated: January 20th, 2025 by parth.sundarka

Cluster fails to launch with error, “user specified an invalid argument”

Change the cluster owner to an active user....

Last updated: January 28th, 2025 by parth.sundarka

Job executions failing on clusters using Docker Container Services with MalformedInputException error

Specify the correct character encoding when reading the file and change the LANG settings. ...

Last updated: January 28th, 2025 by G Yashwanth Kiran

Enabling Dynamic Allocation leads to NODES_LOST scenario

Enable Autoscaling when you create a Databricks cluster. ...

Last updated: January 29th, 2025 by MuthuLakshmi.AN

Clusters using Docker databricksruntime:latest tag are not starting

Replace latest tag with LTS for Databricks Docker image....

Last updated: January 30th, 2025 by zhengxian.huang

Jobs failing with BindException error after upgrading to Databricks Runtime 11.3 LTS or above

Change the ipywidgets default port to another available port....

Last updated: January 30th, 2025 by zhengxian.huang

Cluster fails to initialize after a Databricks Runtime upgrade

Check your init scripts and then your Apache Spark configurations....

Last updated: January 30th, 2025 by walter.camacho

Missing the audit log event of a cluster deletion

Pin your cluster and monitor usage....

Last updated: February 19th, 2025 by allia.khosla

Increased job execution time after migrating from all-purpose to job cluster

Increase the Hive client pool size in the job cluster configuration to match the previous all-purpose compute setting....

Last updated: February 27th, 2025 by manikandan.ganesan

NODES_LOST error during cluster upsizing when Apache Spark dynamic allocation is enabled

Remove the spark.dynamicAllocation.enabled Spark config from the compute configuration....

Last updated: February 28th, 2025 by Gihyeon Lee

Listing Hive metastore tables in Catalog Explorer failing with error getting schemas

Remove the Apache Spark configuration "spark.databricks.session.share true". ...

Last updated: March 7th, 2025 by shashank.chaudhary

Change the minor version of Python in a cluster

Use an init script to install the desired versions of Python and pyenv....

Last updated: March 10th, 2025 by Adam Pavlacka

Row value assignments not reflecting expected output in code that loops through temporary views

Avoid using temporary views with the same name when using loops in Spark Connect. ...

Last updated: March 19th, 2025 by raul.goncalves

Unresolved column error when using Apache Spark Connect to run a query to create a temporary view

Use unique names for each temporary view. ...

Last updated: March 19th, 2025 by raul.goncalves

scala.collection.immutable.HashMap$HashMap1 class leading to OOM error in driver

Change the webUI events location to RockDB....

Last updated: March 19th, 2025 by fernando.soster

Unable to attach init scripts to an interactive cluster

Ensure the cluster creator exists in the workspace and has the necessary group memberships....

Last updated: March 19th, 2025 by jose.rojas

How to restrict selection of specific Databricks runtimes in the compute creation UI

Implement a regular expression (regex) validation to ensure only certain selections are available....

Last updated: March 20th, 2025 by guruprasad.bn

Cannot select a compute policy for a DLT Pipeline

Ensure that you have permission to use the policy and are using the correct compute type....

Last updated: March 22nd, 2025 by jose.salgado

How to restrict cluster creation to single-node only

Add a JSON configuration to your compute policy....

Last updated: March 25th, 2025 by guruprasad.bn

Workflows are failing with a 'Could not reach driver of the cluster' error

Use a larger driver instance or increase the REPL timeout....

Last updated: March 27th, 2025 by kingshuk.das

Cannot create cluster: spark conf: 'spark.databricks.cluster.profile' is not allowed when choosing an access mode

Use the flag is_single_node to create single node compute....

Last updated: April 9th, 2025 by parth.sundarka

Launch the web terminal on a DCS enabled cluster

Use ttyd to launch the web terminal with a custom Docker image on a cluster....

Last updated: April 14th, 2025 by ismael.khalique

FileNotFoundError when trying to use Android Development Bridge (ADB) command line tool on a cluster

Explicitly install the ADB tool on your cluster and set in the correct system path....

Last updated: April 17th, 2025 by parth.sundarka

Spark UI is empty for the job clusters after termination

For non-Spark tasks the Spark UI should be empty....

Last updated: April 17th, 2025 by kunal.jadhav

502 error when trying to access the Spark UI

Enable Spark UI to store its data on disk instead of in memory....

Last updated: April 24th, 2025 by Guilherme Leite

Pulumi fails to deploy workflows in serverless mode

Update the Pulumi provider, check for configuration errors, and remove the job cluster definition from the payload....

Last updated: April 26th, 2025 by kingshuk.das

Unable to run interactive workloads using a dedicated (formerly single user) compute assigned to a service principal where you have the Service Principal User role

Use a dedicated compute linked to a user or group instead, or switch to standard compute to run interactive workloads....

Last updated: May 5th, 2025 by david.vega

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.


© Databricks 2022-2025. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Notice (Updated) | Terms of Use | Your Privacy Choices | Your California Privacy Rights Privacy Rights icon


Knowledge Base Software powered by Helpjuice

Definition by Author

0
0