Databricks Knowledge Base

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Clusters (Azure)

These articles can help you manage your Apache Spark clusters.

42 Articles in this category

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.

  • Home
  • All articles
  • Clusters (Azure)

Enable OpenJSSE and TLS 1.3

Add OpenJSSE to allow the use of TLS 1.3 for encrypted data transmission....

Last updated: March 2nd, 2022 by Adam Pavlacka

How to calculate the number of cores in a cluster

Learn how to calculate the number of cores in a Databricks cluster....

Last updated: March 2nd, 2022 by Adam Pavlacka

Install a private PyPI repo

How to install libraries from private PyPI repositories....

Last updated: March 4th, 2022 by darshan.bargal

IP access list update returns INVALID_STATE

Cannot update IP access list. INVALID_STATE error message....

Last updated: March 4th, 2022 by Gobinath.Viswanathan

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Last updated: March 4th, 2022 by jordan.hicks

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Last updated: March 4th, 2022 by Gobinath.Viswanathan

Cluster failed to launch

Learn how to resolve cluster launch failures....

Last updated: March 4th, 2022 by Adam Pavlacka

Custom Docker image requires root

Custom Docker containers must be configured to start as the root user when used with Databricks....

Last updated: March 4th, 2022 by dayanand.devarapalli

Job fails due to cluster manager core instance request limit

Learn how to troubleshoot Databricks errors related to API rate limits....

Last updated: March 4th, 2022 by Adam Pavlacka

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Last updated: March 4th, 2022 by Adam Pavlacka

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Last updated: March 4th, 2022 by arvind.ravish

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Last updated: March 4th, 2022 by kavya.parag

Persist Apache Spark CSV metrics to a DBFS location

Persist Spark CSV metrics to a sink in a DBFS location....

Last updated: March 4th, 2022 by Adam Pavlacka

Replay Apache Spark events in a cluster

Use a single node cluster to replay another cluster's event log in the Spark UI....

Last updated: February 10th, 2023 by arjun.kaimaparambilrajan

Set Apache Hadoop core-site.xml properties

Set Apache Hadoop core-site.xml properties in a Databricks cluster....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Set executor log level

Learn how to set the log levels on Databricks executors....

Last updated: March 4th, 2022 by Adam Pavlacka

Apache Spark job doesn’t start

Learn how to troubleshoot a Databricks Spark job that won't start....

Last updated: March 4th, 2022 by Adam Pavlacka

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

Last updated: August 23rd, 2022 by navya.athiraram

Unexpected cluster termination

Learn how to troubleshoot a Databricks cluster that stopped unexpectedly....

Last updated: March 4th, 2022 by Adam Pavlacka

How to configure single-core executors to run JNI libraries

Learn how to configure single-core executors to run JNI libraries on Databricks....

Last updated: March 4th, 2022 by Adam Pavlacka

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Last updated: December 8th, 2022 by Adam Pavlacka

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Last updated: March 4th, 2022 by Adam Pavlacka

Apache Spark UI shows less than total node memory

Learn what to do when the Spark UI shows less memory than is actually available on the node....

Last updated: July 22nd, 2022 by Adam Pavlacka

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Last updated: December 8th, 2022 by xin.wang

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Last updated: December 8th, 2022 by xin.wang

Enable retries in init script

Add a retry function to your init script....

Last updated: March 4th, 2022 by arjun.kaimaparambilrajan

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Last updated: September 13th, 2022 by prakash.jha

Run a custom Databricks runtime on your cluster

Configure your cluster to run a custom Databricks runtime image via the UI or API....

Last updated: October 26th, 2022 by rakesh.parija

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Last updated: October 31st, 2022 by harrison.schueler

Slow cluster launch and missing nodes

Learn how to resolve a "nodes could not be acquired" error when starting a Databricks ....

Last updated: December 8th, 2022 by Adam Pavlacka

IP address limit prevents cluster creation

Learn how to fix a public IP address quota limit Cloud Provider Launch error when starting a Databricks cluster....

Last updated: December 8th, 2022 by Adam Pavlacka

CPU core limit prevents cluster creation

Learn how to fix a CPU core quota limit Cloud Provider Launch error when starting a Databricks cluster....

Last updated: December 8th, 2022 by Adam Pavlacka

Custom garbage collection prevents cluster launch

Using a custom garbage collection algorithm on Databricks Runtime 10.0 and above prevents the cluster from starting....

Last updated: December 8th, 2022 by harikrishnan.kunhumveettil

SSH to the cluster driver node

How to SSH to the Apache Spark cluster driver node in an Azure virtual network...

Last updated: March 15th, 2023 by xin.wang

Adding a configuration setting overwrites all default spark.executor.extraJavaOptions settings

Learn how to resolve overwritten configuration settings in Databricks....

Last updated: December 8th, 2022 by Adam Pavlacka

UnknownHostException on cluster launch

Troubleshoot an UnknownHostException on cluster launch. This is often a DNS configuration issue....

Last updated: December 8th, 2022 by arnab.saha

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Last updated: December 21st, 2022 by simran.arora

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

Last updated: December 21st, 2022 by simran.arora

R commands fail on custom Docker cluster

R version 4.2.0 changed the way Renviron.site is initialized, so you must set an environment variable when using custom Docker clusters....

Last updated: January 20th, 2023 by Atanu.Sarkar

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Last updated: March 17th, 2023 by vivian.wilfred

Legacy global init script migration notebook

Easily migrate your legacy global init scripts to the current global init script framework....

Last updated: March 30th, 2023 by Adam Pavlacka


© Databricks 2022-2023. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Notice (Updated) | Terms of Use | Your Privacy Choices | Your California Privacy Rights Privacy Rights icon

Definition by Author

0
0