Updated May 9th, 2022 by arvind.ravish

Get Apache Spark config in DBConnect

You can always view the Spark configuration (AWS | Azure | GCP) for your cluster by reviewing the cluster details in the workspace. If you are using DBConnect (AWS | Azure | GCP) you may want to quickly review the current Spark configuration details without switching over to the workspace UI. This example code shows you how to get the current Spark ...

0 min reading time
Updated May 24th, 2022 by arvind.ravish

Drop database without deletion

By default, the DROP DATABASE (AWS | Azure | GCP) command drops the database and deletes the directory associated with the database from the file system. Sometimes you may want to drop the database, but keep the underlying database directory intact. Example code You can use this example code to drop the database without dropping the underlying stora...

0 min reading time
Updated March 4th, 2022 by arvind.ravish

Cluster fails to start with dummy does not exist error

Problem You try to start a cluster, but it fails to start. You get an Apache Spark error message. Internal error message: Spark error: Driver down You review the cluster driver and worker logs and see an error message containing java.io.FileNotFoundException: File file:/databricks/driver/dummy does not exist. 21/07/14 21:44:06 ERROR DriverDaemon$: X...

1 min reading time
Updated May 11th, 2022 by arvind.ravish

Init script fails to download Maven JAR

Problem You have an init script that is attempting to install a library via Maven, but it fails when trying to download a JAR. https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.1/rapids-4-spark_2.12-0.4.1.jar%0D Resolving repo1.maven.org (repo1.maven.org)... Connecting to repo1.maven.org (repo1.maven.org)|151.101.248....

0 min reading time
Updated February 25th, 2023 by arvind.ravish

Troubleshoot key vault access issues

You are trying to access secrets, when you get an error message. com.databricks.common.client.DatabricksServiceException: INVALID_STATE: Databricks could not access keyvault: https://xxxxxxx.vault.azure.net/. There is not a single root cause for this error message, so you will have to do some troubleshooting. Confirm permissions are correctly set on...

0 min reading time
Updated May 20th, 2022 by arvind.ravish

RStudio server backend connection error

Problem You get a backend connection error when using RStudio server. Error in Sys.setenv(EXISTING_SPARKR_BACKEND_PORT = system(paste0("wget -qO - 'http://localhost:6061/?type=\"com.databricks.backend.common.rpc.DriverMessages$StartRStudioSparkRBackend\"' --post-data='{\"@class\":\"com.databricks.backend.common.rpc.DriverMessages$StartRStudioSparkRB...

1 min reading time
Updated December 7th, 2022 by arvind.ravish

Configure Simba JDBC driver using Azure AD

This article describes how to access Azure Databricks with a Simba JDBC driver using Azure AD authentication. This can be useful if you want to use an Azure AD user account to connect to Azure Databricks. Info Power BI has native support for Azure AD authentication with Azure Databricks. Review the Power BI documentation for more information. Create...

1 min reading time
Load More