Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message.

Written by arvind.ravish

Last published at: March 4th, 2022


You try to start a cluster, but it fails to start. You get an Apache Spark error message.

Internal error message: Spark error: Driver down

You review the cluster driver and worker logs and see an error message containing File file:/databricks/driver/dummy does not exist.

21/07/14 21:44:06 ERROR DriverDaemon$: XXX Fatal uncaught exception. Terminating driver. File file:/databricks/driver/dummy does not exist
   at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(
   at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(
   at org.apache.spark.SparkContext.addFile(SparkContext.scala:1668)
   at org.apache.spark.SparkContext.addFile(SparkContext.scala:1632)
   at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:511)
   at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:511)
   at scala.collection.immutable.List.foreach(List.scala:392)


You have spark.files dummy set in your Spark Config, but no such file exists.

Spark interprets the dummy configuration value as a valid file path and tries to find it on the local file system. If the file does not exist, it generates the error message. File file:/databricks/driver/dummy does not exist


Option 1: Delete spark.files dummy from your Spark Config if you are not passing actual files to Spark.

Option 2: Create a dummy file and place it on the cluster. You can do this with an init script.

  1. Create the init script.
    touch /databricks/driver/dummy""", True)
  2. Install the init script that you just created as a cluster-scoped init script.
    You will need the full path to the location of the script (dbfs:/databricks/<init-script-folder>/
  3. Restart the cluster

Restart your cluster after you have installed the init script.

Was this article helpful?