Init script fails to download Maven JAR

Cluster init script fails to download a Maven JAR when trying to install a library.

Written by arvind.ravish

Last published at: May 11th, 2022

Problem

You have an init script that is attempting to install a library via Maven, but it fails when trying to download a JAR.

https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.1/rapids-4-spark_2.12-0.4.1.jar%0D
Resolving repo1.maven.org (repo1.maven.org)... 151.101.248.209
Connecting to repo1.maven.org (repo1.maven.org)|151.101.248.209|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-07-30 01:31:11 ERROR 404: Not Found.

Cause

There is a carriage return (%0D) character at the end of one or more of the lines in the init script.

This is usually caused by editing a file in Windows and then uploading it to your Databricks workspace without removing the excess carriage returns.

Solution

Remove the Windows carriage returns by running dos2unix on the file after you have uploaded it to the workspace.

%sh

sudo apt-get install dos2unix -y
dos2unix file <initscript.sh>

Once you have removed the Windows carriage returns from the file, you can configure the init script as normal.