Problem
When you attempt to install the GDAL library on a cluster using an init script, the cluster doesn’t start and on cluster events you see an error message.
{
"reason": {
"code": "INIT_SCRIPT_FAILURE",
"type": "CLIENT_ERROR",
"parameters": {
"instance_id": "<INSTANCE-ID>",
**"databricks_error_message": "Cluster scoped init script <gdal-init.sh> failed: Script exit status is non-zero"**
}
}
}You may also get an error about failing to reach a Ubuntu website.
Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease Cannot initiate the connection to security.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable).Cause
The first error message can be caused by a misconfigured init script.
The error message about not reaching the Ubuntu website can happen if the package repository only uses http by default. apt-get can use HTTPS but the repository must support HTTPS/SSL for this to work. The main package repository at archive.ubuntu.com does not. If apt-get tries to connect via HTTPS and it is not supported, the connection fails with an error.
Solution
These steps resolve most GDAL installation issues.
1. Verify that you are using Databricks Runtime version 13.3 LTS or above.
2. Make sure you are following the official Mosaic GDAL installation guide. This guide provides an up-to-date method for installing GDAL on Databricks clusters.
3. Create the init script based on the instructions in the Mosaic guide. The guide includes a Python function setup_gdal() that generates an init script tailored to your environment.
To generate the init script run the following code:
# First install mosaic
%pip install databricks-mosaic
# Then in another cell import the library and run the setup_gdal() function
%python
import mosaic as mos
mos.enable_mosaic(spark, dbutils)
mos.setup_gdal()This is an example init script generated with setup_gdal().
#!/bin/bash
# --
# This is for Ubuntu 22.04 (Jammy)
# [1] corresponds to DBR 13+
# [2] jammy offers GDAL 3.4.1
# [3] see Mosaic functions (python) to configure
# and pre-stage resources:
# - setup_fuse_install(...) and
# - setup_gdal(...)
# [4] this script has conditional logic based on variables
# [5] stripped back in Mosaic 0.4.2+
# Author: Michael Johns | mjohns@databricks.com
# Last Modified: 29 APR, 2024
# TEMPLATE-BASED REPLACEMENT
# - can also be manually specified
FUSE_DIR='/Workspace/Shared/geospatial/mosaic/gdal/jammy/0.4.2'
# CONDITIONAL LOGIC
WITH_FUSE_SO=0 # <- use fuse dir shared objects (vs wget)
# refresh package info
# 0.4.2 - added "-y"
sudo apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-backports main universe multiverse restricted"
sudo apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-updates main universe multiverse restricted"
sudo apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-security main multiverse restricted universe"
sudo apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) main multiverse restricted universe"
sudo apt-get update -y
# install natives
# 0.4.2 added package lock wait (can change value)
sudo apt-get -o DPkg::Lock::Timeout=-1 install -y unixodbc libcurl3-gnutls libsnappy-dev libopenjp2-7
sudo apt-get -o DPkg::Lock::Timeout=-1 install -y gdal-bin libgdal-dev python3-numpy python3-gdal
# pip install gdal
# matches jammy version
pip install --upgrade pip
pip install gdal==3.4.1
# add pre-build JNI shared object to the path
if [ $WITH_FUSE_SO == 1 ]
then
# copy from fuse dir with no-clobber
sudo cp -n $FUSE_DIR/libgdalalljni.so /usr/lib
sudo cp -n $FUSE_DIR/libgdalalljni.so.30 /usr/lib
sudo cp -n $FUSE_DIR/libgdalalljni.so.30.0.3 /usr/lib
else
# copy from github
GITHUB_REPO_PATH=databrickslabs/mosaic/main/resources/gdal/jammy
sudo wget -nv -P /usr/lib -nc https://raw.githubusercontent.com/$GITHUB_REPO_PATH/libgdalalljni.so
sudo wget -nv -P /usr/lib -nc https://raw.githubusercontent.com/$GITHUB_REPO_PATH/libgdalalljni.so.30
sudo wget -nv -P /usr/lib -nc https://raw.githubusercontent.com/$GITHUB_REPO_PATH/libgdalalljni.so.30.0.3
fi4. Install the GDAL init script as a cluster-scoped init script (AWS | Azure | GCP).
5. If you are seeing any library incompatibilities, review any global init scripts (AWS | Azure | GCP) that are enabled in your workspace and ensure they are not causing a conflict.
Info
You must be a workspace admin to create or see global init scripts. Be careful when editing any global init script as it impacts all workloads running in the workspace.
6. If the GDAL init script is failing due to archive.ubuntu.com/ubuntu being unreachable, you need to use a HTTPS enabled mirror. You will need to update the GDAL init script, replacing archive.ubuntu.com/ubuntu with the address of the mirror you want to use.
Partial mirror list
Asia
https://ftp.sjtu.edu.cn/ubuntu/
https://mirrors.tuna.tsinghua.edu.cn/ubuntu/
Europe
https://mirror.vorboss.net/ubuntu-archive/
https://mirror.plusserver.com/ubuntu/ubuntu/
United States
https://mirrors.bloomu.edu/ubuntu/
https://mirrors.ocf.berkeley.edu/ubuntu/