Accessing secrets with DBUtils fails on Databricks clusters

Problem

You are using the DBUtils module to access secrets on a Databricks cluster. The access attempt fails, and returns a java.lang.SecurityException error.

Py4JJavaError: An error occurred while calling o295.get.
: java.lang.SecurityException: Accessing a secret from Databricks Connect requires
a privileged secrets token. To obtain such a token, you can run the following
in a Databricks workspace notebook:

displayHTML(
"<b>Privileged DBUtils token (expires in 48 hours): </b>" +
dbutils.notebook.getContext.apiToken.get.split("").mkString("<span/>"))

Then, run dbutils.secrets.setToken(<value>) locally to save the token.
Note that these tokens expire after *48 hours*.

Cause

DBUtils has been updated. The legacy method to access secrets is not supported.

Legacy method:

def get_dbutils(spark):
 try:
  from pyspark.dbutils import DBUtils
  dbutils = DBUtils(spark)
except ImportError:
  import IPython
  dbutils = IPython.get_ipython().user_ns["dbutils"]
return dbutils

Solution

Review the Databricks documentation on Accessing DBUtils and use the documented methods to access secrets with the DBUtils module.