Problem

When running general queries against anonymous or temporary functions, you may get an [INSUFFICIENT_PERMISSIONS] Insufficient privileges error message.

An error occurred while calling o790.withColumn.
: org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on anonymous function. SQLSTATE: 42501

Cause

ANONYMOUS FUNCTION controls access to anonymous or temporary functions. If you do not have this privilege when querying those functions, the query fails with the underlying error. 

The error can also occur if you’re using Databricks SQL.  ANONYMOUS FUNCTION objects are not supported in Databricks SQL.

Solution

As a workspace admin, run the following query to provide the affected entity with the required permission. 

%sql

GRANT SELECT ON ANONYMOUS FUNCTION TO `<user@domain-name/group/service-principal>`

In Databricks SQL environments, check if the Apache Spark setting spark.databricks.acl.sqlOnly true is present in your cluster setup. If it is, delete it.

This allows preference for Python or Scala, which work with ANONYMOUS FUNCTION. 

For more information on using ANONYMOUS FUNCTION, review the CREATE FUNCTION (External) documentation.