• Home
  • All articles
  • Libraries
  • You have external location access, but cannot install a JAR on a shared cluster from an S3 path outside of volumes

You have external location access, but cannot install a JAR on a shared cluster from an S3 path outside of volumes

Use an Instance profile with proper access OR store the JAR in a Unity Catalog volume.

Written by kingshuk.das

Last published at: March 27th, 2025

Problem

You are trying to install a JAR from an S3 location that is not part of one of your volumes. You have access to external locations, but you still get an access denied error. This happens when using both Unity Catalog-enabled clusters and non-Unity Catalog-enabled clusters.
 

Example error message

S3 Path: s3://<my-bucket>/<my-library>/<my-jar-file>.jar - Reason: Access Denied Description: The cluster is unable to access the specified S3 path because no instance profile (IAM role) is attached to the cluster to allow S3 access.

 

Cause

The cluster does not have an instance profile. Instance profiles are required to authenticate the cluster's S3 permissions when accessing files directly from any S3 location other than Unity Catalog volumes. When using a Unity Catalog cluster and trying to install a JAR file from a volume's S3 locations, an instance profile is not required.

Volumes are tightly coupled with Unity Catalog infrastructure, and when a file is kept in a volume, the authentication (IAM role and privileges checking) is done via external locations. For S3 locations that are not part of a volume, the instance profile needs to authenticate the cluster's S3 permissions.

 

Solution

There are multiple ways to resolve this issue.
 

Attach an instance profile to the cluster

If you will be accessing multiple JARs (and potentially other files), you should attach an instance profile to your cluster with the permissions needed to access the S3 bucket. The instance profile must have the s3:GetObject permission for the S3 bucket where your file is stored.
 

Install from workspace files

If this is a limited use case, you can work around the issue by uploading the JAR file to your workspace as a workspace file and then installing from there.
 

Use Unity Catalog volumes with shared clusters

There is a known issue with shared clusters and libraries. The IAM role passthrough is not compatible with a library installation, which requires a cluster instance profile's permission to download.  In these cases, you must store your JAR files on Unity Catalog volumes and install them from there. 

You must allow the volume under the Unity Catalog metastore section Allow JARs & Init scripts. For more information, review the Allowlist libraries and init scripts on compute with standard access mode (formerly shared access mode) documentation.
 

Single-user clusters

You must use instance profiles that have access to the Databricks control plane for JAR installation.