Upload large files using DBFS API 2.0 and PowerShell

Using the Databricks REST API to interact with your clusters programmatically can be a great way to streamline workflows with scripts. The API can be called with various tools, including PowerShell. In this article, we are going to take a look at an example DBFS put command using curl and then show you how to execute that same command using PowerShe...

2 min reading time

Users unable to view job results when using remote Git source

Problem You are running a job using notebooks that are stored in a remote Git repository (AWS | Azure | GCP). Databricks users with Can View permissions (who are not a workspace admin or owners of the job) cannot access or view the results of ephemeral jobs submitted via dbutils.notebook.run() from parent notebook. Cause When job visibility control ...

2 min reading time

Set instance_profile_arn as optional with a cluster policy

In this article, we review the steps to create a cluster policy for the AWS attribute instance_profile_arn and define it as optional. This allows you to start a cluster with a specific AWS instance profile. You can also start a cluster without an instance profile. Note You must be an admin user in order to manage cluster policies. Create a new clust...

0 min reading time