Update notification settings for jobs with the Jobs API

Email notifications can be useful when managing multiple jobs. If you have many jobs configured without notifications, manually adding notifications can be time consuming. Instead, you can use the Jobs API (AWS | Azure | GCP) to add email notifications to the jobs in your workspace. Instructions In order to call the Jobs API, you first need to setup...

2 min reading time

Use iptables to access the EC2 metadata server

Problem You are trying to access the metadata server for your EC2 instance but cannot connect. Cause This is the intended, default behavior. It is functioning as designed. Solution Use an init script to apply a custom iptables configuration to your Databricks cluster which enables access to the metadata server. Define a location to store the init sc...

0 min reading time

Copy installed libraries from one cluster to another

If you have a highly customized Databricks cluster, you may want to duplicate it and use it for other projects. When you clone a cluster, only the Apache Spark configuration and other cluster configuration information is copied. Installed libraries are not copies by default. To copy the installed libraries, you can run a Python script after cloning ...

1 min reading time