Email notifications can be useful when managing multiple jobs. If you have many jobs configured without notifications, manually adding notifications can be time consuming. Instead, you can use the Jobs API (AWS | Azure | GCP) to add email notifications to the jobs in your workspace.

Instructions

In order to call the Jobs API, you first need to setup a personal access token and a secret scope. This allows you to interact with the API via a script.

After the secret scopes have been setup, you can run the example script in a notebook to update all the jobs in your workspace at once.

Create a Databricks personal access token

Follow the Personal access tokens for users (AWS | Azure | GCP) documentation to create a personal access token.

Create a secret scope

Follow the Create a Databricks-backed secret scope (AWS | Azure | GCP) documentation to create a secret scope.

Store your personal access token and your Databricks instance in the secret scope

Follow the Create a secret in a Databricks-backed scope (AWS | Azure | GCP) documentation to store the personal access token you created and your Databricks instance as new secrets within your secret scope.

Your Databricks instance is the hostname for your workspace, for example, xxxxx.cloud.databricks.com.

Use a Python script to update job notifications for all the jobs in the workspace 

You need to replace the following values in the script before running:

• <email-address> - Email address to notify.
  
• <scope-name> - The name of your scope that holds the secrets.
• <secret-name-1> - The name of the secret that holds your Databricks instance.
• <secret-name-2> - The name of the secret that holds your personal access token.

import json
import requests

API_URL = dbutils.secrets.get(scope = "<scope-name>", key = "<secret-name1>") 
TOKEN = dbutils.secrets.get(scope = "<scope-name>", key = "<secret-name2>")  
url = f"{API_URL}/api/2.0/jobs/list"  #Get all the jobs created inside the workspace
payload={}
headers = {
  'Authorization': 'Bearer ' + TOKEN
}
response = requests.request("GET", url, headers=headers, data=payload)
response.json()
for job in response.json()['jobs']:
    job_id = job["job_id"]
    payload = {
"job_id": job_id,
"new_settings": {
"email_notifications": {
"on_start": [
"<email-address>"  #user's email ID
],
"on_success": [
"<email-address>"   
],
"on_failure": [
"<email-address>"
]
}
}
    }
    url2 = f"{API_URL}/api/2.1/jobs/update"
    r = requests.post(url2, data = json.dumps(payload), headers=headers)
print("successfully added the email_notification to jobs")

This version of the sample code adds an if condition, checking an email address against the value creator_user_name. This filters the jobs based on the job creator.

Replace <job-creator-email-address> with the email address you want to filter on.

import json
import requests


API_URL = dbutils.secrets.get(scope = "<scope-name>", key = "<secret-name1>") 
TOKEN = dbutils.secrets.get(scope = "<scope-name>", key = "<secret-name2>")  
url = f"{API_URL}/api/2.0/jobs/list"  #Get all the jobs created inside the workspace
payload={}
headers = {
  'Authorization': 'Bearer ' + TOKEN
}
response = requests.request("GET", url, headers=headers, data=payload)
response.json()
for job in response.json()['jobs']:
    if job['creator_user_name']=="<job-creator-email-address>":  # filtering the jobs based on the job creator
        job_id = job["job_id"]
        payload = {
        "job_id": job_id,
        "new_settings": {
        "email_notifications": {
        "on_start": [
        "<email-address>"  #user's email ID
                      ],
        "on_success": [
        "<email-address>"   
                    ],
        "on_failure": [
        "<email-address>"
                       ]
                    }
                        }
                    }
        url2 = f"{API_URL}/api/2.1/jobs/update"
        r = requests.post(url2, data = json.dumps(payload), headers=headers)
print("successfully added the email_notification to jobs")

Verify status in the Job UI

Once the code runs successfully you can verify the updated notification status by checking your jobs in the Job UI.

1. In the left nav menu, click on Workflows.
   
2. Click the name of a job you want to verify.
3. On the right side of the job details page, scroll down to the Notifications section.
4. The email address you added to the sample script is now present and configured for notifications.