Problem: Invalid Access Token Error When Running Jobs with Airflow

Problem

When you run scheduled Airflow Databricks jobs, you get this error:

Invalid Access Token : 403 Forbidden Error

Cause

To run or schedule Databricks jobs through Airflow, you need to configure the Databricks connection using the Airflow web UI. Any of the following incorrect settings can cause the error:

  • Set the host field to the Databricks workspace hostname.
  • Set the login field to token.
  • Set the password field to the Databricks-generated personal access token.
  • Set the Extra field to a JSON string, where the key is token and the value is your personal access token.

The Databricks-generated personal access token is normally valid for 90 days. If the token expires, then this 403 Forbidden Error occurs.

Solution

  • Verify that the Extra field is correctly configured with the JSON string:

    {"token": "<your personal access token>"}

  • Verify that the token is mentioned in both the password field and the Extra field.

  • Verify that the host, login, and password fields are configured correctly.

  • Verify that the personal access token has not expired.

If necessary, generate a new token.