Invalid Access Token
error when running jobs with Airflow
Problem
When you run scheduled Airflow Databricks jobs, you get this error:
Invalid Access Token : 403 Forbidden Error
Cause
To run or schedule Databricks jobs through Airflow, you need to configure the Databricks connection using the Airflow web UI. Any of the following incorrect settings can cause the error:
- Set the
host
field to the Databricks workspace hostname. - Set the
login
field totoken
. - Set the
password
field to the Databricks-generated personal access token. - Set the
Extra
field to a JSON string, where the key istoken
and the value is your personal access token.
The Databricks-generated personal access token is normally valid for 90 days. If the token expires, then this 403 Forbidden Error
occurs.
Solution
Verify that the
Extra
field is correctly configured with the JSON string:{"token": "<your personal access token>"}
Verify that the token is mentioned in both the
password
field and theExtra
field.Verify that the
host
,login
, andpassword
fields are configured correctly.Verify that the personal access token has not expired.
If necessary, generate a new token.