Problem

In non-Unity Catalog (UC) workspaces, or workspaces with UC disabled, you experience previously working jobs fail to execute with the following error. 

METASTORE_DOES_NOT_EXIST: No metastore assigned for the current workspace.
Or 
com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=*****-****-****-****-******** ErrorClass=FEATURE_DISABLED] Unity Catalog is not available for feature tier STANDARD_TIER.

Cause

The error occurs for jobs created using the Databricks API without the data_security_mode field in the cluster properties, which are later updated through manual changes to the job cluster in the Databricks UI.

The data_security_mode field is crucial. When this field is missing, the job cluster will default to None. When a job is later updated in the UI, it sets to data_security_mode: SINGLE_USER.

When a cluster has data_security_mode: SINGLE_USER, it becomes UC enabled, triggering the issue.

Solution

Only use the Databricks update job API to update the job cluster, and set the data_security_mode: LEGACY_SINGLE_USER_STANDARD. 

For legacy passthrough, set data_security_mode: LEGACY_SINGLE_USER.

Review the Update jobs settings partially API documentation for more details regarding  data_security_mode.