Problem

You notice that your Databricks log delivery configuration is disabled, but you have not seen any alerts or notifications. Symptoms of this issue include the absence of expected log deliveries and potential disruptions in monitoring and auditing activities.

Cause

You may encounter a disabled log delivery configuration due to repeated user-side failures over a period of 14 days.

You may reach out to Databricks support to determine the cause from the audit logs.

Typically, audit logs can reveal that this occurs due to consistent errors when attempting to write files to the designated storage bucket. A common error in these cases is the 403 Forbidden response from Amazon S3, which can be linked to issues with IAM roles or bucket permissions. This blockage can stem from the AWS side, possibly due to network or configuration constraints that interfere with successful log delivery.

Solution

To resolve this issue, follow these steps:

1. Review the IAM role and storage bucket configuration to ensure they are set up correctly for the audit log delivery according to the Databricks documentation. For detailed instructions, review the following documentation:
   1. For Audit logs: Configure Audit Logs Delivery
   2. For System Tables:  Monitor costs using system tables and Enable System Schema.
2. Re-enable the log delivery configuration using the PATCH API or create a new configuration if necessary.
3. Monitor audit logs and the system.billing.usage table regularly by running any queries to confirm that log delivery is functioning as expected and that no user-side failures are occurring. If you are able to see the desired query results, it means the log configuration is functioning properly.
4. Avoid enforcing an IP blocklist, and use a dedicated bucket for Databricks logs to help prevent similar disruptions in the future.