Problem

When deploying Databricks Asset Bundles (DABs) through the CICD pipeline, the process fails with a 403 error, despite successful manual deployments from the Linux server. 

Error message

403 Forbidden</h1></cente... (22 more bytes) pid=57 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload sdk=true
17:03:47 DEBUG non-retriable error: unable to parse response. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues

Cause

The Asset Bundle deployment process is inadvertently including hidden system directories (such as .git and .cache) during workspace deployment. When the CICD pipeline attempts to process these hidden directories, it triggers permission-related operations that are not authorized in the Databricks workspace environment. 

While manual deployments from the Linux server may succeed due to different permission handling, the CICD pipeline's strict security controls prevent unauthorized modifications to these system-level directories, resulting in the 403 permission denied error. This behavior specifically manifests in the CICD environment due to its more restricted security context compared to local deployments. 

Solution

Configure the .gitignore file in your GitLab repository to exclude unnecessary hidden directories from the Asset Bundle deployment. Excluding unnecessary hidden directories prevents the CICD pipeline from attempting to process system-level directories that shouldn't be part of the workspace deployment.

Update your repository's .gitignore file to exclude the following directories.

• .git/
• .cache/.
• Any other hidden system directories that aren't required for your Databricks workspace.

This solution maintains the security integrity of your deployment process while ensuring that only relevant assets are included in the Asset Bundle.