Problem

Jobs integrated with Git repositories that are triggered by a Databricks-managed service principal fail with the following error. 

Failed to checkout Git repository: PERMISSION_DENIED: Missing Git provider credentials. Go to User Settings > Git Integration to set up your Git credentials. 

You notice similar jobs triggered by individual users run successfully.

Cause

The service principal lacks the necessary Git credentials and permissions to access the GitHub repository. (When a user triggers a job, their Git credentials are used, allowing the job to access the GitHub repository without issues.)

Solution

Create an OAuth secret and then generate Git credentials for the service principal. 

Create OAuth secret

For steps, refer to the Authorize service principal access to Databricks with OAuth (AWS | Azure | GCP) documentation. 

Generate Git credentials for the service principal

You can generate Git credentials using the Databricks CLI (recommended) or the Git credentials API.

Using the Databricks CLI (recommended)

As a prerequisite, ensure you have Databricks CLI already installed. If not, refer to the Install or update the Databricks CLI (AWS | Azure | GCP)  documentation before continuing. 

1. Access the Databricks CLI from your local terminal or the web terminal inside your workspace. 
2. Configure ~/.databrickscfg to define a profile. You can use the following code. Set the following variables with your own values. 

• host with your workspace instance without the trailing forward slash. 
• client_id with the service principal application ID you copied in the previous section.
• client_secret with  the OAuth secret also copied in the previous section.

[<profile-name>]
host = <workspace-instance-without-trailing-slash>
client_id = <application-id>
client_secret = <oauth-client-secret>

3. Set your Git credential with the following command. Replace:

• <git-provider-short-name> with your Git provider (for example, Github).
• <git-provider-user-name> with your username or email address associated with the Git provider.
• <git-provider-access-token> with the personal access token (PAT) that you generate from your Git provider for authentication.
• <profile-name> with the profile name you configured in step 2. 

databricks git-credentials create <git-provider-short-name> --git-username <git-provider-user-name> --personal-access-token <git-provider-access-token> -p <profile-name>

4. Run the command. Verify the response output. It should display your credential id, your Git provider, and your Git username. 

    {
"credential_id": "XXXXXXX",
      "git_provider": "github",
      "git_username": "<your-user-name>"
    }

Using Git credential API

1. Generate an OAuth token for the service principal using the following command. Provide your own values for the variable placeholders.

export CLIENT_ID=<client-id>
export CLIENT_SECRET=<client-secret>
curl --request POST \
--url <token-endpoint-URL> \
--user "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'

Verify the response includes an access token, token type, and an expiration. You need the access token for the next step.

{
  "access_token": "<token>",
  "token_type": "Bearer",
  "expires_in": 3600
}

2. Generate the Git credentials. For details, refer to the Create a credential entry (AWS | Azure | GCP) API documentation. Provide the access token you created in the previous step for “personal_access_token”. 

"git_provider": <git-provider>,
"git_username": <git-user-name>,
"personal_access_token": <git-provider-access-token>

Confirm the service principal has Git credentials. You should see output from the API call similar to the following, with your own values provided. 

{
  "credential_id": "<credential-id>",
  "git_provider": <git-provider>,
  "git_username": <git-user-name>
}