SSL exception when connecting to GCP secret manager

GCP secret manager requires GCM cipher suites to be installed on your cluster. Databricks Runtime 10.4 LTS and above have GCM cipher suites enabled by default.

Written by John.Lourdu

Last published at: January 20th, 2023
Delete

Info

This article applies to clusters using Databricks Runtime 7.3 LTS and 9.1 LTS. 

Problem

Secrets stored in the GCP secret manager service can be retrieved using the google-cloud-secret-manager client library.

Your code may fail with an SSLHandshakeException error message on Databricks Runtime 9.1 LTS and below.

Sample code:

import com.google.cloud.secretmanager.v1.AccessSecretVersionResponse;
import com.google.cloud.secretmanager.v1.ProjectName;
import com.google.cloud.secretmanager.v1.Replication;
import com.google.cloud.secretmanager.v1.Secret;
import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretPayload;
import com.google.cloud.secretmanager.v1.SecretVersion;
import com.google.protobuf.ByteString;
import com.google.cloud.secretmanager.v1.SecretName;
import com.google.cloud.secretmanager.v1.SecretManagerServiceSettings
import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.auth.Credentials

def access_secret_version(secret_id, version_id="latest"):
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the secret version.
    name = f"projects/{PROJECT_ID}/secrets/{secret_id}/versions/{version_id}"

    # Access the secret version.
    response = client.access_secret_version(name=name)

    # Return the decoded payload.
    return response.payload.data.decode('UTF-8')    
import hashlib

def secret_hash(secret_value): 
  # return the sha224 hash of the secret value
 
 return hashlib.sha224(bytes(secret_value, "utf-8")).hexdigest()

Error message:

UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
Caused by: StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
Caused by: SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]

Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)

Cause

GCM (Galois/Counter Mode) cipher suites are not enabled by default on Databricks Runtime 9.1 LTS and below.

Without the GCM cipher suites, there is no protocol available to establish the expected SSL connection to the GCP secret manager.

You can use the nmap utility to verify which cipher suites are required by the external server.

%sh

nmap --script ssl-enum-ciphers -p 443 secretmanager.googleapis.com

Delete

Info

If nmap is not installed, run sudo apt-get install -y nmap to install it on your cluster.

Solution

You must enable GCM cipher suites to connect to the GCP secret manager service.

If you upgrade to Databricks Runtime 10.4 LTS and above, GCM cipher suites are enabled by default.

If you stay on Databricks Runtime 9.1 LTS and below, you should follow the instructions in the Enable GCM cipher suites knowledge base article to install GCM cipher suites on your cluster.

Once you have enabled GCM cipher suites, you can connect to the GCP secret manager service.