EC2 instance is performing an unusual DNS over HTTPS communication to Google DNS

This is a safe, and known, Ngrok fallback mechanism. No action required.

Written by julian.campabadal

Last published at: July 24th, 2025

Problem

Your SQL warehouse cluster is attempting to connect to the Google DNS 8.8.4.4, causing you some concern regarding security. You may see the following alert in GuardDuty.

"The EC2 instance <instance> is performing an unusual DNS Over HTTPS (DoH) communication with server 8.8.4.4.

 

Cause

The underlying cause of the observed DNS requests to the Google DNS 8.8.4.4 is related to Databricks' secure cluster connectivity feature and its fallback mechanism when the primary connection method fails. For details, refer to the “What is secure cluster connectivity?” section of the Classic compute plane networking documentation.

 

Databricks uses a secure tunnel using HTTPS (port 443) for clusters to connect to the control plane. This use is facilitated by an internal service that utilizes ngrok for creating a secure tunnel. 

 

If the ngrok tunnel relay fails due to network issues, ngrok timeout, firewall rules, or a bug, Databricks employs a fallback mechanism. This mechanism involves DNS-based resolution to maintain communication with the control plane. 

 

When the data plane loses network connectivity, it triggers the fallback mechanism. As a result, the data plane attempts to reach the control plane using the public internet to reestablish the session. This involves querying Google DNS for the A record related to the deployment region.

 

Solution

There is nothing you need to do. This is a known and safe behavior.

 

The fallback mechanism is logged in the VM logs, indicating the attempt to use Google DNS. The only reason that Databricks uses Google DNS is for fallback by ngrok for any transient failures. Otherwise, the cluster VMs will use the default VPC DNS settings.

 

The following message is an example of the fallback mechanism in the driver backend logs.

Aug 11 11:26:58 xxxxxxxxxxxxx ngrok[22145]: t=2023-08-11T11:26:58+0000 lvl=eror msg="failed to reconnect session" obj=csess id=<id> err="Get https://dns.google.com/resolve?cd=true&name=tunnel.<region>.azuredatabricks.net&type=A: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"