Problem

You’re trying to connect to a custom endpoint within the same VPC/VNet as your workspace. The endpoint is hosted on non-standard ports (ports other than HTTP-80 or HTTPS-443). You receive a Connection Refused error during connection attempts. 

You notice the issue when working on a standard (formerly shared) access mode cluster on Databricks Runtime 11.3 LTS or below. 

Cause

Databricks Runtimes 11.3 LTS and below restrict outbound access to certain ports by default on standard access mode clusters, even within the same VPC/VNet. 

Solution

Databricks recommends using Databricks Runtime 12.2 LTS or above with standard access mode. 

If you prefer to continue using Databricks Runtime 11.3 LTS or below, use the following cluster-scoped init script to allow access to the custom endpoint. This script continuously ensures that outbound traffic to the specified port and CIDR is allowed from the cluster nodes.

#!/bin/bash
cat << 'EOF' > /tmp/set_rules.sh
#!/bin/bash
set -x
sleep_interval=30s

port="<your-target-port>"  ## Change this to your target port
cidr="<your-workspace-VPC-or-VNET-CIDR>"  ## Replace with your workspace VPC/VNet CIDR

while true; do
    rules=$(iptables -L | grep -i "$port")
    if [[ "$rules" != *"dpt:$port"* && $(getent group spark-users) ]]; then
        echo "Changing rules at $(date)" 
        iptables -I OUTPUT 2 -d $cidr -j ACCEPT -p tcp --dport $port
    fi
    sleep ${sleep_interval}
done
EOF

chmod a+x /tmp/set_rules.sh
/tmp/set_rules.sh >> /tmp/set_rules.log & disown