Use iptables to access the EC2 metadata server

You must setup custom iptables rules in order to access the EC2 metadata server. You cannot access it by default.

Written by manoj.hegde

Last published at: October 25th, 2022

Problem

You are trying to access the metadata server for your EC2 instance but cannot connect.

Cause

This is the intended, default behavior. It is functioning as designed.

Solution

Use an init script to apply a custom iptables configuration to your Databricks cluster which enables access to the metadata server.

  1. Define a location to store the init script. If you do not already have a folder for your init script, you must create one. For example, using dbfs:/databricks/<init-script-folder>:
    %scala
    
    dbutils.fs.mkdirs("dbfs:/databricks/<init-script-folder>/")
  2. Create the init script:
    %scala
    
    dbutils.fs.put("dbfs:/databricks/<init-script-folder>/iptables.sh","""
    #!/bin/bash 
    sudo iptables -A INPUT -s 169.254.169.254 -j ACCEPT 
    sudo iptables -A OUTPUT -d 169.254.169.254 -j ACCEPT
    """,True)
  3. Verify that the init script was created on your cluster:
    %scala
    
    display(dbutils.fs.ls("dbfs:/databricks/<init-script-folder>/iptables.sh"))
  4. Configure the init script as a cluster-scoped init script on your cluster.
  5. Restart your cluster. 

After the cluster restarts, the init script takes effect. You now have access to the metadata server for your EC2 instance.

Was this article helpful?