Problem

You notice your jobs fail after upgrading Databricks Runtime to 11.3 LTS or above with a BindException error message.  

Caused by: java.net.BindException: Address already in use. 

Cause

Ipywidgets, introduced as of Databricks Runtime 11.2, by default occupies port 6062. If you have a third-party integration tool installed on your cluster, such as Datadog, when ipywidgets attempts to use port 6062, it will find the port already occupied by another service and throw the error.

Solution

Change the default port used by ipywidgets to another available port. 

1. Verify that other services in your environment are not already using your chosen port. Databricks also recommends avoiding the ports its services use: 443, 3306, 6666, 2443, and 8443. 
2. Set the Apache Spark configuration spark.databricks.driver.ipykernel.commChannelPort to a different port number when you create your cluster. Use the following code snippet.

spark.databricks.driver.ipykernel.commChannelPort <your-port-number>

For more information on configuring ipywidgets, refer to the ipywidgets (AWS | Azure | GCP) documentation.