Clusters using Docker databricksruntime:latest tag are not starting

Replace latest tag with LTS for Databricks Docker image.

Written by zhengxian.huang

Last published at: January 30th, 2025

Problem

When using Docker containers based off databricksruntime and using the latest tag, your clusters with Docker Container Services (DCS) in Databricks hang or fail to execute Python cells even though they were previously operational.

 

usr/lib/python3.8/asyncio/base_events.py:1829: RuntimeWarning: coroutine 'DatabricksKernel.do_execute' was never awaited
handle = self._ready.popleft()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

 

 

Cause

Use of the latest tag is no longer supported. 

 

Solution

Update the Docker image to use a specific Databricks Runtime LTS version instead of the latest tag. For example, use databricksruntime/standard:12.2-LTS.

 

Preventative measures

  • Avoid using the latest tag for Docker images and always specify a supported runtime version. 
  • Regularly review Databricks release notes and updates to stay informed about changes that may impact your environment.

For further reading and setting up clusters using the image, refer to the Customize containers with Databricks Container Service (AWS | Azure) documentation.