Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline.

Written by pratik.bhawsar

Last published at: July 1st, 2022

Problem

You are using Delta Live Tables and want to change the cluster configuration.

You create a pipeline, but only have options to enable or disable Photon and select the number of workers.

Cause

When you create a Delta Live Table pipeline, most parameters are configured with default values. These values cannot be configured before the pipeline is created. 

Solution

You can change the cluster configuration after the pipeline is created.

  1. Click Workflows in the sidebar.
  2. Click the Delta Live Tables tab.
  3. Click the name of your pipeline.
  4. Click Settings.
    Pipeline details screenshot.
  5. On the Edit Pipeline Settings pop-up, click JSON.
    Edit pipeline settings screenshot.
  6. Edit the JSON to specify your cluster configuration. You can update all of the Delta Live Table settings (AWS | Azure | GCP) in the JSON file.
  7. Click Save.
  8. Click Start to start your pipeline with the new cluster configuration.