Problem
When creating an MLFlow experiment for feature engineering, you encounter an issue with the meta.yaml
file. You receive an error message.
Exception: Yaml file '/Workspace/Users/.bundle/<sample-model>/feature_engineering/features/mlruns/0/meta.yaml' exists as '/Workspace/Users/.bundle/<sample-model>/feature_engineering/features/mlruns/0/meta.yaml'
Cause
When MLflow is attempting to create a new experiment, it expects to write a fresh meta.yaml
file. If the file already exists, especially if it contains data from a previous, incomplete, or corrupted run, MLflow raises an exception to prevent accidental overwriting.
Solution
1. Use the below configuration in your code to set a tracking URI. This command properly directs MLflow to the Databricks-hosted tracking server and ensures conflicts within the local file structure are corrected.
mlflow.set_tracking_uri("databricks")
2. Ensure you have set DATABRICKS_HOST
and DATABRICKS_TOKEN
environment variables.
For more information, please refer to the “Where MLflow runs are logged” section of the Track model development using MLflow (AWS | Azure | GCP ) documentation.
Alternatively, remove the corrupted meta.yaml
file using the following command in a notebook.
%sh rm -rf “/Workspace/Users/.bundle/<sample-model>/feature_engineering/features/mlruns/0/meta.yaml”