Conda fails to download packages from Anaconda

Conda fails to download packages with PackagesNotFoundError when you try to install packages from Anaconda....

Download artifacts from MLflow

How to download artifacts from MLflow to local storage....

How to extract feature information for tree-based Apache SparkML pipeline models

Learn how to extract feature information for tree-based ML pipeline models in Databricks....

Fitting an Apache SparkML model throws error

Learn how to resolve errors thrown by Databricks when fitting a SparkML model or pipeline....

H2O.ai Sparkling Water cluster not reachable

H2O.ai Sparkling Water cluster not reachable if the version of the Sparkling Water package does not match the version of Spark used on your cluster....

How to perform group K-fold cross validation with Apache Spark

Learn how to perform group K-fold cross validation with Apache Spark on Databricks....

Error when importing OneHotEncoderEstimator

You get an error message when trying to import OneHotEncoderEstimator....

MLflow project fails to access an Apache Hive table

Resolve "Table or view not found" error when an MLflow project fails to access an Apache Hive table....

How to speed up cross-validation

Learn how to improve cross-validation performance in SparkML with Databricks....

Hyperopt fails with maxNumConcurrentTasks error

Do NOT install Hyperopt on a Databricks Runtime for Machine Learning cluster....

Incorrect results when using documents as inputs

Your model does not return expected results when documents are input using TfidfVectorizer. JSON array...

Errors when accessing MLflow artifacts without using the MLflow client

Resolve errors when attempting to access MLflow artifacts without using the MLflow client...

Experiment warning when custom artifact storage location is used

Resolve experiment warnings when a custom artifact storage location is used instead of the MLflow managed location....

Experiment warning when legacy artifact storage location is used

Resolve experiment warnings when a legacy artifact storage location is used instead of the MLflow managed location....

KNN model using pyfunc returns ModuleNotFoundError or FileNotFoundError

Predictions using pyfunc on a KNN model returns a ModuleNotFoundError or FileNotFoundError....

OSError when accessing MLflow experiment artifacts

Resolve an `OSError` when trying to access, download, or log MLflow experiment artifacts....

PERMISSION_DENIED error when accessing MLflow experiment artifact

Resolve a PERMISSION_DENIED error when trying to access MLflow experiment artifacts....

Python commands fail on Machine Learning clusters

Python commands are failing on Databricks Runtime for Machine Learning clusters. Conda....

Runs are not nested when SparkTrials is enabled in Hyperopt

When SparkTrials is enabled in Hyperopt, MLflow runs are not nested under the parent run....

MLflow 'invalid access token' error

Long running ML tasks require an access token with an extended lifetime to ensure the tasks complete before the token expires....

Unable to update a serving endpoint even with necessary permissions

Find the creating user in the workspace and re-add the permissions....

Vector search index contains incorrect number of rows

Ensure that the Delta table has a unique primary key....

ModuleNotFoundError: No module named 'packaging' when creating GPU Model Serving endpoint

Add pip_requirements during model logging. ...

Status FAILURE when attempting to update a Direct Vector Access Index

Ensure that the embedding_dimension parameter matches the length of the embedding field. ...

Table not available while creating AutoML experiment model

Migrate to Unified Compute (UC), with the option to use AutoML with the Python API in the meantime....

Failing API calls in MLflow because of Float64 column values

Explicitly log the input schema while creating the model....

Disable or restrict access to the foundation model APIs

Set rate limits to zero....

Trying to load an MLflow model using a Python script returns Py4JJavaError

Upgrade MLflow to version 2.15.0 or above. ...

TypeError with an unexpected keyword argument 'query_type' when attempting to perform hybrid similarity search using the databricks-vectorsearch package

Ensure you have the latest version of the databricks-vectorsearch package installed in your environment....

TABLE_ONLINE_VECTOR_INDEX_REPLICA does not support Lakehouse Federation

Use the Python SDK or the REST API for querying a vector search endpoint. ...

Error message when trying to send a JSON input data request to a model endpoint

Pass one more input parameter from the list of input columns in the endpoint’s model signature....

CUDA out of memory error message in GPU clusters

Change the GPU device used by your driver and/or worker nodes....

Slow model fitting when implementing Alternating Least Squares using Apache Spark PySpark

Override the block default to match the total cores available and consider using compute-optimized instances. ...

Vector search index queries with the `%` character not performing partial-string matches

When using the `LIKE` operator in vector search filters, specify the exact string you want to match...

Column name error when using Apache Spark Mlib feature transformers

When flattening the DataFrame, rename nested columns using an underscore instead of a dot....

Logging a model with MLflow in a PySpark pipeline throws a TempDir class assertion error

Upgrade your MLflow version to 2.16.0 or higher....

MLflow API 429 errors when transitioning models

Add retry logic with exponential backoff to avoid hitting the rate limit....

Model serving endpoint creation succeeds but deployment fails and error stack trace has message _ARRAY_API not found

Include NumPy as a pip dependency and specify the version range to be installed. ...

Azure Storage gives error AuthorizationFailure while using MLflow

Configure both DFS and Blob storage endpoints....

Slowness when using the foundational model API with pay-per-token mode

Switch to provisioned throughput mode for high throughput and performance guarantee requirements....

Getting ValueError: ndarray is not supported by dataframe_to_mds when converting an Apache Spark DataFrame to MDS format using Mosaic Streaming

Properly pass the data type of the elements of the array column in the mds_kwargs....

Spark ML to ONNX Model Conversion does not produce the same model - predictions differ

Define the TARGET_OPSET and then pass it as the target_opset parameter of the convert_sparkml function. ...

MLflow exception error when trying to migrate models from Workspace Model Registry

Set the registry URI to the Workspace Model Registry before running the MLflow operations. ...

Loading models using MLflow causes TypeError around unexpected number of arguments

Run your cluster using a Databricks Runtime version that has the same Python version as the one used to log and register your model....

ExecutionException error when trying to use Conda as an environment manager in MLflow

Update the MLflow environment manager from Conda to virtualenv. ...

MLflow error "INVALID_PARAMETER_VALUE" during model training and logging process

Ensure each MLflow run maintains a unique set of parameters or use nested runs to log each parameter distinctly within a session. ...

SparkException error when trying to use an Apache Spark UDF to create and dynamically pass a prompt to the ai_query() function

Use Unity Catalog (UC) UDFs instead of Spark UDFs. ...

MLflow models saving to local path when registered instead of to the desired registry

Databricks recommends managing your model lifecycle in Unity Catalog....

“Connection pool is full” error when pulling models from S3 with MLflow

Increase the maximum size of the MLflow connection pool....

Using MLflow API call to load a model taking the same amount of time every call and artifacts downloading from scratch

Save the model artifacts locally and then load the model from a local path....

java.lang.OutOfMemoryError error when using collect() from sparklyr

Use arrow_collect() in a custom function to avoid Spark’s 2GB limit when collecting large datasets in R....

Parameter workload_size always executing SMALL when using the databricks-agents library to update existing model serving endpoints

Update the databricks-agents library to version 0.17.0 or later....

'INVALID_PARAMETER_VALUE' error when creating a Google Vertex AI serving endpoint

Ensure the entire private key is used....

Model serving endpoint creation fails with BadRequest error

Shorten the catalog name or specify the endpoint_name explicitly when using agents.deploy....

Not enough disk space error when downloading a model from Hugging Face

Change the default download directory from the root partition to a location with available space....

SHAP figure not appearing in the artifacts after running the mflow.evaluate() call, despite setting log_model_explainability = True

Ensure that all features are in the expected format and re-run the call. ...

Error when trying to use VectorAssembler while on a standard access mode cluster with a non-ML Databricks Runtime

Use a Dedicated (formerly single user) access mode cluster, assigned to a group of users....

Getting NoModuleFound or attribute error when using the Flash Attention model in MLflow

Use mlflow.transformers.log_model with a custom wheel version of flash-attn....

GPU metrics indicate that the GPU is not being used during model inference

Ensure that you are sending the model to the GPU in your code....

AttributeError: 'ExportMetricsResponse' when retrieving serving endpoint metrics

Use a custom Python script to download and save the serving endpoint metrics to a file in the Prometheus format....

MlflowClient().search_runs returns only a subset of runs

Use mlflow.search_runs() instead of MlflowClient.search_runs()....

Function ai_similarity failing with “Unexpected server response" error

Remove NULL values from the data before passing it to the function. ...

Tackling schema issues that arise for ML models trained outside of Databricks

Use the code from the external environment to retrain the model within Databricks before fine-tuning....

Serving an AutoML model failing when deployed to an endpoint with "Failed to deploy modelName: served entity creation aborted" error

Ensure the model environment includes an explicit version pin for NumPy....

SSL error when invoking Databricks model serving endpoint

Reduce or split the payload size....

PERMISSION_DENIED error while running AutoML experiment with group-assigned cluster

Create a /Workspace/Groups/ folder for each group you plan to use with a group cluster to ensure correct permissions....