Find your workspace ID

Learn how to find your Databricks workspace ID in the web UI as well as via a notebook command....

Cannot start Databricks workspace from Google Cloud console

Check to make sure your Databricks subscription is active and that you are using the correct Google account....

Failed to add user error due to email or username already existing with a different case

You should ensure casing for usernames is consistent across all accounts and providers in your system....

Cannot access Databricks secrets when using a "No isolation shared" cluster

You cannot use dbutils.secrets.get() when admin protection for No isolation shared clusters is enabled in your account....

Setup cross account bucket access in Google Cloud

Use service accounts to grant access to storage buckets in another account....

Configure Simba ODBC driver with a proxy in Windows

How to configure the Simba ODBC driver to connect through a proxy server when using Windows....

Troubleshooting JDBC and ODBC connections

Learn how to troubleshoot Databricks JDBC and ODBC connection errors....

Install a private PyPI repo

How to install libraries from private PyPI repositories....

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Admin user cannot restart cluster to run job

Learn how to re-grant privileges to Databricks Admin users....

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Set executor log level

Learn how to set the log levels on Databricks executors....

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Enable retries in init script

Add a retry function to your init script....

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Run a custom Databricks runtime on your cluster

Configure your cluster to run a custom Databricks runtime image via the UI or API....

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Append to a DataFrame

Learn how to append to a DataFrame in Databricks....

How to improve performance with bucketing

Learn how to improve Databricks performance by using bucketing....

How to handle blob data contained in an XML file

Learn how to handle blob data contained in an XML file....

Simplify chained transformations

Learn how to simplify chained transformations on your DataFrame in Databricks....

Hive UDFs

Learn how to create and use a Hive UDF for Databricks....

Prevent duplicated columns when joining two DataFrames

Learn how to prevent duplicated columns when joining two DataFrames in Databricks....

Revoke all user privileges

Use a regex and a series of for loops to revoke all privileges for a single user....

How to list and delete files faster in Databricks

Learn how to list and delete files faster in Databricks....

How to handle corrupted Parquet files with different schema

Learn how to read Parquet files with a specific schema using Databricks....

No USAGE permission on database

User does not have USAGE permission on the database....

Nulls and empty strings in a partitioned column save as nulls

Learn why nulls and empty strings in a partitioned column save as nulls in Databricks....

Behavior of the randomSplit method

Learn about inconsistent behaviors when using the randomSplit method in Databricks....

Generate schema from case class

Learn how to generate a schema from a Scala case class....

How to specify skew hints in dataset and DataFrame-based join commands

Learn how to specify skew hints in Dataset and DataFrame-based join commands in Databricks....

How to update nested columns

Learn how to update nested columns in Databricks....

Incompatible schema in some files

Learn how to resolve incompatible schema in Parquet files with Databricks....

Unable to infer schema for ORC error

Apache Spark returns an error for ORC files if no schema is defined when reading from an empty directory or a base path with multiple subfolders....

User does not have permission SELECT on ANY File

Regular users cannot create tables without permission when access control is enabled....

Create tables on JSON datasets

Create tables on JSON datasets; requires SerDe JAR....

Optimize read performance from JDBC data sources

Learn how to optimize performance when reading from JDBC data sources in Databricks....

Failure to detect encoding in JSON

Learn how to resolve a failure to detect encoding of input JSON files when using BOM with Databricks....

Inconsistent timestamp results with JDBC applications

Timestamp records are inconsistent with JDBC applications when daylight saving time adjustments are made....

Kafka client terminated with OffsetOutOfRangeException

Kafka client is terminated with `OffsetOutOfRangeException` when trying to fetch messages...

Recursive references in Avro schema are not allowed

Apache Avro data sources cannot have recursive references in the schema when used with Spark....

SQL access control error when using Snowflake as a data source

Snowflake does not officially support schema as an option; you must use sfschema....

FileReadException on DBFS mounted filesystem

Use dbutils.fs.refreshMounts() to refresh mount points before referencing a DBFS path in your Spark job....

Null column values display as NaN

Null column values correctly display as NaN in Databricks SQL....

Retrieve queries owned by a disabled user

How to retrieve queries owned by a disabled user in Databricks SQL....

Job timeout when connecting to a SQL endpoint over JDBC

Increase the SocketTimeout value in the JDBC connection URL to prevent thread requests from timing out....

Slowness when fetching results in Databricks SQL

Ensure that cloud fetch is enabled for best performance when using ODBC/JDBC to fetch results....

ZORDER results in "Hilbert indexing can only be used on 9 or fewer columns" error

OPTIMIZE ZORDER BY command has a hard limit of nine columns....

Cannot customize Apache Spark config in Databricks SQL warehouse

You can only configure a limited set of global Spark properties when using a SQL warehouse....

Apache Spark session is null in DBConnect

A `sparkSession is null while trying to executeCollectResult` error message occurs when using DBConnect....

Failed to create process error with Databricks CLI in Windows

Databricks CLI may not work correctly in Windows if your Python path has a space in it....

GeoSpark undefined function error with DBConnect

Use GeoSpark code with a DBConnect session....

Get Apache Spark config in DBConnect

Use a REST API call and DBConnect to get the Apache Spark configuration for your cluster....

ProtoSerializer stack overflow error in DBConnect

A stack overflow error in DBConnect indicates that you need to allocate more memory on the local PC....

Use tcpdump to create pcap files

Analyze network traffic between nodes on a specific cluster by using tcpdump to create pcap files....

Terraform registry does not have a provider error

You cannot install the Databricks Terraform provider if the required_providers block is not defined in your modules....

Databricks Connect job fails after a Databricks Runtime update

Use the most recent version of Databricks Connect that matches your Databricks Runtime version to avoid an error....

Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Identify duplicate data on append operations

...

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Recover from a DELTA_LOG corruption error

Learn how to repair a Delta table that reports an IllegalStateException error when queried....

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table

Delta Sharing has limits on the metadata size of a shared table. If you exceed these limits it generates an error....

Found duplicate columns error blocks creation of a Delta table

Duplicate column names are not allowed in Delta tables....

Distinguish active and dead jobs

Learn how to distinguish between active and dead Databricks jobs....

Spark job fails with Driver is temporarily unavailable

Job failure due to Driver being unavailable or unresponsive....

How to delete all jobs using the REST API

Learn how to delete all Databricks jobs using the REST API....

Job cluster limits on notebook output

Job clusters have a maximum notebook output size of 20 MB. If the output is larger, it results in an error....

Job fails, but Apache Spark tasks finish

Your job fails, but all of the Apache Spark tasks have completed successfully. You are using spark.stop() or System.exit(0) in your code....

Job fails due to job rate limit

Learn how to resolve Databricks job failures due to job rate limits....

Job fails with invalid access token

Jobs that run more than 48 hours fail with invalid access token error when the dbutils token expires....

Task deserialization time is high

Configure cluster-installed libraries to install on executors at cluster launch vs executor launch to speed up your job task runs....

Pass arguments to a notebook as a list

Use a JSON file to temporarily store arguments that you want to use in your notebook....

Uncommitted files causing data duplication

Partially uncommitted files from a failed write can result in apparent data duplication. Adjust VACUUM settings to resolve the issue....

Multi-task workflows using incorrect parameter values

If parallel tasks running on the same cluster use Scala companion objects the wrong values can be used due to sharing a single class in the JVM....

Job fails with Spark Shuffle FetchFailedException error

Disable the default Spark Shuffle service to work around a FetchFailedException error....

Users unable to view job results when using remote Git source

Databricks does not manage permission for remote repos, so you must sync changes with a local notebook so non-admin users can view results....

Single scheduled job tries to run multiple times

Ensure your cron syntax is correct when scheduling jobs. A wildcard in the wrong space can produce unexpected results....

Add custom tags to a Delta Live Tables pipeline

Manually edit the JSON configuration file to add custom tags....

Update notification settings for jobs with the Jobs API

You can use the Jobs API to add email notifications to some, or all, of the jobs in your workspace....

Stop all scheduled jobs

Use the included sample code to stop all of your scheduled jobs in the workspace....

Cannot import TabularPrediction from AutoGluon

Cannot import TabularPrediction from AutoGluon v0.0.14 due to a namespace collision. Upgrade to AutoGluon v0.0.15....

How to correctly update a Maven library in Databricks

Learn how to correctly update a Maven library in Databricks....

Init script fails to download Maven JAR

Cluster init script fails to download a Maven JAR when trying to install a library....

Install package using previous CRAN snapshot

Avoid a package install error by installing from an earlier CRAN snapshot....

Install PyGraphViz

Install PyGraphViz with all required dependencies....

Install Turbodbc via init script

Install Turbodbc and its dependencies, libboost-all-dev, unixodbc-dev, and python-dev, with an init script....

Cannot uninstall library from UI

Learn what to do when you can't uninstall a library using the Databricks user interface....

Error when installing Cartopy on a cluster

Cartopy installation fails if libgeos and libproj are not installed....

Error when installing pyodbc on a cluster

Learn how to troubleshoot an error when installing pyodbc on a Databricks cluster....

Libraries fail with dependency exception

Learn why notebook-scoped libraries trigger an Apache Spark dependency exception; return a requirement cannot be satisfied error....

Reading .xlsx files with xlrd fails

xlrd no longer supports .xlsx files. Use openpyxl to read .xlsx files....

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath....

Replace a default library jar

Learn how to replace a default Java or Scala library jar with another version....

Python command fails with AssertionError: wrong color format

Resolve a wrong color format AssertionError caused by nbconvert when a Python command fails....

PyPMML fails with Could not find py4j jar error

...

TensorFlow fails to import

TensorFlow fails to import if you have an incompatible version of protobuf installed on your cluster....

Verify the version of Log4j on your cluster

Verify the version of Log4j installed on your cluster and upgrade if required....

Apache Spark jobs fail with Environment directory not found error

Spark jobs appear to time out after you install a library because security rules are preventing workers from resolving the Python executable path....

Copy installed libraries from one cluster to another

Copy libraries from a source cluster to a target cluster with a custom Python script....

Failed to install Elasticsearch via Maven

If library dependencies are already installed, it can result in a library installation failure....

Cluster fails to start with Cloud provider launch failure error

If the network security group policy is not correctly configured your clusters will fail to start....

Add libraries to a job cluster to reduce idle time

How to add libraries to a job cluster and reduce idle time in Databricks...

PyArrow hotfix breaking change

PyArrow versions 0.14 - 14.0.0 contain a security vulnerability....

Conda fails to download packages from Anaconda

Conda fails to download packages with PackagesNotFoundError when you try to install packages from Anaconda....

Download artifacts from MLflow

How to download artifacts from MLflow to local storage....

How to extract feature information for tree-based Apache SparkML pipeline models

Learn how to extract feature information for tree-based ML pipeline models in Databricks....

Fitting an Apache SparkML model throws error

Learn how to resolve errors thrown by Databricks when fitting a SparkML model or pipeline....

H2O.ai Sparkling Water cluster not reachable

H2O.ai Sparkling Water cluster not reachable if the version of the Sparkling Water package does not match the version of Spark used on your cluster....

How to perform group K-fold cross validation with Apache Spark

Learn how to perform group K-fold cross validation with Apache Spark on Databricks....

MLflow project fails to access an Apache Hive table

Resolve "Table or view not found" error when an MLflow project fails to access an Apache Hive table....

How to speed up cross-validation

Learn how to improve cross-validation performance in SparkML with Databricks....

Hyperopt fails with maxNumConcurrentTasks error

Do NOT install Hyperopt on a Databricks Runtime for Machine Learning cluster....

Incorrect results when using documents as inputs

Your model does not return expected results when documents are input using TfidfVectorizer. JSON array...

Experiment warning when custom artifact storage location is used

Resolve experiment warnings when a custom artifact storage location is used instead of the MLflow managed location....

Experiment warning when legacy artifact storage location is used

Resolve experiment warnings when a legacy artifact storage location is used instead of the MLflow managed location....

KNN model using pyfunc returns ModuleNotFoundError or FileNotFoundError

Predictions using pyfunc on a KNN model returns a ModuleNotFoundError or FileNotFoundError....

OSError when accessing MLflow experiment artifacts

Resolve an `OSError` when trying to access, download, or log MLflow experiment artifacts....

PERMISSION_DENIED error when accessing MLflow experiment artifact

Resolve a PERMISSION_DENIED error when trying to access MLflow experiment artifacts....

Runs are not nested when SparkTrials is enabled in Hyperopt

When SparkTrials is enabled in Hyperopt, MLflow runs are not nested under the parent run....

Autoscaling is slow with an external metastore

Improve autoscaling performance by only installing metastore jars to the driver....

Data too long for column error

If a column exceeds 4000 characters it is too big for the default datatype and returns an error....

Drop database without deletion

Use Hive commands to drop a database without deleting the underlying storage folder....

Error in CREATE TABLE with external Hive metastore

CREATE TABLE error with MySQL 8.0 in external Hive metastore due to charset....

Japanese character support in external metastore

Use Japanese characters in tables in an external metastore....

Parquet timestamp requires Hive metastore 1.2 or above

Update the Hive metastore to version 1.2 or above to use TIMESTAMP with a Parquet table....

JSON reader parses values as null

When you read a JSON file, the Spark JSON reader returns null values instead of the actual data....

display() does not show microseconds correctly

Use show() to display timestamp values with microsecond precision. display() is limited to millisecond precision....

Error: Received command c on object id p0

You see the error message `INFO:py4j.java_gateway:Received command c on object id p0` after running Python code with imported libraries....

Failure when accessing or mounting storage

Do not mount storage to the root mount path....

Update job permissions for multiple users

Use the job permissions API to update permissions for multiple users....

Generate browser HAR files

Learn how to record HAR files in your web browser. These are very useful when troubleshooting UI issues....

Recover deleted notebooks from the Trash

Deleted items can be recovered from the Trash for 30 days after deletion....

Get workspace configuration details

Display the complete configuration details for your Databricks workspace....

Iterate through all jobs in the workspace using Jobs API 2.1

Use the Jobs API 2.1 to iterate through and display a list of jobs in your workspace....

Too many execution contexts are open right now

Reduce the number of notebooks used to limit the number of execution contexts required for your job....

Generate a list of all workspace admins

Use the included sample code to generate a list of all workspace admins....

SSL exception when connecting to GCP secret manager

GCP secret manager requires GCM cipher suites to be installed on your cluster. Databricks Runtime 10.4 LTS and above have GCM cipher suites enabled by default....

Append output is not supported without a watermark

Append output mode is not supported on aggregated DataFrames without a watermark....

Apache Spark DStream is not supported

DStreams are not supported in Databricks. Migrate from DStream API to Structured Streaming....

Streaming with File Sink: Problems with recovery if you change checkpoint or output directories

Learn how to resolve issues that occur with recovery if you change checkpoint or output directories when streaming with File Sink....

Get the path of files consumed by Auto Loader

Get the path and filename of all files consumed by Auto Loader and write them out as a new column....

How to restart a structured streaming query from last written offset

Learn how to restart a structured streaming query from the last written offset....

Kafka error: No resolvable bootstrap urls

A 'No resolvable bootstrap urls' error occurs when you try to read or write data to a Kafka stream....

readStream() is not whitelisted error when running a query

readStream() is not whitelisted error on clusters that have table access control enabled....

Checkpoint files not being deleted when using display()

Learn how to prevent display(streamingDF) checkpoint files from using a large amount of storage....

Checkpoint files not being deleted when using foreachBatch()

Learn how to prevent foreachBatch() checkpoint files from using a large amount of storage....

Conflicting directory structures error

You should use distinct paths in the storage location, otherwise conflicting directory structures may result in an error....

RocksDB fails to acquire a lock

When using RocksDB as a state store, you may need to increase the acquire timeout in the SQL config....

Streaming job gets stuck writing to checkpoint

Streaming job appears to be stuck even though no error is thrown. You are using DBFS for checkpoint storage, but it has filled up....

Explicit path to data or a defined schema required for Auto loader

If you do not specify an explicit path to your data or define your data schema, you get an IllegalArgumentException error when you start an Auto loader job....

Optimize streaming transactions with .trigger

Use .trigger to define the storage update interval. A higher value reduces the number of storage transactions....

Structured streaming jobs slow down on every 10th batch

Automatic compaction of the metadata folder can slow down structured streaming jobs....

Get last modification time for all files in Auto Loader and batch jobs

Define a UDF to list all files in the path and return the last modification time for each one....

AttributeError: ‘function’ object has no attribute

Using protected keywords from the DataFrame API as column names results in a function object has no attribute error message....

Convert Python datetime object to string

Display date and time values in a column, as a datetime object, and as a string....

Display file and directory timestamp details

Display file creation date and modification date using Python....

Reading large DBFS-mounted files using Python APIs

Learn how to resolve errors when reading large DBFS-mounted files using Python APIs....

How to import a custom CA certificate

Learn how to import a custom CA certificate into your Databricks cluster for Python use....

Job remains idle before starting

Apache Spark jobs remain idle for a long time before starting....

List all workspace objects

List all Databricks workspace objects under a given path....

Python commands fail on high concurrency clusters

Python commands fail on high concurrency clusters with Apache Spark process isolation and shared session enabled. WARN error message....

Cluster cancels Python command execution after installing Bokeh

Learn what to do when your Databricks cluster cancels Python command execution after you install Bokeh....

Cluster cancels Python command execution due to library conflict

Learn what to do when your Databricks cluster cancels Python command execution due to a library conflict....

Python command execution fails with AttributeError

Learn what to do when a Python command in your Databricks notebook fails with AttributeError....

Job fails with Java IndexOutOfBoundsException error

When groupby() is used along with applyInPandas it generates an exception due to an arrow buffer limitation....

Job fails with NoSuchElementException error

NoSuchElementException errors can occur when using Apache Arrow....

Job fails with IndexOutOfBoundsException and ArrowBuf errors

When Groupby is used with applyinPandas it can result in Apache Arrow buffer size estimation errors....

Field name sorting changes in Apache Spark 3.x

Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically....

Job fails with "not enough memory to build the hash map" error

You should use adaptive query execution instead of explicit broadcast hints to perform joins on Databricks Runtime 11.3 LTS and above....

Change version of R (r-base)

Learn how to change the version of R on your Databricks cluster....

Fix the version of R packages

Learn how to fix the version of R packages....

How to parallelize R code with gapply

Learn how to parallelize R code using gapply....

How to parallelize R code with spark.lapply

Learn how to parallelize R code using spark.lapply....

Install rJava and RJDBC libraries

Learn how to install rJava and RJDBC libraries on your Databricks cluster....

Rendering an R markdown file containing sparklyr code fails

Learn how to resolve failures when rendering an R markdown file containing sparklyr....

Resolving package or namespace loading error

Learn how to resolve package or namespace loading errors in a Databricks notebook....

RStudio server backend connection error

RStudio server backend connection error occurs if you exceed the maximum number of RBackends on your cluster....

Verify R packages installed via init script

Verify that R packages successfully installed via an init script. List all R packages that failed to install....

Apache Spark UI is not in sync with job

Status of Spark jobs gets out of sync with the Spark UI when events drop from the event queue before being processed....

Apache Spark job fails with Parquet column cannot be converted error

Parquet column cannot be converted error appears when you are reading decimal data in Parquet format and writing to a Delta table....

Best practice for cache(), count(), and take()

Learn best practices for using `cache()`, `count()`, and `take()` with a Spark DataFrame....

Cannot import timestamp_millis or unix_millis

Cannot use timestamp_millis or unix_millis directly with a DataFrame. You must first use selectExpr() or use SQL commands....

Cannot modify the value of an Apache Spark config

You cannot modify the value of a Spark config setting within a notebook. It must be set at the cluster level....

Convert nested JSON to a flattened DataFrame

How to convert a flattened DataFrame to nested JSON using a nested case class....

Create a DataFrame from a JSON string or Python dictionary

Create an Apache Spark DataFrame from a variable containing a JSON string or a Python dictionary....

Decimal$DecimalIsFractional assertion error

Using `round()` or casing a double to decimal results in a `Decimal$DecimalIsFractional` assertion error. java.lang.AssertionError assertion failed...

from_json returns null in Apache Spark 3.0

Spark 3.0 and above cannot parse JSON arrays as structs; from_json returns null....

Manage the size of Delta tables

Recommendations that can help you manage the size of your Delta tables....

Select files using a pattern match

Use a glob pattern match to select specific files in a folder....

Job fails with ExecutorLostFailure due to “Out of memory” error

Resolve executor failures where the root cause is due to the executor running out of memory.....

Job fails with ExecutorLostFailure because executor is busy

Resolve executor failures where the root cause is due to the executor being busy....

Understanding speculative execution

Learn how speculative execution works, how to identify it, and when you should use it....

Use custom classes and objects in a schema

You must define custom classes and objects inside a package if you want to use them in a notebook. ...

Jobs fails with a TimeoutException error

This error is usually caused by a Broadcast join that takes excessively long to complete....

Date functions only accept int values in Apache Spark 3.0

Date functions only accept int values in Apache Spark 3.0; fractional and string values return AnalysisException error....

Duplicate columns in the metadata error

Spark job fails while processing a Delta table with org.apache.spark.sql.AnalysisException Found duplicate column(s) in the metadata error....

Generate unique increasing numeric values

Use Apache Spark functions to generate unique and increasing numbers in a column in a table in a file or DataFrame....

Error in SQL statement: AnalysisException: Table or view not found

Learn how to resolve the AnalysisException SQL error "Table or view not found"....

Error when downloading full results after join

If you have duplicate columns after a join, you will get an error when trying to download the full results....

Error when running MSCK REPAIR TABLE in parallel

Do not run `MSCK REPAIR` commands in parallel. It results in a read timed out or out of memory error message....

Find the size of a table

How to find the size of a table....

Inner join drops records in result

Avoid dropped records when performing an inner join....

Data is incorrect when read from Snowflake

Data read from Snowflake is incorrect when time zone value is not set correctly....

JDBC write fails with a PrimaryKeyViolation error

JDBC write to a SQL database fails with a `PrimaryKeyViolation` error or results in duplicate data...

Query does not skip header row on external table

External Hive tables do not skip the header row when queried from Spark SQL....

SHOW DATABASES command returns unexpected column name

Running the `SHOW DATABASES` command returns an unexpected column name....

Cannot view table SerDe properties

SHOW CREATE TABLE only returns the Apache Spark DDL. It does not show the SerDe properties....

Parsing post meridiem time (PM) with to_timestamp() returns null

When converting 12-hour time to 24-hour time with to_timestamp() the hours variable must be lowercase....

to_json() results in Cannot use null as map key error

You must filter or replace null values in your input data before using to_json()....

Set nullability when using SaveAsTable with Delta tables

Learn how to create a Delta table with the nullability of columns set to false....

Ensure consistency in statistics functions between Spark 3.0 and Spark 3.1 and above

Statistics functions in Databricks Runtime 7.3 LTS and below return NaN when a divide by zero occurs. Set a Spark config to return null instead....

Using datetime values in Spark 3.0 and above

How to correctly use datetime functions in Spark SQL with Databricks runtime 7.3 LTS and above....

ANSI compliant DECIMAL precision and scale

Learn how to enable ANSI compliant error messages when incorrect values are used for DECIMAL precision and scale....

Recreate LISTAGG functionality with Spark SQL

Use collect_list and concat_ws in Spark SQL to achieve the same functionality as LISTAGG on other platforms....

Decreased performance when using DELETE with a subquery on Databricks Runtime 10.4 LTS

Auto optimize should be disabled when you have a DELETE with a subquery where one side is small enough to be broadcast....

SYNC command fails with a mismatched input error

Unity Catalog must be enabled on your cluster before you can use SYNC to migrate a legacy Hive table to Unity Catalog....

Unable to access Unity Catalog views

You cannot access a view if you do not have sufficient permissions on the underlying table....

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error

External tables cannot overlap with catalog/schema storage locations. They should be created in a subdirectory instead....

Cannot select a Databricks Runtime version when using a Delta Live Tables pipeline

Delta Live Tables do not allow you to directly configure the Databricks Runtime version....

Delta Live Tables job fails when using collect()

You should not use functions such as collect(), count(), toPandas(), save(), and saveAsTable() within the table and view function definitions....