Apache Spark UI is not in sync with job

Status of Spark jobs gets out of sync with the Spark UI when events drop from the event queue before being processed....

Apache Spark job fails with Parquet column cannot be converted error

Parquet column cannot be converted error appears when you are reading decimal data in Parquet format and writing to a Delta table....

Best practice for cache(), count(), and take()

Learn best practices for using `cache()`, `count()`, and `take()` with a Spark DataFrame....

Cannot import timestamp_millis or unix_millis

Cannot use timestamp_millis or unix_millis directly with a DataFrame. You must first use selectExpr() or use SQL commands....

Cannot modify the value of an Apache Spark config

You cannot modify the value of a Spark config setting within a notebook. It must be set at the cluster level....

Convert flattened DataFrame to nested JSON

How to convert a flattened DataFrame to nested JSON using a nested case class....

Convert nested JSON to a flattened DataFrame

How to convert a flattened DataFrame to nested JSON using a nested case class....

Create a DataFrame from a JSON string or Python dictionary

Create an Apache Spark DataFrame from a variable containing a JSON string or a Python dictionary....

Decimal$DecimalIsFractional assertion error

Using `round()` or casing a double to decimal results in a `Decimal$DecimalIsFractional` assertion error. java.lang.AssertionError assertion failed...

from_json returns null in Apache Spark 3.0

Spark 3.0 and above cannot parse JSON arrays as structs; from_json returns null....

Intermittent NullPointerException when AQE is enabled

When adaptive query execution (AQE) is enabled, and cluster scales down and loses shuffle data, you can get a `NullPointerException` error....

Manage the size of Delta tables

Recommendations that can help you manage the size of your Delta tables....

Trouble reading external JDBC tables after upgrading from Databricks Runtime 5.5

Fail to read external JDBC tables after upgrading from Databricks Runtime 5.5 to 6.0 and above....

Running C++ code in Scala

Learn how to run C++ code in Scala with this example notebook....

Select files using a pattern match

Use a glob pattern match to select specific files in a folder....

Multiple Apache Spark JAR jobs fail when run concurrently

Apache Spark JAR jobs failing with an AnalysisException error when run concurrently....

Write a DataFrame with missing columns to a Redshift table

Use `FILLRECORD` to populate missing columns when writing a DataFrame to a Redshift table....

Job fails with ExecutorLostFailure due to “Out of memory” error

Resolve executor failures where the root cause is due to the executor running out of memory.....

Job fails with ExecutorLostFailure because executor is busy

Resolve executor failures where the root cause is due to the executor being busy....

Understanding speculative execution

Learn how speculative execution works, how to identify it, and when you should use it....

Use custom classes and objects in a schema

You must define custom classes and objects inside a package if you want to use them in a notebook. ...

Jobs fails with a TimeoutException error

This error is usually caused by a Broadcast join that takes excessively long to complete....