Apache Spark UI is not in sync with job

Status of Spark jobs gets out of sync with the Spark UI when events drop from the event queue before being processed....

Apache Spark job fails with Parquet column cannot be converted error

Parquet column cannot be converted error appears when you are reading decimal data in Parquet format and writing to a Delta table....

Cannot import timestamp_millis or unix_millis

Cannot use timestamp_millis or unix_millis directly with a DataFrame. You must first use selectExpr() or use SQL commands....

Cannot modify the value of an Apache Spark config

You cannot modify the value of a Spark config setting within a notebook. It must be set at the cluster level....

Convert nested JSON to a flattened DataFrame

How to convert a flattened DataFrame to nested JSON using a nested case class....

Decimal$DecimalIsFractional assertion error

Using `round()` or casing a double to decimal results in a `Decimal$DecimalIsFractional` assertion error. java.lang.AssertionError assertion failed...

from_json returns null in Apache Spark 3.0

Spark 3.0 and above cannot parse JSON arrays as structs; from_json returns null....

Manage the size of Delta tables

Recommendations that can help you manage the size of your Delta tables....

Select files using a pattern match

Use a glob pattern match to select specific files in a folder....

Job fails with ExecutorLostFailure due to “Out of memory” error

Resolve executor failures where the root cause is due to the executor running out of memory.....

Job fails with ExecutorLostFailure because executor is busy

Resolve executor failures where the root cause is due to the executor being busy....

Understanding speculative execution

Learn how speculative execution works, how to identify it, and when you should use it....

Use custom classes and objects in a schema

You must define custom classes and objects inside a package if you want to use them in a notebook. ...

Jobs fails with a TimeoutException error

This error is usually caused by a Broadcast join that takes excessively long to complete....

Sort failed after writing partitioned data to parquet using PySpark on Databricks Runtime 13.3 LTS

Set the Apache Spark configuration to set the sorted data after writing partitioned data to parquet....

Reading Avro files with Structured Streaming using wildcards in the path fails with error ArrayIndexOutOfBoundsException

Add an option to enable recursively reading bulk Avro files using a wildcard path....

WithColumn operation when using in-loop slows performance

Use the select operator instead....

Extract timestamps with precision up to nano seconds from a long column

Create a UDF to extract the nanoseconds from the LongType....

Apache Spark job failing with GC overhead limit exceeded error

Analyze JOIN columns and deduplicate JOIN keys....