Date functions only accept int values in Apache Spark 3.0

Date functions only accept int values in Apache Spark 3.0; fractional and string values return AnalysisException error....

Duplicate columns in the metadata error

Spark job fails while processing a Delta table with org.apache.spark.sql.AnalysisException Found duplicate column(s) in the metadata error....

Generate unique increasing numeric values

Use Apache Spark functions to generate unique and increasing numbers in a column in a table in a file or DataFrame....

Error in SQL statement: AnalysisException: Table or view not found

Learn how to resolve the AnalysisException SQL error "Table or view not found"....

Error when downloading full results after join

If you have duplicate columns after a join, you will get an error when trying to download the full results....

Error when running MSCK REPAIR TABLE in parallel

Do not run `MSCK REPAIR` commands in parallel. It results in a read timed out or out of memory error message....

Find the size of a table

How to find the size of a table....

Inner join drops records in result

Avoid dropped records when performing an inner join....

Data is incorrect when read from Snowflake

Data read from Snowflake is incorrect when time zone value is not set correctly....

JDBC write fails with a PrimaryKeyViolation error

JDBC write to a SQL database fails with a `PrimaryKeyViolation` error or results in duplicate data...

Query does not skip header row on external table

External Hive tables do not skip the header row when queried from Spark SQL....

SHOW DATABASES command returns unexpected column name

Running the `SHOW DATABASES` command returns an unexpected column name....

Cannot view table SerDe properties

SHOW CREATE TABLE only returns the Apache Spark DDL. It does not show the SerDe properties....

Parsing post meridiem time (PM) with to_timestamp() returns null

When converting 12-hour time to 24-hour time with to_timestamp() the hours variable must be lowercase....

to_json() results in Cannot use null as map key error

You must filter or replace null values in your input data before using to_json()....

Set nullability when using SaveAsTable with Delta tables

Learn how to create a Delta table with the nullability of columns set to false....

Ensure consistency in statistics functions between Spark 3.0 and Spark 3.1 and above

Statistics functions in Databricks Runtime 7.3 LTS and below return NaN when a divide by zero occurs. Set a Spark config to return null instead....

Using datetime values in Spark 3.0 and above

How to correctly use datetime functions in Spark SQL with Databricks runtime 7.3 LTS and above....

ANSI compliant DECIMAL precision and scale

Learn how to enable ANSI compliant error messages when incorrect values are used for DECIMAL precision and scale....

Recreate LISTAGG functionality with Spark SQL

Use collect_list and concat_ws in Spark SQL to achieve the same functionality as LISTAGG on other platforms....

Decreased performance when using DELETE with a subquery on Databricks Runtime 10.4 LTS

Auto optimize should be disabled when you have a DELETE with a subquery where one side is small enough to be broadcast....