Date functions only accept int values in Apache Spark 3.0

Date functions only accept int values in Apache Spark 3.0; fractional and string values return AnalysisException error....

Duplicate columns in the metadata error

Spark job fails while processing a Delta table with org.apache.spark.sql.AnalysisException Found duplicate column(s) in the metadata error....

Generate unique increasing numeric values

Use Apache Spark functions to generate unique and increasing numbers in a column in a table in a file or DataFrame....

Error in SQL statement: AnalysisException: Table or view not found

Learn how to resolve the AnalysisException SQL error "Table or view not found"....

Error when downloading full results after join

If you have duplicate columns after a join, you will get an error when trying to download the full results....

Error when running MSCK REPAIR TABLE in parallel

Do not run `MSCK REPAIR` commands in parallel. It results in a read timed out or out of memory error message....

Find the size of a table snapshot

How to find the size of a table....

Inner join drops records in result

Avoid dropped records when performing an inner join....

Data is incorrect when read from Snowflake

Data read from Snowflake is incorrect when time zone value is not set correctly....

JDBC write fails with a PrimaryKeyViolation error

JDBC write to a SQL database fails with a `PrimaryKeyViolation` error or results in duplicate data...

Query does not skip header row on external table

External Hive tables do not skip the header row when queried from Spark SQL....

SHOW DATABASES command returns unexpected column name

Running the `SHOW DATABASES` command returns an unexpected column name....

Cannot view table SerDe properties

SHOW CREATE TABLE only returns the Apache Spark DDL. It does not show the SerDe properties....

Parsing post meridiem time (PM) with to_timestamp() returns null

When converting 12-hour time to 24-hour time with to_timestamp() the hours variable must be lowercase....

to_json() results in Cannot use null as map key error

You must filter or replace null values in your input data before using to_json()....

Set nullability when using SaveAsTable with Delta tables

Learn how to create a Delta table with the nullability of columns set to false....

Ensure consistency in statistics functions between Spark 3.0 and Spark 3.1 and above

Statistics functions in Databricks Runtime 7.3 LTS and below return NaN when a divide by zero occurs. Set a Spark config to return null instead....

Using datetime values in Spark 3.0 and above

How to correctly use datetime functions in Spark SQL with Databricks runtime 7.3 LTS and above....

ANSI compliant DECIMAL precision and scale

Learn how to enable ANSI compliant error messages when incorrect values are used for DECIMAL precision and scale....

Recreate LISTAGG functionality with Spark SQL

Use collect_list and concat_ws in Spark SQL to achieve the same functionality as LISTAGG on other platforms....

Decreased performance when using DELETE with a subquery on Databricks Runtime 10.4 LTS

Auto optimize should be disabled when you have a DELETE with a subquery where one side is small enough to be broadcast....

Automatic VACUUM on write does not work with non-Delta tables

Manually run VACUUM to clear uncommitted files from the entire table....

LEFT JOIN resulting in null values when joining timestamp column and date column

Cast the value of the timestamp column to date datatype when joining it with a column of 'date' datatype....

Handling case sensitivity issues in Delta Lake nested fields

Set a specific property in your Spark configuration to handle the case sensitivity of nested fields in Delta tables. ...

Trailing zeros in decimal values appear when reading Parquet files in Apache Spark

Use the format_number function to format decimal values without altering data precision....

SQL transformations involving timestamp columns giving different results in an interactive cluster versus serverless compute

Use SQL’s type casting to handle precision or upgrade your JDK version. ...

Job failures when running Apache Spark jobs processing MongoDB data

Validate your source data to make sure data types match, or disable ANSI compliance in Spark SQL....

Job ID column not consistently showing values in the Apache Spark UI for Sub Execution IDs

Click on a Sub Execution ID column and then click again to sort and see all the IDs together....

COPY INTO command failing on partition columns with STRING data types that start with an integer

Disable partition column type inference. ...

NO SUCH CATALOG EXCEPTION error when trying to create row filters

Specify the function location when you use the ALTER TABLE command to apply a row filter on a table. ...

Time zones converted from a local zone to UTC and back not reverting to original values in Apache Spark and SQL Warehouse

Set spark.sql.datetime.java8API.enabled to true on the cluster....

Regular expression (regex) not filtering as expected when using [:alnum:] and [:digit:] in the SQL query

Use \p{Alnum} or \p{Digit} instead....

Trying to perform WRITE over UNION ALL causes error

Use Databricks Runtime 15.4 LTS or above....

Different results when using rlike with regex in SQL queries vs Spark SQL queries

You must properly escape the backslash character in rlike patterns....

[FIELDS_ALREADY_EXISTS] error in spark.sql when changing column name capitalization

Spark.sql is not case sensitive by default. Set spark.sql.caseSensitive to true to change the default behavior....

Round() function not returning the number of decimal places indicated in the parameters

Use string formatting in Python or cast the result to a DECIMAL type in SQL....

DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION error when querying views of system tables

Implement schema evolution for views in Databricks Runtime 15.4 LTS or above to accommodate underlying table changes....

INSERT OVERWRITE DIRECTORY with Hive format failing with “specified path already exists” error

Add USING PARQUET after the INSERT OVERWRITE DIRECTORY in your query....