AttributeError: ‘function’ object has no attribute

Using protected keywords from the DataFrame API as column names results in a function object has no attribute error message....

Convert Python datetime object to string

Display date and time values in a column, as a datetime object, and as a string....

Display file and directory timestamp details

Display file creation date and modification date using Python....

Reading large DBFS-mounted files using Python APIs

Learn how to resolve errors when reading large DBFS-mounted files using Python APIs....

How to import a custom CA certificate

Learn how to import a custom CA certificate into your Databricks cluster for Python use....

Job remains idle before starting

Apache Spark jobs remain idle for a long time before starting....

List all workspace objects

List all Databricks workspace objects under a given path....

Python commands fail on high concurrency clusters

Python commands fail on high concurrency clusters with Apache Spark process isolation and shared session enabled. WARN error message....

Cluster cancels Python command execution after installing Bokeh

Learn what to do when your Databricks cluster cancels Python command execution after you install Bokeh....

Cluster cancels Python command execution due to library conflict

Learn what to do when your Databricks cluster cancels Python command execution due to a library conflict....

Python command execution fails with AttributeError

Learn what to do when a Python command in your Databricks notebook fails with AttributeError....

Job fails with Java IndexOutOfBoundsException error

When groupby() is used along with applyInPandas it generates an exception due to an arrow buffer limitation....

Job fails with NoSuchElementException error

NoSuchElementException errors can occur when using Apache Arrow....

Job fails with IndexOutOfBoundsException and ArrowBuf errors

When Groupby is used with applyinPandas it can result in Apache Arrow buffer size estimation errors....

Field name sorting changes in Apache Spark 3.x

Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically....

Job fails with "not enough memory to build the hash map" error

You should use adaptive query execution instead of explicit broadcast hints to perform joins on Databricks Runtime 11.3 LTS and above....

Create a DataFrame from a JSON string or Python dictionary

Create an Apache Spark DataFrame from a variable containing a JSON string or a Python dictionary....

Behavioral changes for the CHAR data type on Serverless

Pad your reads with spaces to match the declared length of the CHAR field or set the legacy charVarcharAsString config to true....

Column value errors when connecting from Apache Spark to Databricks using Spark JDBC

Use overriding quote identifiers in the JdbcDialect class and register them under JDBCDialects in Java or Python....

Trying to decode a protocol buffer and getting error [PROTOBUF_DEPENDENCY_NOT_FOUND]

Use the option --include_imports while creating the protobuf descriptor file, and then use this descriptor file in the from_protobuf() function....

Error java.io.FileNotFoundException when job attempts to read or write intermediary files

Cache the DataFrame before performing write operations and ensure you are using a compatible version of the com.crealytics.spark.excel library. ...

Using Pyspark testing library assertDataFrameEqual throws OutOfMemoryError

Verify schemas are equivalent, then either ensure sufficient driver memory or compare a subset of DataFrames. ...

Expensive transformation on DataFrame is recalculated even when cached

Understand how Apache Spark DataFrame caching works....

Error java.lang.UnsupportedOperationException when trying to read datetime data files

Set spark.sql.legacy.parquet.datetimeRebaseModeInRead to LEGACY. ...

Apache Spark PySpark job using a Python threading API function taking hours instead of minutes

Use the Databricks Spark connector and ensure your cluster configuration is optimized for the workload....

PySparkValueError when working with UDFs in Apache Spark

Ensure that the Python UDF output matches the schema defined in the source code. ...

Unable to parallelize the code using the apply API from Pandas on PySpark

Directly use the apply function from pyspark.pandas without wrapping it in a lambda function....

Runtimes increase when using .loc() and assignment(=) operations

Use vectorized operations instead....

Unable to get Apache Spark SparkEnv settings via PySpark

To get the same output using PySpark, broadcast the “test” value to the executors so you can perform the map operation on the executors....

RESOURCES_EXHAUSTED error message when trying to perform self-joins with Spark Connect

Increase the max message size using the spark.sql.session.localRelationCacheThreshold config or use temporary views. ...

DataFrame in an interactive cluster still showing cached data after calling unpersist() function

Use unpersist(blocking=True) to ensure unpersist() is performed before proceeding with further actions....

Apache Spark Submit job clusters do not terminate after sc.stop()

Explicitly invoke System.exit(0) after SparkContext.stop()....

Error PySparkNotImplementedError when using an RDD to extract distinct values on a standard cluster

Use .collect() and list comprehension to extract distinct column values....

Use snappy and zstd compression types in a Delta table without rewriting entire table

Test your compression type, generate, and insert sample records using zstd, then write the zstd files to your Delta table....

Using collect_list after transformations such as JOIN returns inconsistent counts even though the underlying data doesn’t change

Sort the collect_list output using array_sort before performing joins....