AttributeError: ‘function’ object has no attribute

Using protected keywords from the DataFrame API as column names results in a function object has no attribute error message....

Convert Python datetime object to string

Display date and time values in a column, as a datetime object, and as a string....

Display file and directory timestamp details

Display file creation date and modification date using Python....

Install and compile Cython

Learn how to install and compile Cython with Databricks....

Reading large DBFS-mounted files using Python APIs

Learn how to resolve errors when reading large DBFS-mounted files using Python APIs....

Use the HDFS API to read files in Python

Learn how to read files directly by using the HDFS API in Python....

How to import a custom CA certificate

Learn how to import a custom CA certificate into your Databricks cluster for Python use....

Job remains idle before starting

Apache Spark jobs remain idle for a long time before starting....

List all workspace objects

List all Databricks workspace objects under a given path....

Load special characters with Spark-XML

Special characters are not rendering correctly. Use charset with Spark-XML....

Python commands fail on high concurrency clusters

Python commands fail on high concurrency clusters with Apache Spark process isolation and shared session enabled. WARN error message....

Cluster cancels Python command execution after installing Bokeh

Learn what to do when your Databricks cluster cancels Python command execution after you install Bokeh....

Cluster cancels Python command execution due to library conflict

Learn what to do when your Databricks cluster cancels Python command execution due to a library conflict....

Python command execution fails with AttributeError

Learn what to do when a Python command in your Databricks notebook fails with AttributeError....

Python REPL fails to start in Docker

Learn how to fix a Python virtualenv error that prevents REPL from starting in a Docker container...

How to run SQL queries from Python scripts

Learn how to run SQL queries using Python scripts....

Run C++ code in Python

Learn how to run C++ code in Python....

Python 2 sunset status

Learn about the sunset status of Python 2 in Databricks....

Job fails with Java IndexOutOfBoundsException error

When groupby() is used along with applyInPandas it generates an exception due to an arrow buffer limitation....

Job fails with NoSuchElementException error

NoSuchElementException errors can occur when using Apache Arrow....

Job fails with IndexOutOfBoundsException and ArrowBuf errors

When Groupby is used with applyinPandas it can result in Apache Arrow buffer size estimation errors....

Field name sorting changes in Apache Spark 3.x

Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically....

Job fails with "not enough memory to build the hash map" error

You should use adaptive query execution instead of explicit broadcast hints to perform joins on Databricks Runtime 11.3 LTS and above....

Create a DataFrame from a JSON string or Python dictionary

Create an Apache Spark DataFrame from a variable containing a JSON string or a Python dictionary....

Apache Spark driver stops and restarts while reading JSON file data in an S3 bucket

Use a predefined schema to read the files, or read folder contents as text instead. ...

Behavioral changes for the CHAR data type on Serverless

Pad your reads with spaces to match the declared length of the CHAR field or set the legacy charVarcharAsString config to true....

Column value errors when connecting from Apache Spark to Databricks using Spark JDBC

Use overriding quote identifiers in the JdbcDialect class and register them under JDBCDialects in Java or Python....

Trying to decode a protocol buffer and getting error [PROTOBUF_DEPENDENCY_NOT_FOUND]

Use the option --include_imports while creating the protobuf descriptor file, and then use this descriptor file in the from_protobuf() function....

Error java.io.FileNotFoundException when job attempts to read or write intermediary files

Cache the DataFrame before performing write operations and ensure you are using a compatible version of the com.crealytics.spark.excel library. ...

Using Pyspark testing library assertDataFrameEqual throws OutOfMemoryError

Verify schemas are equivalent, then either ensure sufficient driver memory or compare a subset of DataFrames. ...

Expensive transformation on DataFrame is recalculated even when cached

Understand how Apache Spark DataFrame caching works....

Error java.lang.UnsupportedOperationException when trying to read datetime data files

Set spark.sql.legacy.parquet.datetimeRebaseModeInRead to LEGACY. ...

Apache Spark PySpark job using a Python threading API function taking hours instead of minutes

Use the Databricks Spark connector and ensure your cluster configuration is optimized for the workload....

PySparkValueError when working with UDFs in Apache Spark

Ensure that the Python UDF output matches the schema defined in the source code. ...

Unable to parallelize the code using the apply API from Pandas on PySpark

Directly use the apply function from pyspark.pandas without wrapping it in a lambda function....

Runtimes increase when using .loc() and assignment(=) operations

Use vectorized operations instead....

Unable to get Apache Spark SparkEnv settings via PySpark

To get the same output using PySpark, broadcast the “test” value to the executors so you can perform the map operation on the executors....

RESOURCES_EXHAUSTED error message when trying to perform self-joins with Spark Connect

Increase the max message size using the spark.sql.session.localRelationCacheThreshold config or use temporary views. ...

DataFrame in an interactive cluster still showing cached data after calling unpersist() function

Use unpersist(blocking=True) to ensure unpersist() is performed before proceeding with further actions....

Apache Spark Submit job clusters do not terminate after sc.stop()

Explicitly invoke System.exit(0) after SparkContext.stop()....

Error PySparkNotImplementedError when using an RDD to extract distinct values on a standard cluster

Use .collect() and list comprehension to extract distinct column values....

Use snappy and zstd compression types in a Delta table without rewriting entire table

Test your compression type, generate, and insert sample records using zstd, then write the zstd files to your Delta table....

Using collect_list after transformations such as JOIN returns inconsistent counts even though the underlying data doesn’t change

Sort the collect_list output using array_sort before performing joins....

Getting a base64 error when executing a UDF on a serverless cluster

Replace UNBASE64 with TRY_TO_BINARY(uid, 'BASE64') in the UDF....

Getting Error: [NOT_IMPLEMENTED] toJSON() is not implemented when trying to convert a DataFrame to a JSON string

...

Getting “Cannot Operate on a Handle That Is Closed” error when running an Apache Spark job

Use multiple streams or switch to supported Scala thread pools....

Job execution returning [UDF_MAX_COUNT_EXCEEDED] error

Increase the UDF limit and adjust memory tracking settings....

Getting a ConcurrentModificationException in PySpark CrossValidator on Databricks Runtime 15.4 LTS

Enable spark.databricks.property.standardClone or upgrade your Databricks Runtime version....

Getting AssertionError when using the withColumns() function in Databricks Runtime 14.3 LTS or above

Use the col() wrapper when using the withColumns() function. ...

How to retrieve Parquet file metadata

...

Table writes failing when trying to read from a Delta table

Upgrade to Databricks Runtime 13.3 LTS or above....