Apache Spark job fails with Failed to parse byte string

Problem Spark-submit jobs fail with a Failed to parse byte string: -1 error message. java.util.concurrent.ExecutionException: java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Failed to parse byte string: -1 at java.util.concurre...

0 min reading time

Apache Spark job fails with a Connection pool shut down error

Problem A Spark job fails with the error message java.lang.IllegalStateException: Connection pool shut down when attempting to write data into a Delta table on S3. Cause Spark jobs writing to S3 are limited to a maximum number of simultaneous connections. The java.lang.IllegalStateException: Connection pool shut down occurs when this connection pool...

0 min reading time

AttributeError: ‘function’ object has no attribute

Problem You are selecting columns from a DataFrame and you get an error message. ERROR: AttributeError: 'function' object has no attribute '_get_object_id' in job Cause The DataFrame API contains a small number of protected keywords. If a column in your DataFrame uses a protected keyword as the column name, you will get an error message. For example...

0 min reading time