Databricks Knowledge Base

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Job execution (AWS)

These articles can help you tune and troubleshoot Apache Spark job execution.

3 Articles in this category

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.

  • Home
  • Amazon
  • Job execution (AWS)

Increase the number of tasks per stage

When using the spark-xml package, you can increase the number of tasks per stage by changing the configuration setting spark.hadoop.mapred.max.split.size to a lower value in the cluster’s Spark config (AWS | Azure ). This configuration setting controls the input block size. When data is read from DBFS, it is divided into input blocks, which are then...

Last updated: May 11th, 2022 by Adam Pavlacka

Maximum execution context or notebook attachment limit reached

Problem Notebook or job execution stops and returns either of the following errors: Run result unavailable: job failed with error message Context ExecutionContextId(1731742567765160237) is disconnected. Can’t attach this notebook because the cluster has reached the attached notebook limit. Detach a notebook and retry. Cause When you attach a noteboo...

Last updated: May 11th, 2022 by Adam Pavlacka

Serialized task is too large

If you see the follow error message, you may be able to fix this error by changing the Spark config (AWS | Azure ) when you start the cluster. Serialized task XXX:XXX was XXX bytes, which exceeds max allowed: spark.rpc.message.maxSize (XXX bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values. To change ...

Last updated: May 11th, 2022 by Adam Pavlacka


© Databricks 2022. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Policy | Terms of Use

Definition by Author

0
0