Problem:

When using the log delivery feature to ship job/cluster logs to an S3 bucket, the driver folder has a log4j log file but none of the executor folders have it.

Solution:

There is no issue with the log delivery feature. It is designed to generate a log4j log file only for the driver folder and not for the executor folders. This behavior is by design and does not indicate any problem with the feature or the cluster.

The log4j log file in the driver folder contains the logs specific to the driver node, while the executor folders contain logs specific to the executor nodes.

If you need to access the logs from the executor nodes, you can use other logging mechanisms such as Apache Spark's internal logging or custom loggers within your application code.