Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI.

Written by vivian.wilfred

Last published at: March 17th, 2023

Problem

Users of Shared access mode clusters experience intermittent HTTP 500 errors when trying to view task logs in the Apache Spark UI. This also applies to admins.

ErrorCaused by:java.lang.Exception: Log viewing is disabled on this cluster
    at org.apache.spark.deploy.worker.ui.LogPage.render(LogPage.scala:65)
    at org.apache.spark.ui.WebUI$$anonfun$3.apply(WebUI.scala:100)
    at org.apache.spark.ui.WebUI$$anonfun$3.apply(WebUI.scala:100)
    at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:534)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)


Cause

This specific exception is controlled by the spark.databricks.ui.logViewingEnabled Spark property. When this value is set to false, log viewing is disabled. When Spark log viewing is disabled on the cluster, the Spark UI generates an error when you attempt to view the logs.

The spark.databricks.ui.logViewingEnabled property defaults to true, however sometimes other Spark configurations (such as spark.databricks.acl.dfAclsEnabled) can alter its value and set it to false.

Solution

Set spark.databricks.ui.logViewingEnabled to true in the cluster's Spark config (AWS | Azure | GCP).

spark.databricks.ui.logViewingEnabled true

This restores the default configuration in case it is accidentally overwritten.