Rendering an R markdown file containing sparklyr
code fails
Problem
After you install and configure RStudio in the Databricks environment, when you launch RStudio and click the Knit button to knit a Markdown file that contains code to initialize a sparklyr
context, rendering fails with the following error:
failed to start sparklyr backend:object 'DATABRICKS_GUID' not found Calls: <Anonymous>… tryCatch -> tryCatchList-> tryCatchOne -> <Anonymous> Execution halted
Cause
If you try to initialize a sparklyr
context in a Markdown notebook with code similar to the following, the Markdown page fails to render because the knitr
process spawns a new namespace that is missing the 'DATABRICKS_GUID'
global variable.
case library(sparklyr)
sc <- spark_connect(method = "databricks")