Rendering an R markdown file containing sparklyr code fails

Problem

After you install and configure RStudio in the Databricks environment, when you launch RStudio and click the Knit button to knit a Markdown file that contains code to initialize a sparklyr context, rendering fails with the following error:

failed to start sparklyr backend:object 'DATABRICKS_GUID' not found Calls: <Anonymous>… tryCatch -> tryCatchList-> tryCatchOne -> <Anonymous> Execution halted

Cause

If you try to initialize a sparklyr context in a Markdown notebook with code similar to the following, the Markdown page fails to render because the knitr process spawns a new namespace that is missing the 'DATABRICKS_GUID' global variable.

case library(sparklyr)
sc <- spark_connect(method = "databricks")

Solution

Instead of rendering the Markdown page by clicking the Knit button in the R Markdown console, use the following script:

rmarkdown::render("your_doc.Rmd")

Render the Markdown file using the R console, and then you can access the file in the RStudio Files tab.