Cannot set a custom PYTHONPATH

Problem When you are trying to set a custom PYTHONPATH environment variable in a cluster-scoped init script, but the values are overridden at driver startup.  Cause Setting a custom PYTHONPATH in an init scripts does not work and is not supported. Additionally, you cannot set a custom PYTHONPATH when using Databricks Container Services. Solution You...

0 min reading time

Reading .xlsx files with xlrd fails

Problem You are have xlrd installed on your cluster and are attempting to read files in the Excel .xlsx format when you get an error. XLRDError: Excel xlsx file; not supported Cause xlrd 2.0.0 and above can only read .xls files. Support for .xlsx files was removed from xlrd due to a potential security vulnerability. Solution Use openpyxl to open .xl...

0 min reading time