Problem

PyPMML is a Python PMML scoring library. You use the following code to install PyPMML on your cluster. 

%python

from pypmml import Model
modelb = Model.fromFile('/dbfs/<username>/DecisionTreeIris.pmml')

Your notebook or workflow subsequently fails with the following error.

Error : Py4JError: Could not find py4j jar at

Cause

The Py4J library that comes preinstalled with Databricks Runtime is installed in a different location than the Py4J package you are installing. As a result, when PyPMML attempts to invoke Py4J from the standard path, it fails.

Solution

Install your Py4J library into the expected location. 

1. Check the Py4J version your cluster uses. Run the following code to identify the current Py4J version. 

import py4j
print(py4j.__version__)

2. Install the PyPMML library along with the appropriate version of Py4J based on your Databricks Runtime version (obtained from step 1). Then restart the Python environment.

!pip install pypmml
!pip install py4j==<version-based-on-your-Databricks-Runtime>
dbutils.library.restartPython()

3. You are expected to encounter an error, ValueError: invalid literal for int() with base 10: b'[Global flags]\n'. Run the following code to avoid the error. Refer to the Getting ValueError when trying to import PMML files using PyPMML KB article for detailed information.

import os
tmpval = os.environ.get("JAVA_OPTS", "")
tmpval = tmpval.replace("-XX:+PrintFlagsFinal", "")
tmpval = tmpval.replace("-verbose:gc", "")
os.environ["JAVA_OPTS"] = tmpval

4. Once the environment is set up, proceed to load your PMML model.