Problem

You have imported Python libraries, but when you try to execute Python code in a notebook you get a repeating message as output.

INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0
INFO:py4j.java_gateway:Received command c on object id p0

Cause

The default log level for py4j.java_gateway is ERROR.

If any of the imported Python libraries set the log level to INFO you will see this message.

Solution

You can prevent the output of the INFO messages by setting the log level back to ERROR after importing the libraries.

%python

import logging
logger = spark._jvm.org.apache.log4j
logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)