Problem
You have imported Python libraries, but when you try to execute Python code in a notebook you get a repeating message as output.
INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0 INFO:py4j.java_gateway:Received command c on object id p0
Cause
The default log level for py4j.java_gateway is ERROR.
If any of the imported Python libraries set the log level to INFO you will see this message.
Solution
You can prevent the output of the INFO messages by setting the log level back to ERROR after importing the libraries.
%python import logging logger = spark._jvm.org.apache.log4j logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)