spark-streaming

pyspark streaming example based on article here https://www.neerajbyte.com/post/spark-streaming-from-text-files-using-pyspark-api/

usage [2 terminals will be needed]

git clone https://github.com/InterruptSpeed/spark-streaming
cd spark-streaming
mkdir log

/home/spark/spark-2.4.0-bin-hadoop2.7/bin/spark-submit streaming.py
python3 file.py