ApacheSpark-Kafka-Python-Streaming

This project aims to consume tweets message from a python script through kafka broker and transform the data to get maximum tagged word through ApacheSpark Streaming.

Flow: Python Script --> Kafka --> ApacheSpark

Deployment

To deploy this project run the following in order:

1. zookeeper

  zookeeper-server-start.bat config\zookeeper.properties 

2. Kafka

  kafka-server-start.bat config/server.properties 

To check for messages in kafka (my topic's name is 'awesome'):

kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic awesome

3. Data Streaming from Twitter Python Script

  Python Twitter_to_Kafka.py

4. Spark Script

  spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.7 Spark_kafka_twitter_tags.py

Authors

• @AminSiddique

🚀 About Me

Hi, I'm Amin! 👋

I am a passionate Data Engineer