![]() Each record in the DStream will be a tweet. Then we define our main DStream dataStream that will connect to the socket server we created before on port 9009 and read the tweets from that port. We defined a checkpoint here in order to allow periodic RDD checkpointing this is mandatory to be used in our app, as we’ll use stateful transformations (will be discussed later in the same section). Notice we have set the log level to ERROR in order to disable most of the logs that Spark writes. Let’s build up our Spark streaming app that will do real-time processing for the incoming tweets, extract the hashtags from them, and calculate how many hashtags have been mentioned.įirst, we have to create an instance of Spark Context sc, then we created the Streaming Context ssc from sc with a batch interval two seconds that will do the transformation on all streams received every two seconds. Setting Up Our Apache Spark Streaming Application S = socket.socket(socket.AF_INET, socket.SOCK_STREAM) Then we’ll call the get_tweets method, which we made above, for getting the tweets from Twitter and pass its response along with the socket connection to send_tweets_to_spark for sending the tweets to Spark. We’ll configure the IP here to be localhost as all will run on the same machine and the port 9009. Now, we’ll make the main part which will make the app host socket connections that spark will connect with. def send_tweets_to_spark(http_resp, tcp_connection): After that, it sends every tweet to Spark Streaming instance (will be discussed later) through a TCP connection. Then, create a function that takes the response from the above one and extracts the tweets’ text from the whole tweets’ JSON object. Response = requests.get(query_url, auth=my_auth, stream=True) Now, we will create a new function called get_tweets that will call the Twitter API URL and return the response for a stream of tweets. My_auth = requests_oauthlib.OAuth1(CONSUMER_KEY, CONSUMER_SECRET,ACCESS_TOKEN, ACCESS_SECRET) Import the libraries that we’ll use as below: import socketĪnd add the variables that will be used in OAuth for connecting to Twitter as below: # Replace the values below with yours It should be easy to follow for any professional Python developer.įirst, let’s create a file called twitter_app.py and then we’ll add the code in it together as below. In this step, I’ll show you how to build a simple client that will get the tweets from Twitter API using Python and passes them to the Spark Streaming instance. Your new access tokens will appear as below.Īnd now you’re ready for the next step. Then click on “Generate my access token.” Second, go to your newly created app and open the “Keys and Access Tokens” tab. In order to get tweets from Twitter, you need to register on TwitterApps by clicking on “Create new app” and then fill the below form click on “Create your Twitter app.” ![]() Creating Your Own Credentials for Twitter APIs In this article, I’ll teach you how to build a simple application that reads online streams from Twitter using Python, then processes the tweets using Apache Spark Streaming to identify hashtags and, finally, returns top trending hashtags and represents this data on a real-time dashboard. Allow me to demonstrate a real-life example: dealing, analyzing, and extracting insights from social network data in real time using one of the most important big data echo solutions out there-Apache Spark, and Python. One of the main sources of data today are social networks. ![]() ![]() Due to this staggering growth rate, big data platforms had to adopt radical solutions in order to maintain such huge volumes of data. Currently, around 90% of all data generated in our world was generated only in the last two years. Nowadays, data is growing and accumulating faster than ever before. ![]()
0 Comments
Leave a Reply. |