twitter_api_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Note\n",
    "This jupyter notebook is for learning purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First install the neccessary python packages from requirements.txt\n",
    "\n",
    "`pip install -r requirements.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this document I use the searchtweets python wrapper. I recommend this option, however you can also make HTTP requests to the twitter api directly\n",
    "\n",
    "[searchtweets documentation](https://github.com/twitterdev/search-tweets-python/tree/v2)\n",
    "\n",
    "[twitter api HTTP requests sample code](https://github.com/twitterdev/Twitter-API-v2-sample-code)\n",
    "\n",
    "Refer to the [Twitter api V2 Docs](https://developer.twitter.com/en/docs/twitter-api)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import os\n",
    "import json\n",
    "from searchtweets import ResultStream, gen_request_parameters, load_credentials, collect_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_archive_seach_args = load_credentials(filename=\"./twitter_keys.yaml\",\n",
    "                 yaml_key=\"search_tweets_fullarchive_dev\",\n",
    "                 env_overwrite=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Full Archive Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its a good idea to set the MAX_TWEETS per call as a constant. The max is 500, but if you're testing its better to keep this limit smaller. Our limit is 10 million tweets a month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_TWEETS = 10 # 500 is the max per call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See this [link](https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet) for documentation on the tweet fields\n",
    "\n",
    "The tweet object has many fields. You have to list the ones you want to retrieve as a comma delimited list.\n",
    "`tweet_fields='id,created_at,text,public_metrics'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See this [link](https://developer.twitter.com/en/docs/twitter-api/expansions) for documentation on expansions\n",
    "\n",
    "expansions are also a comma delimited list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"query\": \"from:aei\", \"start_time\": \"2020-01-01T00:00:00Z\", \"end_time\": \"2021-01-30T00:00:00Z\", \"max_results\": 10, \"tweet.fields\": \"id,author_id,created_at,text,public_metrics,referenced_tweets,entities\", \"expansions\": \"author_id\"}\n"
     ]
    }
   ],
   "source": [
    "rule = gen_request_parameters(\"from:aei\",\n",
    "                            granularity=None,\n",
    "                          start_time=\"2020-01-01\", #UTC 2020-01-01 00:00\n",
    "                          end_time=\"2021-01-30\",#UTC 2020-01-30 00:00\n",
    "                          tweet_fields='id,author_id,created_at,text,public_metrics,referenced_tweets,entities',\n",
    "                          expansions=\"author_id\",\n",
    "                        results_per_call=MAX_TWEETS)\n",
    "print(rule)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets = collect_results(rule, max_tweets=MAX_TWEETS, result_stream_args=full_archive_seach_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "1\n",
      "dict_keys(['data', 'includes', 'meta'])\n"
     ]
    }
   ],
   "source": [
    "print(type(tweets))\n",
    "print(len(tweets))\n",
    "print(tweets[0].keys())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`collect_results` returns a list of dicts. Each dict represents a page of results. Since we set `results_per_call` and `max_tweets` both as `MAX_TWEETS` there will only be one element in that list. See the `Crawling all tweets made by a user` section below for more information\n",
    "\n",
    "that dict has `data`, `meta`, `includes` fields\n",
    "\n",
    "`data` has all the tweet data so most of the stuff you need is there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "{'author_id': '30864583', 'id': '1355254021024526337', 'referenced_tweets': [{'type': 'retweeted', 'id': '1355239914858835969'}], 'text': 'RT @AEIecon: There is an opportunity for bipartisan cooperation on expanding coverage and controlling costs — but only if the parties set a…', 'public_metrics': {'retweet_count': 2, 'reply_count': 0, 'like_count': 0, 'quote_count': 0}, 'created_at': '2021-01-29T20:38:32.000Z', 'entities': {'mentions': [{'start': 3, 'end': 11, 'username': 'AEIecon', 'id': '809552311'}]}}\n"
     ]
    }
   ],
   "source": [
    "print(len(tweets[0]['data'])) # max len for this is same as MAX_TWEETS\n",
    "print(tweets[0]['data'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`meta` has the first and last ids that it fetched so useful for constructing the next query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'newest_id': '1355254021024526337', 'oldest_id': '1354856778647941121', 'result_count': 10, 'next_token': 'b26v89c19zqg8o3foskt5kx7wqa14ffp6hpnqd8sxqc8t'}\n"
     ]
    }
   ],
   "source": [
    "print(tweets[0]['meta'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`includes` has all the expansion data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'users': [{'id': '30864583', 'name': 'AEI', 'username': 'AEI'}]}\n"
     ]
    }
   ],
   "source": [
    "print(tweets[0]['includes'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To construct a url for a tweet all you need is the tweet id and the author's username\n",
    "\n",
    "the tweet id is the `id` field found in `data`\n",
    "the username is the `username` field found in `includes`\n",
    "\n",
    "`https://twitter.com/[username]/status/[tweet id]`\n",
    "\n",
    "so for the above example it would be\n",
    "\n",
    "https://twitter.com/AEI/status/1355254021024526337"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crawling all tweets made by a user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a query has more that 500 tweet results, there will be multiple pages of results. To get all the results we have to get the results from all pages.\n",
    "\n",
    "See the [Pagination documentation](https://developer.twitter.com/en/docs/twitter-api/pagination)\n",
    "\n",
    "Essentially we use the `next_token` field from `meta` to get the next page of results. We continue this process until the next token is empyt (i.e no more result pages)\n",
    "\n",
    "**However with searchtweets,** pagination is handle automatically with `collect_results`. All you have to do is set `max_tweets` to a large number. Our monthly limit is 10 million so experiment to find a suitable max.\n",
    "\n",
    "It will return a list of page results. Each element in that list represent a call. Each eall will have `results_per_call` tweets (max is 500). you set this in `gen_request_parameters`\n",
    "\n",
    "see Fast Way in [searchtweets documentation](https://github.com/twitterdev/search-tweets-python/tree/v2#fast-way)\n",
    "\n",
    "Below is an example on how to get all tweets by AEI in 2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"query\": \"from:aei\", \"start_time\": \"2021-01-01T00:00:00Z\", \"end_time\": \"2022-01-01T00:00:00Z\", \"max_results\": 500, \"tweet.fields\": \"id,author_id,created_at,text,public_metrics,referenced_tweets,entities\"}\n"
     ]
    }
   ],
   "source": [
    "MAX_TWEETS_PER_CALL = 500 # set this to the max tweets we can get per call (500)\n",
    "ruleFullCrawl = gen_request_parameters(\"from:aei\",\n",
    "                            granularity=None,\n",
    "                          start_time=\"2021-01-01\", #UTC 2021-01-01 00:00\n",
    "                          end_time=\"2022-01-01\",#UTC 2021-12-31 00:00\n",
    "                          tweet_fields='id,author_id,created_at,text,public_metrics,referenced_tweets,entities',\n",
    "                        results_per_call=MAX_TWEETS_PER_CALL)\n",
    "print(ruleFullCrawl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_TWEETS_PER_SEARCH=100000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will return a list where each element in that list will contain `MAX_TWEETS_PER_CALL` tweets. The total number of tweets will not exeed `MAX_TWEETS_PER_SEARCH`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " HTTP Error code: 429: {\"title\":\"Too Many Requests\",\"detail\":\"Too Many Requests\",\"type\":\"about:blank\",\"status\":429} | Too Many Requests\n",
      " Request payload: {'query': 'from:aei', 'start_time': '2021-01-01T00:00:00Z', 'end_time': '2022-01-01T00:00:00Z', 'max_results': 500, 'tweet.fields': 'id,author_id,created_at,text,public_metrics,referenced_tweets,entities'}\n",
      "Rate limit hit... Will retry...\n",
      "Will retry in 4 seconds...\n"
     ]
    }
   ],
   "source": [
    "tweets = collect_results(ruleFullCrawl, max_tweets=MAX_TWEETS_PER_SEARCH, result_stream_args=full_archive_seach_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    }
   ],
   "source": [
    "print(len(tweets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "all result pages before the last page will have `MAX_TWEETS_PER_CALL` tweets\n",
    "\n",
    "the last page will have 1-500 tweets. Therefore we can calculate the total number of tweets obtain as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3595\n"
     ]
    }
   ],
   "source": [
    "total_num_tweets = 7*500 + tweets[7]['meta']['result_count']\n",
    "print(total_num_tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What the results look like"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result you get from `collect_results` will look something like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nit returns an array of results page objects\\n\\ncollect_results_output = [results_page1,results_page2 , ... ]\\n\\na results page has three fields: data, meta, includes\\nif you dont have any expansions you will not see includes\\n\\ndata is an array of tweet data objects\\nmeta is an object with fields: newest_id, oldest_id, result_count, next_token\\nincludes is an object which depends on what expansions you specify in expansions\\n\\n\\nresults_page1 = {\\n                    data:[data_for_tweet1, data_for_tweet2, ... ],\\n                    meta:{\\n                        'newest_id': '1355254021024526337',\\n                        'oldest_id': '1354856778647941121',\\n                        'result_count': 10,\\n                        'next_token': 'b26v89c19zqg8o3foskt5kx7wqa14ffp6hpnqd8sxqc8t'\\n                    },\\n                    includes:{\\n                        'users': [{'id': '30864583', 'name': 'AEI', 'username': 'AEI'}]\\n                    }\\n                }\\n\\ntweet data object depends on what tweet_fields you specify\\n\\ndata_for_tweet1 = {\\n                    'author_id': '30864583', \\n                    'id': '1355254021024526337', \\n                    'referenced_tweets': [{'type': 'retweeted', 'id': '1355239914858835969'}], \\n                    'text': 'RT @AEIecon: There is an opportunity for bipartisan cooperation on expanding coverage and controlling costs — but only if the parties set a…', \\n                    'public_metrics': {'retweet_count': 2, 'reply_count': 0, 'like_count': 0, 'quote_count': 0}, 'created_at': '2021-01-29T20:38:32.000Z', \\n                    'entities': {'mentions': [{'start': 3, 'end': 11, 'username': 'AEIecon', 'id': '809552311'}]}\\n                  }\\n\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "it returns an array of results page objects\n",
    "\n",
    "collect_results_output = [results_page1,results_page2 , ... ]\n",
    "\n",
    "a results page has three fields: data, meta, includes\n",
    "if you dont have any expansions you will not see includes\n",
    "\n",
    "data is an array of tweet data objects\n",
    "meta is an object with fields: newest_id, oldest_id, result_count, next_token\n",
    "includes is an object which depends on what expansions you specify in expansions\n",
    "\n",
    "\n",
    "results_page1 = {\n",
    "                    data:[data_for_tweet1, data_for_tweet2, ... ],\n",
    "                    meta:{\n",
    "                        'newest_id': '1355254021024526337',\n",
    "                        'oldest_id': '1354856778647941121',\n",
    "                        'result_count': 10,\n",
    "                        'next_token': 'b26v89c19zqg8o3foskt5kx7wqa14ffp6hpnqd8sxqc8t'\n",
    "                    },\n",
    "                    includes:{\n",
    "                        'users': [{'id': '30864583', 'name': 'AEI', 'username': 'AEI'}]\n",
    "                    }\n",
    "                }\n",
    "\n",
    "tweet data object depends on what tweet_fields you specify\n",
    "\n",
    "data_for_tweet1 = {\n",
    "                    'author_id': '30864583', \n",
    "                    'id': '1355254021024526337', \n",
    "                    'referenced_tweets': [{'type': 'retweeted', 'id': '1355239914858835969'}], \n",
    "                    'text': 'RT @AEIecon: There is an opportunity for bipartisan cooperation on expanding coverage and controlling costs — but only if the parties set a…', \n",
    "                    'public_metrics': {'retweet_count': 2, 'reply_count': 0, 'like_count': 0, 'quote_count': 0}, 'created_at': '2021-01-29T20:38:32.000Z', \n",
    "                    'entities': {'mentions': [{'start': 3, 'end': 11, 'username': 'AEIecon', 'id': '809552311'}]}\n",
    "                  }\n",
    "'''\n",
    "                  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2be5faf79681da6f2a61fdfdd5405d65d042280f7fba6178067603e3a2925119"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}