06_exploring_wikipedia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0593d3cd",
   "metadata": {},
   "source": [
    "# Exploring Wikipedia using its API\n",
    "\n",
    "No need to introduce Wikipedia, the online encyclopedia is now the default online reference for a lot of subjects.\n",
    "Each page contains one or more links to other pages, it is therefore natural to apply the graph exploration methods we have seen previously. The whole graph of wikipedia pages is quite big, the english version has ca. 7M pages and 500M links, and the graph is highly connected.\n",
    "\n",
    "A great thing about wikipedia is that almost all the data are open. Data is either available as [full dumps](https://dumps.wikimedia.org/) or [using the API](https://www.mediawiki.org/wiki/API:Main_page). Dumps are better for offline processing. There are a number of tools dedicated to processing data dumps from Wikipedia, e.g. [Sparkwiki](https://github.com/epfl-lts2/sparkwiki). In this tutorial we will be using the API to access the [English edition of Wikipedia](https://en.wikipedia.org), although adapting the code for another language is fairly trivial.\n",
    "\n",
    "## Experimenting the API using the sandbox\n",
    "There are a lot of possibilities to use the API, we only use the `query` action to retrieve data about pages. The [documentation](https://www.mediawiki.org/w/api.php?action=help&modules=query) provides a list to properties that can be retrieved for each page.\n",
    "\n",
    "The [API Sandbox](https://en.wikipedia.org/wiki/Special:ApiSandbox) helps testing and building queries quickly. If we need to retrieve\n",
    "the categories of Albert Einstein's wikipedia page it can be tried [here](https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=categories&titles=Albert%20Einstein) or use the following sandbox query:\n",
    "```\n",
    "https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=categories&titles=Albert%20Einstein\n",
    "```\n",
    "\n",
    "Once we are satisfied with the data we retrieve, it can be performed directly. Let us load the links from Einstein's wikipedia page using python `requests`. We add the `cllimit` parameter to retrieve the first 20 categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88e68e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec4ad3d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get('https://en.wikipedia.org/w/api.php?action=query&format=json&prop=categories&titles=Albert%20Einstein&cllimit=20')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e24bca2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "einstein_cats = r.json()\n",
    "einstein_cats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daa0b314",
   "metadata": {},
   "source": [
    "The result of the query is JSON-formatted and converted to a Python dict directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a78ddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "einstein_cats['query']['pages']['736']['title']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd63522",
   "metadata": {},
   "source": [
    "The query only return the 20 first categories of the page. While the `cllimit`parameter could be increased, there are cases where multiple queries are needed to retrieve all the data of a page. The response provides a `continue` key providing information about how to retrieve the following categories. In this case, appending `clcontinue=736|American_science_writers` to our query will retrieve the next categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a2aaffe",
   "metadata": {},
   "outputs": [],
   "source": [
    "rc = requests.get('https://en.wikipedia.org/w/api.php?action=query&format=json&prop=categories&titles=Albert%20Einstein&cllimit=20&clcontinue=736|American_science_writers')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f0af5b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "rc.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e325558",
   "metadata": {},
   "source": [
    "*Exercise*: experiment queries yourself, retrieve data for different pages. The `categories`, `links`, `pageviews` are of interest for us.\n",
    "\n",
    "## Using the Wikipedia-API package\n",
    "While it is fairly simple to make complete retrieval of the data using multiple requests fully automatic, we will leave this as an exercise for the readers and use a helper library that will handle this for us. \n",
    "\n",
    "There are multiple options available, we will use the [Wikipedia-API](https://github.com/martin-majlis/Wikipedia-API) library. You may want to check its [documentation](https://wikipedia-api.readthedocs.io/en/latest/API.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21283ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import wikipediaapi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d029db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the api object\n",
    "api = wikipediaapi.Wikipedia('en')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "119267f7",
   "metadata": {},
   "source": [
    "We can simply create a `Page` object and get its properties (which are lazily evaluated to limit the number of requests actually sent to Wikipedia API)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47c1348a",
   "metadata": {},
   "outputs": [],
   "source": [
    "albert = api.page('Albert Einstein')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52291b5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(albert.categories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055d16b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(albert.links)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8958e9f",
   "metadata": {},
   "source": [
    "It is interesting to see which requests are sent by increasing the logging level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23675cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "# helper function\n",
    "def set_wikipediaapi_logging(level):\n",
    "    wikipediaapi.logging.getLogger('wikipediaapi').handlers.clear() # ugly - remove handlers to avoid duplicates\n",
    "    wikipediaapi.log.setLevel(level=level)\n",
    "    # Set handler if you use Python in interactive mode\n",
    "    out_hdlr = wikipediaapi.logging.StreamHandler(sys.stderr)\n",
    "    out_hdlr.setFormatter(wikipediaapi.logging.Formatter('%(asctime)s %(message)s'))\n",
    "    out_hdlr.setLevel(level)\n",
    "    wikipediaapi.log.addHandler(out_hdlr)\n",
    "\n",
    "set_wikipediaapi_logging(wikipediaapi.logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55f7f210",
   "metadata": {},
   "outputs": [],
   "source": [
    "quantum = api.page('Quantum mechanics')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c239c17",
   "metadata": {},
   "outputs": [],
   "source": [
    "quantum.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f0c2d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "quantum.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13f9c210",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_wikipediaapi_logging(wikipediaapi.logging.WARN) # reset logging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5aaabbb",
   "metadata": {},
   "source": [
    "## Graph exploration\n",
    "The methods presented in this tutorial can be used to explore the Wikipedia page graph. However, the package [littleballoffur](https://github.com/benedekrozemberczki/littleballoffur) used to demonstrate the concepts cannot be used directly with the API. Therefore those methods have been implemented into a different package: [spikexplore](https://github.com/epfl-lts2/spikexplore).\n",
    "\n",
    "The package has no release (yet) so install it using pip:\n",
    "```\n",
    "pip install git+https://github.com/epfl-lts2/spikexplore.git\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa0a181d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "from spikexplore import graph_explore\n",
    "from spikexplore.backends.wikipedia import WikipediaNetwork\n",
    "from spikexplore.config import SamplingConfig, GraphConfig, DataCollectionConfig, WikipediaConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddf8cb5c",
   "metadata": {},
   "source": [
    "Spikexplore supports different backends: NetworkX (mostly for testing, in this case you can use littleballoffur), Twitter (requires the creation of a developer account to obtain API keys), and Wikipedia.\n",
    "\n",
    "You must create first the sampling backend you will use to acquire data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29bbda13",
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki_config = WikipediaConfig(lang='en') # adapt to your favorite language\n",
    "wiki_config.pages_ignored = [] # you can supply a list of page titles you want to ignore\n",
    "sampling_backend = WikipediaNetwork(wiki_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e964ec5",
   "metadata": {},
   "source": [
    "A second configuration object contains the parameters used for the graph creation. In our case the page graph is not weighted so keep the minimum edge weight to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d215019",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_config = GraphConfig(min_degree=1, min_weight=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c11fff8",
   "metadata": {},
   "source": [
    "Exploration parameters are stored in another object. In the example below, we sample randomly 10 % of the edges encountered at each hop. The `max_nodes_per_hop` provides an additional fine-tuning parameter to limit the growth of the graph, limiting the number of new neighbors for each node. The `Fireball` expansion will make it close to the forest fire sampling seen previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "093f4f71",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collection_config = DataCollectionConfig(exploration_depth=2, random_subset_mode=\"percent\",\n",
    "                                              random_subset_size=10, expansion_type=\"fireball\",\n",
    "                                              max_nodes_per_hop=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0789ebf9",
   "metadata": {},
   "source": [
    "Finally the graph and data collection confuguration objects are combined into a single one that will be passed to spikexplore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9db05c4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sampling_config = SamplingConfig(graph_config, data_collection_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c577117",
   "metadata": {},
   "source": [
    "Let us define starting points to explore the graph (you might want to adapt this to your needs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06bd27ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_nodes = ['Albert Einstein', 'Quantum mechanics', 'Theory of relativity']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11c26b43",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set_wikipediaapi_logging(wikipediaapi.logging.INFO) # optional, peep under the hood what is happening !! PRINTS A LOT OF OUTPUT !!\n",
    "graph_result =  graph_explore.explore(sampling_backend, initial_nodes, sampling_config)\n",
    "print('Collected {} nodes and {} edges'.format(graph_result.number_of_nodes(), graph_result.number_of_edges()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec3848b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.write_gexf(graph_result, 'wiki_einstein.gexf') # save result to view in Gephi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3fe5c1d",
   "metadata": {},
   "source": [
    "You can now open the result graph in Gephi and see if the communities make sense. Use different sets of parameters (sampling ratio, different initial nodes, etc.)\n",
    "\n",
    "With only 3 starting nodes and 2 hops, the graph is growing quickly and takes around 1 minute to collect. Adding an extra hop would take longer (but should remain reasonable), try it yourself if you have some time. Given the large number of connections in Wikipedia (the Albert Einstein page has more than 1000 links), exploring using Snowball sampling would lead to a much larger number of connections. You can try it by setting `random_subset_size` to 100 and set `max_nodes_per_hop` to a very large value to avoid being capped in any way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b88c2fca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# WARNING - this is going take some time to collect (ca. 10 minutes) !!\n",
    "data_collection_config_full = DataCollectionConfig(exploration_depth=2, random_subset_mode=\"percent\",\n",
    "                                              random_subset_size=100, expansion_type=\"coreball\",\n",
    "                                              degree=2, max_nodes_per_hop=1000000)\n",
    "sampling_config_full = SamplingConfig(graph_config, data_collection_config_full)\n",
    "# Uncomment this if you want to collect the full neighborhood\n",
    "# graph_full = graph_explore.explore(sampling_backend, initial_nodes, sampling_config_full)\n",
    "# print('Collected {} nodes and {} edges'.format(graph_full.number_of_nodes(), graph_full.number_of_edges()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b2478ac",
   "metadata": {},
   "source": [
    "Collecting the full neighborhood of the 3 initial nodes using Snowball sampling yields a graph having ca. 1800 nodes and 126000 edges ! You can download the resulting graph using [this link](https://drive.switch.ch/index.php/s/qJr6qqBLnOEOR2f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5011680",
   "metadata": {},
   "source": [
    "## Adding features to the graph\n",
    "Thanks to the wikipedia API, it is possible to retrieve the number of times a page has been viewed using the `pageviews`property. Unfortunately this is not part of the Wikipedia-API package. We need a small helper function to get those values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ab19ae8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from urllib.parse import quote\n",
    "\n",
    "# get the pageview data for a page. this is not efficient, you can send requests simultaneously\n",
    "# for multiple pages. There is no error checking, sending bad data will return you exceptions.\n",
    "def get_pageviews(page_titles, number_of_days=7, lang='en'):\n",
    "    pageviews = {}\n",
    "    # pageview query supports at most 50 pages per request (in practice MUCH less), split page titles into smaller chunks\n",
    "    chunk_size = 5 # try bigger values but you can have empty results..\n",
    "    nodesplit = np.array_split(page_titles, len(page_titles)//chunk_size)\n",
    "    \n",
    "    # get pageview data for eahc chunk\n",
    "    for ns in nodesplit:\n",
    "        pages = quote('|'.join(ns))\n",
    "        url = 'https://{}.wikipedia.org/w/api.php?action=query&format=json&prop=pageviews&titles={}&pvipdays={}'.format(lang, pages, number_of_days)\n",
    "        res = requests.get(url).json()\n",
    "        for k,v in res['query']['pages'].items():\n",
    "            pv = list(filter(None, v['pageviews'].values())) # some pageviews value might be None -> do not take them into account\n",
    "            pageviews[v['title']] = pv\n",
    "\n",
    "    return pageviews"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda8ec20",
   "metadata": {},
   "source": [
    "Let us retrieve the page views for all the nodes in the graph, for the last 10 days "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7197e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_stats = get_pageviews(graph_result.nodes(), number_of_days=10)\n",
    "graph_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8a64ac0",
   "metadata": {},
   "source": [
    "We will now compute the mean of the pageviews for each page and store it as an attribute in the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70c75afa",
   "metadata": {},
   "outputs": [],
   "source": [
    "pageviews_mean_graph = {k: np.mean(v) for k,v in graph_stats.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95c461aa",
   "metadata": {},
   "source": [
    "Store those values as attribute, and save the resulting graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "736e7d9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.set_node_attributes(graph_result, pageviews_mean_graph, name='mean_pageviews')\n",
    "nx.write_gexf(graph_result, 'wiki_einstein_pageviews.gexf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b513d41",
   "metadata": {},
   "source": [
    "You can now use Gephi to open the graph and define the node size according to the number of visits they received. It can also be used to remove some nodes from the graph that have little importance visit-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e45c6d6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Average pageviews range from min={} to max={}'.format(np.min(list(pageviews_mean_graph.values())), np.max(list(pageviews_mean_graph.values()))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8469ecc9",
   "metadata": {},
   "source": [
    "*Exercise*: Acquire a bigger graph with pageviews data and use it to remove nodes that receive few visits. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daa29419",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}