pq_parser.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse ProQuest Metadata\n",
    "This notebook includes a python function to parse newspaper articles downloaded from ProQuest Global Newsstream into one CSV file with metadata and full text (when full text is available). \n",
    "\n",
    "#### Download PQ files to use as input\n",
    "The script below takes as input .txt file downloads available via ProQuest Global Newsstream. These are available from ProQuest in batches of up to 100 articles per file. To save those files from your ProQuest search results:\n",
    "1. Select each article you want to save (or select all results on page) using result checkboxes.\n",
    "2. From the *...* button dropdown, select \"TXT - Text Only\"\n",
    "\n",
    "    ![Screenshot of Saving results from Proquest](imgs/pq_save_feb20.png \"Save results from PQ\")\n",
    "3. Accept all defaults and continue to save .txt file of bundled article downloads.\n",
    "4. Save the downloaded .txt file (or files) to a folder in the same directory as this notebook. In the example below, that folder is called \"txt_input\" but you can use any path name that you will then call in the final cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 1: import libraries required to run Python code\n",
    "import os\n",
    "import re\n",
    "import glob\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect metadata\n",
    "We need to tell the script which fields to collect from the .txt files. You can inspect the text files yourself to look for field names at the beginning of new lines such as `Title: ` or `Publication year: ` and then add them to the list variable called `fieldnames` below. Here is a list of field names available in the ProQuest text downloads as of July 2019:\n",
    "\n",
    "`'Title', 'Publication title', 'Publication year', 'Document URL', 'Full text', 'Links', 'Section', 'Publication subject', 'ISSN', 'Copyright', 'Abstract', 'Publication info', 'Last updated', 'Place of publication', 'Location', 'Author', 'Publisher', 'Identifier / keyword', 'Source type', 'ProQuest document ID', 'Country of publication', 'Language of publication', 'Publication date', 'Subject', 'Database', 'Document type'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fieldnames = ['Title', 'Publication title', 'Publication year', 'Document URL', 'Full text', 'Dateline', 'People', 'Business indexing term', 'Company / organization', 'Credit', 'CREDIT', 'Pages', 'Links', 'Section', 'Publication subject', 'ISSN', 'Copyright', 'Abstract', 'Publication info', 'Last updated', 'Place of publication', 'Location', 'Author', 'Publisher', 'Identifier / keyword', 'Source type', 'ProQuest document ID', 'Country of publication', 'Language of publication', 'Publication date', 'Subject', 'Database', 'Document type']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the script\n",
    "The function below:\n",
    "1. Accepts a path to a directory full of .txt files you want to process as its argument (e.g., `parsePQ(\"txt_input/\")`. \n",
    "2. Creates a csv file called `pq_metadata.csv`\n",
    "3. Cycles through every text file in the path you identified in step 1.\n",
    "4. Splits each document into individual articles using the `sep` separator.\n",
    "5. Splits each article into lines.\n",
    "6. For each line, matches `fieldnames` with existing metadata tags in each document. \n",
    "7. Saves the fieldname and following content (values) to a dictionary, `metadata_dict`.\n",
    "7. Writes the fieldnames (keys) and content (values) of the `metadata_dict` for each article to its own row in the CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sep = \"____________________________________________________________\"\n",
    "\n",
    "## function that takes a directory of .txt files from ProQuest as input \n",
    "def parsePQ(path, file_output=''): \n",
    "    '''This function parses text file downloads from ProQuest into metadata and full-text.\n",
    "    \n",
    "    Optional: set the second parameter to 'txt' to also output individual articles as .txt files in /output/\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    \n",
    "    path : str\n",
    "        path to .txt files that will be parsed (e.g., 'txts/' or 'pdfs/')\n",
    "    file_output : str   \n",
    "        change to file_output='txt' to return individual articles as text files; \n",
    "        \n",
    "    '''\n",
    "    \n",
    "    # create a dataframe with fieldname as column headers \n",
    "    df = pd.DataFrame(columns = fieldnames)\n",
    "    \n",
    "    #cycle through every text file in the directory given as an argument\n",
    "    files_all = glob.iglob(path + \"*.txt\")\n",
    "        \n",
    "    for filename in files_all:\n",
    "\n",
    "        #remove the path, whitespace, and '.txt' from filename to later use when printing output\n",
    "        file_id = filename[:-4].strip(path)\n",
    "\n",
    "        with open(filename, 'r', encoding='utf-8') as in_file:\n",
    "            # text var for string of all docs\n",
    "            text = in_file.read()\n",
    "\n",
    "            # split string by separator into single articles\n",
    "            docs = re.split(sep, text)\n",
    "\n",
    "            # remove first and last items from docs list: first item is empty string; last is copyright info\n",
    "            docs = docs[1:-1]\n",
    "\n",
    "            # loop through every doc to collect metadata and full text\n",
    "            for i, doc in enumerate(docs):\n",
    "                if file_output == 'txt':\n",
    "                    new_file = 'output/' + file_id + str(i) + '.txt'\n",
    "                    txt_file = open(new_file,'w') \n",
    "                # remove white space from beginning and end of each article\n",
    "                doc = doc.strip()\n",
    "\n",
    "                # skip any empty docs\n",
    "                if doc==\"\":\n",
    "                    continue\n",
    "                   \n",
    "                if file_output == 'txt':\n",
    "                    txt_file.write(doc) \n",
    "                    txt_file.close() \n",
    "                    \n",
    "                # split doc on every new line\n",
    "                metadata_lines = doc.split('\\n\\n')\n",
    "\n",
    "                #remove first \"line\" from article which is the article title without any field title\n",
    "                metadata_lines = metadata_lines[1:]\n",
    "\n",
    "                #declare a new dictionary\n",
    "                metadata_dict = {}\n",
    "\n",
    "                #for each element add the fieldname/key and following value to a dictionary\n",
    "                for line in metadata_lines:\n",
    "                    \n",
    "                    #ignore lines that do not have a field beginning \"Xxxxxx:\" (e.g. \"Publication title: \")\n",
    "                    if not re.match(r'^[^:]+: ', line):\n",
    "                        continue\n",
    "                    #looks for beginning of new line following structure of \"Publication year: \" splitting on the colon space\n",
    "                    (key,value) = line.split(sep=': ', maxsplit=1)\n",
    "                    \n",
    "                    #only add to dictionary if the key is in fieldnames\n",
    "                    if key in fieldnames:\n",
    "                        metadata_dict[key] = value\n",
    "                    \n",
    "                    #look for full text that was cut off by double line breaks and append it to the full text field\n",
    "                    else:\n",
    "\n",
    "                        if 'Full text' in metadata_dict.keys():\n",
    "                            metadata_dict['Full text'] = metadata_dict['Full text'] + line\n",
    "                \n",
    "                #write the article to a row in the dataframe\n",
    "                df.loc[len(df)] = metadata_dict        \n",
    "            print(\"Saving\", file_id)\n",
    "    \n",
    "    #return dataframe \n",
    "    return(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the script\n",
    "Running the cell above loads the function, but doesn't do anything yet. \n",
    "To *execute* the function, run the cell below, replacing `txts/` with the name of your folder full of .txt files. \n",
    "\n",
    "Add a second parameter, file_output='txt', if you would like to return individual articles as text files. In this case make sure there is an ```/output/``` directory available to store the text files.\n",
    "\n",
    "```df = parsePQ(\"txt_input/\", file_output='txt')```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#run the function and save to a pandas dataframe\n",
    "df = parsePQ('txts/')\n",
    "\n",
    "#remove empty rows\n",
    "df.dropna(how='all', inplace = True)\n",
    "\n",
    "#combine different credit metadata columns into one\n",
    "df['Credits'] = df['Credit'] + ' ' + df['CREDIT']\n",
    "df = df.drop(['Credit', 'CREDIT'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data can be cleaned or examined as a dataframe here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export to a csv file\n",
    "df.to_csv('pq_metadata.csv', index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Working with the CSV file\n",
    "Microsoft Excel does not properly render the rows for some articles with unique line breaks. If you open the CSV file and find that the metadata for some articles is breaking up improperly over columns, you can follow these directions to work with the dataset:\n",
    "1. Create a new blank Google Sheet and choose File > Import. \n",
    "2. Select the CSV file that you exported and in the pop up change the Separator type to \"Comma\" and uncheck the \"Convert text...\" option. The data should be properly parsed in Google Sheets. \n",
    "![Pop-up box offering how to import the CSV file](imgs/import_file.png \"Import file into Google Sheets\")\n",
    "3. If you prefer to work with the data in Excel, then go to File > Download as > Microsoft Excel and save a copy to your computer.\n",
    "4. When opening the file you may get an alert from Excel. If so, choose \"Yes\" to recover data.\n",
    "\n",
    "![Excel alert warning there may be problems with the metadata](imgs/alert.png \"Excel alert\")\n",
    "\n",
    "Created by Cody Hennesy and David Naughton (University of Minnesota, Twin Cities, Libraries). Feel free to email Cody (chennesy@umn.edu) with any questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}