11_external_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text Files\n",
    "\n",
    "Data is often stored in formats that are not easy to read unless you have some specialised software. Excel spreadsheets are one example; how do you read an Excel spreadsheet if you don't have Excel? This reliance on particular software is not very useful for programmatic analysis of data. Instead if we're going to use a programming language to analyse or process our data we would prefer some \"easy to deal with\" format. Text files or [flatfiles](http://en.wikipedia.org/wiki/Text_file) provide just the sort of format we need (there are other choices). You can think of these as a single sheet from a spreadsheet with the data arranged in rows (separated by our old friend the ```\\n``` character) and columns (separated by some other character)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There are two common characters used to separate the data in text files into columns. The ```\\t```, or tab stop character and the humble comma (,). In both cases fields in our input file are separated by a single tab stop or a single comma. \n",
    "\n",
    "This gives rise to two commonly used file extensions (the bit after the dot in file names e.g. in ```myfile.docx``` the 'docx' is the file extension.). These file extensions are ```.tsv``` for 'tab separated file' and ```.csv``` for 'comma separated file'. Other separators in text files of data may be little used characters such as '|', ':' or spaces. As a quick aside separating fields in data files by spaces is tricky because an individual value might also contain spaces and therefore be inappropriately divided over more than one column.\n",
    "\n",
    "As an illustration if you were to print a couple of lines from a ```.tsv``` file out it would look something like this:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "data_field1\\tdata_field2\\tdata_field3\\n \n",
    "data_field1\\tdata_field2\\tdata_field3\\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each field is separated by a tab stop ```\\t``` and the end of a line is indicated by the ```\\n``` combination.\n",
    "\n",
    "In a ```.csv``` file you would see:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "data_field1,data_field2,data_field3\\n\n",
    "data_field1,data_field2,data_field3\\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Note:\n",
    "\n",
    "It's important to note that there is NO STANDARD for either the layout OR naming of text files. In the exercises that follow the text file we will be working with has the extension ```.csv``` which should mean comma separated but in fact the data is separated by tabs - because that's what I'm in the habit of doing. Please ignore my bad habits!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Opening (and Reading) Files\n",
    "\n",
    "When you open a file in a programme such as Word or Excel you have to select the file you want from a 'File -> Open' dialogue of some kind. In other words you have to point the programme at the specific file you want which is stored at some specific location on your computer. This is also true when you open a file in Python. \n",
    "The difference is that when you open a file in python you have to specify the file location in words (as a **filepath**) rather then selecting from a dialogue. So the first thing we have to do is assign the file location to a variable. This textual representation of the file location is called the filepath. \n",
    "\n",
    "Once we have the filepath we can use the python function ```open()``` to 'get a handle' on the file. This **file handle** should also be assigned to a variable. We can then operate on that variable. Let's see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_io.TextIOWrapper name='data/elderlyHeightWeight.csv' mode='r' encoding='UTF-8'>\n"
     ]
    }
   ],
   "source": [
    "file_loc = 'data/elderlyHeightWeight.csv'\n",
    "f_hand = open(file_loc, 'r')\n",
    "print(f_hand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In the example above we have opened a small file containing height and weight information on a group of elderly men and women from a body composition study."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "In the first line we assign the file path to the variable ```file_loc```. We then use that variable to open a file handle (```f_hand```) using the ```open()``` function. The file location is the first argument to the ```open()``` function and the second argument ```'r'``` indicates that we want to *read* from the file i.e. we want access to the data in the file but we don't want to change the file itself. Finally we use a ```print``` statement to print the file handle. \n",
    "\n",
    "The results of the ```print``` statement might surprise you. Rather than printing the contents of the file what we get is a representation of the location in the computer memory of where the file is i.e. at memory location 0x7... etc in the example above.\n",
    "\n",
    "Whilst we can also open Excel or Word files in python this requires the use of special software libraries. We'll see some of those later in the course. Mostly when we are analysing data in files we open simple text files. Both Excel and Word can save files out as simple text files. \n",
    "\n",
    "The first few lines of the file we will work with are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\r\n",
      "F\t77\t63.8\t155.5\r\n",
      "F\t80\t56.4\t160.5\r\n",
      "F\t76\t55.2\t159.5\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 4 data/elderlyHeightWeight.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "While the file handle does not contain the data from the file (it only points at it) it is easy to construct a ```for``` loop to cycle through the lines of the file and carry out some computation. For example we can easily count the number of lines in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 19 lines in the file.\n"
     ]
    }
   ],
   "source": [
    "count = 0 # initialise\n",
    "\n",
    "for line in f_hand: # iterate\n",
    "    count = count + 1\n",
    "    \n",
    "print('There are %d lines in the file.' % count    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Why doesn't `open()` open the file directly?\n",
    "\n",
    "It might seem stupid that after using ```open()``` a file handle is created but the file contents aren't directly available. The reason the ```open()``` function does not read the whole file immediately but just points to it is to do with file size. If you do not know in advance the size of the files you are dealing with (often the case - how often do you check the size of files you open on your computer?) automatically opening *very* large files could:\n",
    "\n",
    "* Take a long time\n",
    "* Crash the whole computer system - essentially you'd run out of memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For some biological applications the data files (which may well be text files e.g. in RNA Seq data - see comment [here](http://seqanswers.com/forums/showthread.php?t=10787)) can be very large. So it's safer to point to the file rather than automatically open it. This is also true of other data or informatics applications.\n",
    "\n",
    "In the ```for``` loop above python splits the file into lines based on the newline character (```\\n``` - the split is implicit), increments the ```count``` variable by 1 for every line and then discards that line. So there is only ever one line from the file in the computer memory at any given time.\n",
    "\n",
    "If you know that your file is likely to be small you can read the whole file into memory with the ```read()``` method (remember the dot notation!). \n",
    "\n",
    "This reads the entire contents of the file, newlines and all, into one large string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "F\t80\t56.4\t160.5\n",
      "F\t76\t55.2\t159.5\n",
      "F\t77\t58.5\t151\n",
      "F\t82\t64\t165.5\n",
      "F\t78\t51.6\t167\n",
      "F\t85\t54.6\t154\n",
      "F\t83\t71\t153\n",
      "M\t79\t75.5\t171\n",
      "M\t75\t83.9\t178.5\n",
      "M\t79\t75.7\t167\n",
      "M\t84\t72.5\t171.5\n",
      "M\t76\t56.2\t167\n",
      "M\t80\t73.4\t168.5\n",
      "M\t75\t67.7\t174.5\n",
      "M\t75\t93\t168\n",
      "M\t78\t95.6\t168\n",
      "M\t80\t75.6\t183.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_loc = 'data/elderlyHeightWeight.csv'\n",
    "f_hand = open(file_loc, 'r')\n",
    "f_data = f_hand.read()\n",
    "print(f_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "f_hand.close()\n",
    "f_hand = open(file_loc)\n",
    "print(f_hand.readline(), end = '')\n",
    "print(f_hand.readline(), end = '')\n",
    "f_hand.close()\n",
    "f_hand = open(file_loc)\n",
    "lines = f_hand.readlines()\n",
    "print(type(lines))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "286\n",
      "Gender\tAge\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Gender\\tAge\\tbody wt\\tht\\n'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_loc = 'data/elderlyHeightWeight.csv'\n",
    "f_hand = open(file_loc, 'r')\n",
    "f_data = f_hand.read()\n",
    "f_data[:22]\n",
    "print(len(f_data))\n",
    "print( f_data[:10])\n",
    "r'{}'.format(f_data[:22]) # note the tab stop in the output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "In the above example we first create the file handle and then ```read``` the entire contents of the file into one string. We check the length of that string (286 characters including whitespace characters like ```\\n```) and we print the first 10 characters (refer back to the material on slicing if you're unsure how the ```[:10]``` slice works). The ```print``` statement interprets the tab stop properly but if we just ask for the first 22 characters to be returned (i.e. we do not use ```print```) we can see the tab stop and the ```\\n```. Compare this to the illustration of a .tsv file shown above.\n",
    "\n",
    "Using the ```print``` statement and subsetting is fine but not convenient. You might want to print the whole of the first line. The ```readline()``` method will read one line at a time and you can use this to e.g. just display the header line (if you know or suspect that your file has a header line)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "\n",
      "F\t77\t63.8\t155.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc, 'r')\n",
    "line = f_hand.readline() # reads first line\n",
    "print(line)\n",
    "# next line \n",
    "line=f_hand.readline()\n",
    "print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "After reading in the current line ```readline()``` then moves on to the next line. So calling ```readline()``` again uses the next line in the file. One other thing to bear in mind is that ```readline()``` leaves whitespace and in particular the ```\\n``` character at the end of the line. You can see that above (there's a blank line between the printed lines) in the following example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F\\t80\\t56.4\\t160.5\\n'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = f_hand.readline() # note next line has been read\n",
    "line # compare to print above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you're using python to join lines together in some new format that might not be what you want. There is a method,  ```strip()```(see [here](https://docs.python.org/3/library/string.html)) that removes whitespace at the end of lines and can be used to remove this potentially extraneous ```\\n``` character. Note also that methods can be **chained** together so you can use ```readline()``` and ```strip()``` sequentially using the following syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Gender\\tAge\\tbody wt\\tht'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_hand = open(file_loc, 'r') # read in in file again to get header line\n",
    "line = f_hand.readline().strip() # read the line then strip the whitespace at the end of the line\n",
    "line # no \\n!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Use of ```strip()``` has removed the ```\\n``` from our selected line. There are also ```lstrip()``` and ```rstrip()``` methods that strip whitespace from only the left or right sides of a string respectively.\n",
    "\n",
    "Just to confuse you further there's also a ```readlines()``` method that reads all the lines in the file into a ```list```. Again the lines are separated on the invisible ```\\n``` character. This can be handy because you can assign the ```list``` to a variable and then loop through the list to print file lines or simply extract the lines you want using slice notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gender\\tAge\\tbody wt\\tht\\n', 'F\\t77\\t63.8\\t155.5\\n']\n",
      "19\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc, 'r')\n",
    "lines = f_hand.readlines()\n",
    "print(lines[0:2]) # check we have a list\n",
    "print(len(lines))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "F\t80\t56.4\t160.5\n",
      "F\t76\t55.2\t159.5\n"
     ]
    }
   ],
   "source": [
    "for i in range(4):\n",
    "    print (lines[i].strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---\n",
    "\n",
    "### Alternative Implementations (just for fun)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### no. 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "F\t80\t56.4\t160.5\n",
      "F\t76\t55.2\t159.5\n"
     ]
    }
   ],
   "source": [
    "for line in lines[:4]:\n",
    "    print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "###  no. 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "F\t80\t56.4\t160.5\n",
      "F\t76\t55.2\t159.5\n"
     ]
    }
   ],
   "source": [
    "for i, line in enumerate(lines):\n",
    "    if i == 4:\n",
    "        break\n",
    "    print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Just like ```readline()``` the ```readlines()``` method leaves the trailing \\n at the end of the line but you can use ```strip()``` to remove it if you have to as we did above. We had to use the ```strip()``` method on the individual line rather than on the list of lines as lists ```strip()``` does not operate on lists. Try moving the ```strip()``` to the end of ```lines = f_hand.readlines()``` and see what kind if error you get.\n",
    "\n",
    "Finally (and perhaps most usefully) there is the ```splitlines()``` method that does the same as ```readlines()``` but drops the trailing ```\\n``` automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "F\t77\t63.8\t155.5\n",
      "F\t80\t56.4\t160.5\n",
      "F\t76\t55.2\t159.5\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc, 'r')\n",
    "lines = f_hand.read().splitlines() # read file, then split lines to lists, drops trailing \\n\n",
    "for i in range(4):\n",
    "    print (lines[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Notice we didn't have to use ```strip()```.\n",
    "\n",
    "One final thing to note is that whenever we finish with a file we should close it. Leaving files 'open' after data has been read from them can lead to increasing amounts of memory being used and also corruption of the file. Closing files is accomplished by using the ```close()``` method on the file handle. Also illustrated is a simple filter to print out only the male data using the string method ```startswith()``` - which returns a boolean value depending on whether the line begins with the given argument (M in this case) or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "M\t79\t75.5\t171\n",
      "M\t75\t83.9\t178.5\n",
      "M\t79\t75.7\t167\n",
      "M\t84\t72.5\t171.5\n",
      "M\t76\t56.2\t167\n",
      "M\t80\t73.4\t168.5\n",
      "M\t75\t67.7\t174.5\n",
      "M\t75\t93\t168\n",
      "M\t78\t95.6\t168\n",
      "M\t80\t75.6\t183.5\n"
     ]
    }
   ],
   "source": [
    "file_loc = 'data/elderlyHeightWeight.csv' # relative path\n",
    "f_hand = open(file_loc, 'r')\n",
    "lines = f_hand.read().splitlines() # lines to a list\n",
    "print (lines[0]) # header\n",
    "\n",
    "for line in lines: # loop to filter\n",
    "    if line.startswith('M'):\n",
    "        print (line)\n",
    "    \n",
    "f_hand.close()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## `filter` alternative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def filter_function(line):\n",
    "    return line.startswith('M')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\n",
      "M\t79\t75.5\t171\n",
      "M\t75\t83.9\t178.5\n",
      "M\t79\t75.7\t167\n",
      "M\t84\t72.5\t171.5\n",
      "M\t76\t56.2\t167\n",
      "M\t80\t73.4\t168.5\n",
      "M\t75\t67.7\t174.5\n",
      "M\t75\t93\t168\n",
      "M\t78\t95.6\t168\n",
      "M\t80\t75.6\t183.5\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc)\n",
    "lines = f_hand.readlines()\n",
    "male_gender = filter(filter_function, lines)\n",
    "print(lines[0].strip())\n",
    "for ml in male_gender:\n",
    "    print(ml.strip())\n",
    "f_hand.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Using `lambda` expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "M\t79\t75.5\t171\n",
      "M\t75\t83.9\t178.5\n",
      "M\t79\t75.7\t167\n",
      "M\t84\t72.5\t171.5\n",
      "M\t76\t56.2\t167\n",
      "M\t80\t73.4\t168.5\n",
      "M\t75\t67.7\t174.5\n",
      "M\t75\t93\t168\n",
      "M\t78\t95.6\t168\n",
      "M\t80\t75.6\t183.5\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc)\n",
    "male_gender = filter(lambda l: l.startswith('M'), f_hand)\n",
    "for ml in male_gender:\n",
    "    print(ml.strip())\n",
    "f_hand.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Exercises "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Show the content of the file using a Shell command\n",
    "\n",
    "#### Tip 1: The shell command to be used could be `cat`\n",
    "#### Tip 2: Remember the `!` (esclamation mark)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender\tAge\tbody wt\tht\r\n",
      "F\t77\t63.8\t155.5\r\n",
      "F\t80\t56.4\t160.5\r\n",
      "F\t76\t55.2\t159.5\r\n",
      "F\t77\t58.5\t151\r\n",
      "F\t82\t64\t165.5\r\n",
      "F\t78\t51.6\t167\r\n",
      "F\t85\t54.6\t154\r\n",
      "F\t83\t71\t153\r\n",
      "M\t79\t75.5\t171\r\n",
      "M\t75\t83.9\t178.5\r\n",
      "M\t79\t75.7\t167\r\n",
      "M\t84\t72.5\t171.5\r\n",
      "M\t76\t56.2\t167\r\n",
      "M\t80\t73.4\t168.5\r\n",
      "M\t75\t67.7\t174.5\r\n",
      "M\t75\t93\t168\r\n",
      "M\t78\t95.6\t168\r\n",
      "M\t80\t75.6\t183.5\r\n"
     ]
    }
   ],
   "source": [
    "!cat data/elderlyHeightWeight.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Ex. no 2\n",
    "\n",
    "Print all the lines in the file where the `Age` value is in the range `[70, 80)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F\t77\t63.8\t155.5\n",
      "F\t76\t55.2\t159.5\n",
      "F\t77\t58.5\t151\n",
      "F\t78\t51.6\t167\n",
      "M\t79\t75.5\t171\n",
      "M\t75\t83.9\t178.5\n",
      "M\t79\t75.7\t167\n",
      "M\t76\t56.2\t167\n",
      "M\t75\t67.7\t174.5\n",
      "M\t75\t93\t168\n",
      "M\t78\t95.6\t168\n"
     ]
    }
   ],
   "source": [
    "f_hand = open(file_loc)\n",
    "for i, line in enumerate(f_hand):\n",
    "    if i == 0:\n",
    "        continue\n",
    "    line = line.strip()\n",
    "    _, age, *_ = line.split('\\t')\n",
    "    if 70 <= int(age) < 80:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Ex. no 3 \n",
    "\n",
    "Print the two lines in the files for each `gender` corresponding to the two entries with the (relative) maximum value of *body weight* (`body wt`) plus *height* (`ht`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Sol `#1`:  Using a Dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "info = {}  # Dictonary holding per-sex lines info\n",
    "f_hand = open(file_loc)\n",
    "lines = f_hand.read().splitlines()\n",
    "for l in lines[1:]:\n",
    "    l = l.strip()\n",
    "    key = l[0]\n",
    "    info.setdefault(key, [])\n",
    "    info[key].append(tuple(l.split('\\t')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'F': [('F', '77', '63.8', '155.5'),\n",
      "       ('F', '80', '56.4', '160.5'),\n",
      "       ('F', '76', '55.2', '159.5'),\n",
      "       ('F', '77', '58.5', '151'),\n",
      "       ('F', '82', '64', '165.5'),\n",
      "       ('F', '78', '51.6', '167'),\n",
      "       ('F', '85', '54.6', '154'),\n",
      "       ('F', '83', '71', '153')],\n",
      " 'M': [('M', '79', '75.5', '171'),\n",
      "       ('M', '75', '83.9', '178.5'),\n",
      "       ('M', '79', '75.7', '167'),\n",
      "       ('M', '84', '72.5', '171.5'),\n",
      "       ('M', '76', '56.2', '167'),\n",
      "       ('M', '80', '73.4', '168.5'),\n",
      "       ('M', '75', '67.7', '174.5'),\n",
      "       ('M', '75', '93', '168'),\n",
      "       ('M', '78', '95.6', '168'),\n",
      "       ('M', '80', '75.6', '183.5')]}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint  # pprint is for **pretty printing** structures\n",
    "pprint(info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('M', '78', '95.6', '168')\n"
     ]
    }
   ],
   "source": [
    "max_male = max(info['M'], key=lambda e: float(e[2]) + float(e[3]))\n",
    "print(max_male)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Sol. `#2`: Using a list comprehension "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "## Creating Partial Lists using **List Comprehension**\n",
    "males = [l.strip().split('\\t') for l in lines[1:] \n",
    "             if l.startswith('M')]\n",
    "females = [l.strip().split('\\t') for l in lines[1:] \n",
    "               if l.startswith('F')] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['M', '79', '75.5', '171'],\n",
       " ['M', '75', '83.9', '178.5'],\n",
       " ['M', '79', '75.7', '167'],\n",
       " ['M', '84', '72.5', '171.5'],\n",
       " ['M', '76', '56.2', '167'],\n",
       " ['M', '80', '73.4', '168.5'],\n",
       " ['M', '75', '67.7', '174.5'],\n",
       " ['M', '75', '93', '168'],\n",
       " ['M', '78', '95.6', '168'],\n",
       " ['M', '80', '75.6', '183.5']]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "males"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['M', '78', '95.6', '168']\n"
     ]
    }
   ],
   "source": [
    "max_male = max(males, key=lambda e: float(e[2]) + float(e[3]))\n",
    "print(max_male)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The ```csv``` module\n",
    "\n",
    "Getting the data from a file and doing something with it is all well and good. However once we've done our analysis we usually want to save the results to another file. We can do this using base python but it's easier if we use a python **library**, in this case the [```csv```](http://www.pythonforbeginners.com/systems-programming/using-the-csv-module-in-python/) library. We'll learn more about libraries in the next unit but for now just consider libraries as extra python code that you can get access to if you need it. In fact that's exactly what many libraries are. So the quesion arises 'how do we get access to a library?'. We have to tell python we want to use the library up front. To do this we use the ```import``` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It's that simple! Now python makes available to us all the useful code in the ```csv``` library. The ```csv``` library, unsurprisingly, contains python functions and methods to make dealing with csv (and other) text files easier. Let's first see how to open a text file using the ```csv``` library and printing out the first few lines.\n",
    "\n",
    "To read data from a csv file, we use the ```reader()``` function. The ```reader()``` function takes each line of the file and makes a ```reader``` object containing lists made up of each row in the input data. Objects in programming are containers for both data and methods that act on that data (a bit esoteric so don't worry if you don't quite get that). One method the ```reader``` object supports is the ```.next()``` method. We can use this to access each row at a time. Notably once we have processed the line it's gone from the ```reader``` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Note:\n",
    "\n",
    "From **here on**, we are going to keep using the `with/as` statement to handle I/O operations, namely **Context Manager** objects.\n",
    "\n",
    "For more information, see this [notebook](09 Exceptions.ipynb#ctx).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gender', 'Age', 'body wt', 'ht']\n",
      "\n",
      "['F', '77', '63.8', '155.5']\n",
      "['F', '80', '56.4', '160.5']\n",
      "['F', '76', '55.2', '159.5']\n",
      "['F', '77', '58.5', '151']\n"
     ]
    }
   ],
   "source": [
    "# import csv - already done\n",
    "with open('data/elderlyHeightWeight.csv', 'r') as csvfile:\n",
    "    reader = csv.reader(csvfile, delimiter='\\t') # define the field delimiter\n",
    "    header = next(reader)\n",
    "    print (header)\n",
    "    print () # blank line\n",
    "\n",
    "    for i in range(4):\n",
    "        print (next(reader)) # print the first 4 lines after the header"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can see that the ```reader()``` function has processed each line into a single list element based on the field delimiter we supplied. Importantly also note that all the values are now of type ```str``` in each list (everything is in quotes). This is important if you want to do calculations on these values. \n",
    "\n",
    "Using the ```csv``` module makes it easy to select whole columns by selecting the data we want from the ```reader```.  We'll use the ```.next()``` method to find the column order and then iterate over the rows with a ```for``` loop to pull out height and weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gender', 'Age', 'body wt', 'ht']\n",
      "['Weight', '63.8', '56.4', '55.2', '58.5', '64', '51.6', '54.6', '71', '75.5', '83.9', '75.7', '72.5', '56.2', '73.4', '67.7', '93', '95.6', '75.6']\n",
      "['Height', '155.5', '160.5', '159.5', '151', '165.5', '167', '154', '153', '171', '178.5', '167', '171.5', '167', '168.5', '174.5', '168', '168', '183.5']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "with open('data/elderlyHeightWeight.csv', 'r') as csvfile:\n",
    "    reader = csv.reader(csvfile, delimiter='\\t') # define the field delimiter\n",
    "\n",
    "    # use next() method on reader object to id the headers\n",
    "    headers = next(reader)\n",
    "    print(headers)\n",
    "    \n",
    "    # we now know weight index is 2, height index is 3\n",
    "    \n",
    "    weight  = ['Weight'] # list to hold data, put in header \n",
    "    height = ['Height']\n",
    "\n",
    "    for row in reader:\n",
    "        weight.append(row[2])\n",
    "        height.append(row[3])\n",
    "    \n",
    "    print (weight)\n",
    "    print (height)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The ```iterable``` in the \n",
    "\n",
    "```\n",
    "for...\n",
    "``` \n",
    "\n",
    "loop above is each row of the input file. From each row we simply capture the two values we want and add these to lists. We could then further process the data in these two lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Writing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In order to open a file for writing we use the ```'w'``` parameter in our ```open()``` statement. Rather obviously ```'w'``` stands for write. If the file doesn't exist a new file is created with the given name and extension.\n",
    "\n",
    "Note that if the file exists then opening it with the ```'w'``` argument removes any data that was in the file and overwrites it with what you put in. This may not be what you wanted to do. We'll cover how you append data to a file without overwriting the contents shortly. \n",
    "\n",
    "Once we have an open file we can write data to it with the ```write()``` method applied to the file handle.\n",
    "\n",
    "Let's open a file and write some data to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "with open('data/test.txt', 'w') as f_out:\n",
    "    for i in range(10):\n",
    "        line  = 'Line ' + str(i) + '\\n'\n",
    "        f_out.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you run the above code a new file should appear in your data directory (notice we opened the writeable file in the ```/data``` directory) called ```test.txt```. That file should have 10 lines in it with the word 'Line' and a number from 0-9. \n",
    "\n",
    "In the above code we first opened (created) the file ```test.txt``` and then ran through a range of numbers (from 0 to 9) using a ```for``` loop. At each iteration of the loop we concatenated (joined) the word 'Line' to the string representation of the number (note the use of ```str```) and a newline character. Finally we wrote each of the resulting strings to our new file. In the last line we closed the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Putting it together!\n",
    "\n",
    "Write a script that uses the ```csv``` module to open a file after getting a filepath from the user. Use the script to open the ```elderlyHeightWeight.csv``` file. Write out a new file containing only male data. Remember to close all the files once your done. In addition include a ```try\\except``` clause to handle the situation where the requested file doesn't exist.\n",
    "\n",
    "Hint: ```csv.reader``` objects are lists. Recall how you ```.join()``` lists elements into a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Adding data to an existing file\n",
    "\n",
    "As noted above if you open an existing file and write data to it all that pre-existing data gets over written. That's not usually what you want to do. In fact in general you probably never want to write to any file that has raw data you are going to analyse in it - because you might lose or screw-up your original data. Sometimes however you might want to add new measurements (perhaps taken over time) to an existing file. For these cases there's the ```'a'``` argument to the ```open()``` function. The ```a``` stands for append. Let's take the file containing only the male data we wrote in the last exercise, open it in append mode and write the female data to that file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "# assumes your file was called male_data.tsv\n",
    "try:\n",
    "    with open('data/male_data.tsv', 'a') as new_file, open('data/elderlyHeightWeight.csv', 'r') as f_hand: \n",
    "        reader = csv.reader(f_hand, delimiter='\\t') # define the field delimiter\n",
    "        male_data = [line for line in reader if line[0] == 'M']\n",
    "        for line in male_data:\n",
    "            new_file.write('\\t'.join(line)+'\\n')\n",
    "except FileNotFoundError:\n",
    "    print('The file does not exist.')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}