dev/progress_report_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Progress report 3 notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "This notebook loads and parses a sample of the XML data used in the project. A general report that includes more information about the project is available in the [project report](../docs/progress_report.md).\n",
    "\n",
    "This notebook continues [progress_report_1.ipynb](progress_report_1.ipynb) and [progress_report_2.ipynb](progress_report_2.ipynb), integrating improvements, corrections, and new functionality. This is the *existing* option for report 3, renamed for easy access to and comparison of the three stages.\n",
    "\n",
    "We have changed the method in the following two ways:\n",
    "\n",
    "### 1. Replace syllabification with C(C)/V decomposition\n",
    "\n",
    "Instead of decomposing the rhyme zone into syllables, we decompose it into sequences of vowels and consonant clusters (which may be single consonants). This modification was adopted because segments that should be regarded as matching for rhyme analysis purposes may belong to different syllables, which means that syllabification could compromise their identification. For example:\n",
    "\n",
    "<table>\n",
    "    <tr><th>Word</th><th>Phonetic</th><th>|</th><th colspan=\"5\">Syllables</th><th>|</th><th colspan=\"11\">C(C)/V decomposition</th></tr>\n",
    "    <tr>\n",
    "        <td>вы́бора</td>\n",
    "        <td>vIbara</td>\n",
    "        <td>|</td>\n",
    "        <td>vI</td>\n",
    "        <td>-</td>\n",
    "        <td>ba</td>\n",
    "        <td>-</td>\n",
    "        <td>ra</td>\n",
    "        <td>|</td>\n",
    "        <td>v</td>\n",
    "        <td>-</td>\n",
    "        <td>I</td>\n",
    "        <td>-</td>\n",
    "        <td>b</td>\n",
    "        <td>-</td>\n",
    "        <td>a</td>\n",
    "        <td>-</td>\n",
    "        <td>r</td>\n",
    "        <td>-</td>\n",
    "        <td>a</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>вы́борка</td>\n",
    "        <td>vIbarka</td>\n",
    "        <td>|</td>\n",
    "        <td>vI</td>\n",
    "        <td>-</td>\n",
    "        <td>bar</td>\n",
    "        <td>-</td>\n",
    "        <td>ka</td>\n",
    "        <td>|</td>\n",
    "        <td>v</td>\n",
    "        <td>-</td>\n",
    "        <td>I</td>\n",
    "        <td>-</td>\n",
    "        <td>b</td>\n",
    "        <td>-</td>\n",
    "        <td>a</td>\n",
    "        <td>-</td>\n",
    "        <td>rk</td>\n",
    "        <td>-</td>\n",
    "        <td>a</td>\n",
    "    </tr>\n",
    "</table>\n",
    "    \n",
    "\n",
    "### 2. Identify rhymes by segments, rather than distinctive features\n",
    "\n",
    "Our original assumption was that after decomposing syllables into onset, nucleus, and coda we could decompose those parts into segments (the nucleus is always present and monosegmental; the onset and coda are optional and potentially polysegmental). This turns out not to be helpful in situations where a monosegmental component might have to be compared to a polysegmental one, since it isn’t clear where the single segment should be aligned. We may reevaluate this decision later (perhaps feature-level comparison will prove useful in cases of isosegmental columns; perhaps we will introduce alignment logic to assign the segments in anisosegmental columns), and we will nonetheless continue to use phonetic distinctive features in evaluating and analyzing rhyming lines, even if not for identifying them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample data\n",
    "\n",
    "The sample dataset consists of multiple stanzas from a long poem. An artificial one-stanza extract looks like the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```xml\n",
    "<poem>\n",
    "    <stanza stanzaNo=\"001\">\n",
    "        <line lineNo=\"001\">\"Мой дядя самых честных пр<stress>а</stress>вил,</line>\n",
    "        <line lineNo=\"002\">Когда не в шутку занем<stress>о</stress>г,</line>\n",
    "        <line lineNo=\"003\">Он уважать себя заст<stress>а</stress>вил</line>\n",
    "        <line lineNo=\"004\">И лучше выдумать не м<stress>о</stress>г.</line>\n",
    "        <line lineNo=\"005\">Его пример другим на<stress>у</stress>ка;</line>\n",
    "        <line lineNo=\"006\">Но, боже мой, какая ск<stress>у</stress>ка</line>\n",
    "        <line lineNo=\"007\">С больным сидеть и день и н<stress>о</stress>чь,</line>\n",
    "        <line lineNo=\"008\">Не отходя ни шагу пр<stress>о</stress>чь!</line>\n",
    "        <line lineNo=\"009\">Какое низкое ков<stress>а</stress>рство</line>\n",
    "        <line lineNo=\"010\">Полу-живого забавл<stress>я</stress>ть,</line>\n",
    "        <line lineNo=\"011\">Ему подушки поправл<stress>я</stress>ть,</line>\n",
    "        <line lineNo=\"012\">Печально подносить лек<stress>а</stress>рство,</line>\n",
    "        <line lineNo=\"013\">Вздыхать и думать про себ<stress>я</stress>:</line>\n",
    "        <line lineNo=\"014\">Когда же чорт возьмет теб<stress>я</stress>!\"</line>\n",
    "    </stanza>\n",
    "</poem>   \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reload libraries each time, since we’re tinkering with them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xml.dom import pulldom  # parse input XML\n",
    "from xml.dom.minidom import Document  # construct output XML\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "import regex as re\n",
    "from cyr2phon import cyr2phon  # custom package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class and variables for parsing input XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Stack(list):  # keep track of open nodes while constructing XML output\n",
    "    def push(self, item):\n",
    "        self.append(item)\n",
    "\n",
    "    def peek(self):  \n",
    "        return self[-1]\n",
    "\n",
    "\n",
    "open_elements = Stack()\n",
    "WS_RE = re.compile(r'\\s+')  # normalize white space in output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to parse the XML\n",
    "\n",
    "Returns a list of lists, with stanza number, line number, and `<line>` element for each line. We use the light-weight *xml.dom.pulldom* library to parse the input XML and *xml.dom.minidom* to construct the lines as simplified XML, removing elements we don’t care about, such as `<latin>` and `<italic>`, before serializing them to the output. (We actually do care about `<latin>`, but we are ignoring it temporarily, and we’ll return to it at a later stage in the project.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process(input_xml):\n",
    "    stanzaNo = 0\n",
    "    lineNo = 0\n",
    "    inline = 0  # flag to control behavior inside and outside lines\n",
    "    result = []  # array of arrays, one per line, with stanzaNo, lineNo, and serialized XML\n",
    "    doc = pulldom.parse(input_xml)\n",
    "    for event, node in doc:\n",
    "        if event == pulldom.START_ELEMENT and node.localName == 'stanza':\n",
    "            stanzaNo = node.getAttribute(\"stanzaNo\")\n",
    "        elif event == pulldom.START_ELEMENT and node.localName == 'line':\n",
    "            d = Document()  # each line is an output XML document\n",
    "            open_elements.push(d)  # document node\n",
    "            lineNo = node.getAttribute(\"lineNo\")\n",
    "            inline = 1  # we’re inside a line\n",
    "            open_elements.peek().appendChild(node)  # add as child of current node in output tree\n",
    "            open_elements.push(node)  # keep track of open elements\n",
    "        elif event == pulldom.END_ELEMENT and node.localName == 'line':\n",
    "            inline = 0  # when we finish our work here, we’ll no longer be inside a line\n",
    "            open_elements.pop()  # line is finished\n",
    "            # serialize XML, strip declaration, rewrite &quot; entity as character\n",
    "            result.append([int(stanzaNo), int(lineNo),\n",
    "                WS_RE.sub(\" \" ,\n",
    "                open_elements.pop().toxml().replace('<?xml version=\"1.0\" ?>', '').replace('&quot;', '\"'))])\n",
    "        elif event == pulldom.START_ELEMENT and node.localName == 'stress':\n",
    "            open_elements.peek().appendChild(node)  # add as child of current node in output tree\n",
    "            open_elements.push(node)  # keep track of open elements\n",
    "        elif event == pulldom.END_ELEMENT and node.localName == 'stress':\n",
    "            open_elements.pop()  # stress element is finished\n",
    "        elif event == pulldom.CHARACTERS and inline:  # keep text only inside lines\n",
    "            t = d.createTextNode(node.data)\n",
    "            open_elements.peek().appendChild(t)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse the XML into an array of arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1,\n",
       "  1,\n",
       "  '<line lineNo=\"001\">\"Мой дядя самых честных пр<stress>а</stress>вил,</line>'],\n",
       " [1,\n",
       "  2,\n",
       "  '<line lineNo=\"002\">Когда не в шутку занем<stress>о</stress>г,</line>'],\n",
       " [1, 3, '<line lineNo=\"003\">Он уважать себя заст<stress>а</stress>вил</line>'],\n",
       " [1, 4, '<line lineNo=\"004\">И лучше выдумать не м<stress>о</stress>г.</line>'],\n",
       " [1, 5, '<line lineNo=\"005\">Его пример другим на<stress>у</stress>ка;</line>']]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data = \"data_samples/eo1.xml\"\n",
    "data = \"../data/eo-all.xml\"\n",
    "with open(data) as f:\n",
    "    all_lines = process(f)\n",
    "all_lines[:5]  # take a look"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General descriptive information\n",
    "\n",
    "Use `//` for integer division to return *54* instead of *54.0*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 364 14-line stanzas in this sample, with a total of 5101 lines.\n",
      "Since we know that the poem is fully rhymed, there are 2550 rhyme pairs in the sample.\n"
     ]
    }
   ],
   "source": [
    "line_count = len(all_lines)\n",
    "print ('There are ' + str(line_count // 14) + ' 14-line stanzas in this sample, with a total of ' + \n",
    "       str(line_count) + ' lines.\\nSince we know that the poem is fully rhymed, there are ' + \n",
    "       str(line_count // 2) + ' rhyme pairs in the sample.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write the data into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;line lineNo=\"001\"&gt;\"Мой дядя самых честных пр&lt;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;line lineNo=\"002\"&gt;Когда не в шутку занем&lt;stre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;line lineNo=\"003\"&gt;Он уважать себя заст&lt;stress...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;line lineNo=\"004\"&gt;И лучше выдумать не м&lt;stres...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;line lineNo=\"005\"&gt;Его пример другим на&lt;stress...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   StanzaNo  LineNo                                               Text\n",
       "0         1       1  <line lineNo=\"001\">\"Мой дядя самых честных пр<...\n",
       "1         1       2  <line lineNo=\"002\">Когда не в шутку занем<stre...\n",
       "2         1       3  <line lineNo=\"003\">Он уважать себя заст<stress...\n",
       "3         1       4  <line lineNo=\"004\">И лучше выдумать не м<stres...\n",
       "4         1       5  <line lineNo=\"005\">Его пример другим на<stress..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(all_lines, columns=[\"StanzaNo\", \"LineNo\", \"Text\"])\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transliterate all lines and save in new column\n",
    "\n",
    "1. Because only the last stress in the line is marked, the phonetic representation of all words except the last is incorrect. That doesn’t matter for the analysis of end rhyme.\n",
    "1. Words in foreign languages are not being treated specially, and are therefore usually phonetically incorrect. That *does* matter for the analysis of end rhyme. Deal with it later, first by excluding those lines (revise the XML parsing to record that information), and eventually by phoneticizing them correctly.\n",
    "1. The `transliterate()` function is part of the [custom *cyr2phon* package](cyr2phon/cyr2phon.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "      <th>Phonetic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;line lineNo=\"001\"&gt;\"Мой дядя самых честных пр&lt;...</td>\n",
       "      <td>maJ DiDi samix Čistnix prAVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;line lineNo=\"002\"&gt;Когда не в шутку занем&lt;stre...</td>\n",
       "      <td>kagda Nifšutku zaNimOk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;line lineNo=\"003\"&gt;Он уважать себя заст&lt;stress...</td>\n",
       "      <td>an uvažaT SiBi zastAVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;line lineNo=\"004\"&gt;И лучше выдумать не м&lt;stres...</td>\n",
       "      <td>iluČši vidumaT NimOk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;line lineNo=\"005\"&gt;Его пример другим на&lt;stress...</td>\n",
       "      <td>Jiva pRiMir druGim naUka</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   StanzaNo  LineNo                                               Text  \\\n",
       "0         1       1  <line lineNo=\"001\">\"Мой дядя самых честных пр<...   \n",
       "1         1       2  <line lineNo=\"002\">Когда не в шутку занем<stre...   \n",
       "2         1       3  <line lineNo=\"003\">Он уважать себя заст<stress...   \n",
       "3         1       4  <line lineNo=\"004\">И лучше выдумать не м<stres...   \n",
       "4         1       5  <line lineNo=\"005\">Его пример другим на<stress...   \n",
       "\n",
       "                        Phonetic  \n",
       "0  maJ DiDi samix Čistnix prAVil  \n",
       "1         kagda Nifšutku zaNimOk  \n",
       "2        an uvažaT SiBi zastAVil  \n",
       "3           iluČši vidumaT NimOk  \n",
       "4       Jiva pRiMir druGim naUka  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trans_vec = np.vectorize(cyr2phon.transliterate)\n",
    "df[\"Phonetic\"] = trans_vec(df[\"Text\"])\n",
    "df.head()  # take a look"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write the rhyme word into a new column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "      <th>Phonetic</th>\n",
       "      <th>RhymeWord</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;line lineNo=\"001\"&gt;\"Мой дядя самых честных пр&lt;...</td>\n",
       "      <td>maJ DiDi samix Čistnix prAVil</td>\n",
       "      <td>prAVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;line lineNo=\"002\"&gt;Когда не в шутку занем&lt;stre...</td>\n",
       "      <td>kagda Nifšutku zaNimOk</td>\n",
       "      <td>zaNimOk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;line lineNo=\"003\"&gt;Он уважать себя заст&lt;stress...</td>\n",
       "      <td>an uvažaT SiBi zastAVil</td>\n",
       "      <td>zastAVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;line lineNo=\"004\"&gt;И лучше выдумать не м&lt;stres...</td>\n",
       "      <td>iluČši vidumaT NimOk</td>\n",
       "      <td>NimOk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;line lineNo=\"005\"&gt;Его пример другим на&lt;stress...</td>\n",
       "      <td>Jiva pRiMir druGim naUka</td>\n",
       "      <td>naUka</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   StanzaNo  LineNo                                               Text  \\\n",
       "0         1       1  <line lineNo=\"001\">\"Мой дядя самых честных пр<...   \n",
       "1         1       2  <line lineNo=\"002\">Когда не в шутку занем<stre...   \n",
       "2         1       3  <line lineNo=\"003\">Он уважать себя заст<stress...   \n",
       "3         1       4  <line lineNo=\"004\">И лучше выдумать не м<stres...   \n",
       "4         1       5  <line lineNo=\"005\">Его пример другим на<stress...   \n",
       "\n",
       "                        Phonetic RhymeWord  \n",
       "0  maJ DiDi samix Čistnix prAVil    prAVil  \n",
       "1         kagda Nifšutku zaNimOk   zaNimOk  \n",
       "2        an uvažaT SiBi zastAVil  zastAVil  \n",
       "3           iluČši vidumaT NimOk     NimOk  \n",
       "4       Jiva pRiMir druGim naUka     naUka  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"RhymeWord\"] = df[\"Phonetic\"].str.split().str[-1] # clitics have already been joined\n",
    "df.head()  # take a look"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify rhyme zone and write into new column\n",
    "\n",
    "The *rhyme zone* is the portion of the line that participates in line end-rhyme. According to Russian rhyming conventions, the rhyme zone typically begins with the last stressed vowel of the line and continues until the end of the line. The one exception is that open masculine rhyme (that is, rhyme involving stress on a final syllable that ends in a vowel, e.g., **себя́** *[SiBA]*) also requires a *supporting consonant*, that is, it also requires that the consonants *before the stressed vowels* (not otherwise considered part of the rhyme zone* also agree. For example:\n",
    "\n",
    "* _see_ and _tree_ do not rhyme in Russian because this open (ends in a vowel sound) masculine (stress on the final syllable) rhyme does not have a supporting consonant (consonants before the stressed vowels do not agree).\n",
    "* *seat* and *treat* do rhyme in Russian because closed (ends in a consonant sound) masculine (stress on the final syllable) rhyme does not require a supporting consonant, so the lack of phonetic correspondence between the consonants before the stressed vowels does not matter.\n",
    "\n",
    "Russian rhyme may also be *enriched* by phonetic agreement or similarity outside the rhyme zone. For example, *stop* and *strop* constitute a perfect rhyme because the *op* sounds match. Nonetheless, the match of *st* before the rhyme zone enhances, or enriches, the rhyme. The present study ignores enrichment and concentrates only on the core rhyme components, but enrichment will be incorporated into the analysis at a later stage.\n",
    "\n",
    "With that said, this first pass at identifying the rhyme zone removes the pretonic segments, but not the final consonant of a pretonic onset where a supporting consonant is not needed. More cleaning to follow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iaDivalSi\n",
      "têti-à-tÊti\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "      <th>Phonetic</th>\n",
       "      <th>RhymeWord</th>\n",
       "      <th>RhymeZone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;line lineNo=\"001\"&gt;\"Мой дядя самых честных пр&lt;...</td>\n",
       "      <td>maJ DiDi samix Čistnix prAVil</td>\n",
       "      <td>prAVil</td>\n",
       "      <td>AVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;line lineNo=\"002\"&gt;Когда не в шутку занем&lt;stre...</td>\n",
       "      <td>kagda Nifšutku zaNimOk</td>\n",
       "      <td>zaNimOk</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;line lineNo=\"003\"&gt;Он уважать себя заст&lt;stress...</td>\n",
       "      <td>an uvažaT SiBi zastAVil</td>\n",
       "      <td>zastAVil</td>\n",
       "      <td>AVil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;line lineNo=\"004\"&gt;И лучше выдумать не м&lt;stres...</td>\n",
       "      <td>iluČši vidumaT NimOk</td>\n",
       "      <td>NimOk</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;line lineNo=\"005\"&gt;Его пример другим на&lt;stress...</td>\n",
       "      <td>Jiva pRiMir druGim naUka</td>\n",
       "      <td>naUka</td>\n",
       "      <td>Uka</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;line lineNo=\"006\"&gt;Но, боже мой, какая ск&lt;stre...</td>\n",
       "      <td>nabaži maJ kakaJi skUka</td>\n",
       "      <td>skUka</td>\n",
       "      <td>Uka</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>&lt;line lineNo=\"007\"&gt;С больным сидеть и день и н...</td>\n",
       "      <td>zbaLnim SiDiT iDiN inOČ</td>\n",
       "      <td>inOČ</td>\n",
       "      <td>OČ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>&lt;line lineNo=\"008\"&gt;Не отходя ни шагу пр&lt;stress...</td>\n",
       "      <td>NiatxaDi Nišagu prOČ</td>\n",
       "      <td>prOČ</td>\n",
       "      <td>OČ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>&lt;line lineNo=\"009\"&gt;Какое низкое ков&lt;stress&gt;а&lt;/...</td>\n",
       "      <td>kakaJi NiskaJi kavArstva</td>\n",
       "      <td>kavArstva</td>\n",
       "      <td>Arstva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>&lt;line lineNo=\"010\"&gt;Полу-живого забавл&lt;stress&gt;я...</td>\n",
       "      <td>palu-živava zabavLAT</td>\n",
       "      <td>zabavLAT</td>\n",
       "      <td>AT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>&lt;line lineNo=\"011\"&gt;Ему подушки поправл&lt;stress&gt;...</td>\n",
       "      <td>Jimu padušKi papravLAT</td>\n",
       "      <td>papravLAT</td>\n",
       "      <td>AT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>&lt;line lineNo=\"012\"&gt;Печально подносить лек&lt;stre...</td>\n",
       "      <td>PiČiLna padnaSiT LikArstva</td>\n",
       "      <td>LikArstva</td>\n",
       "      <td>Arstva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>&lt;line lineNo=\"013\"&gt;Вздыхать и думать про себ&lt;s...</td>\n",
       "      <td>vzdixaT idumaT praSiBA</td>\n",
       "      <td>praSiBA</td>\n",
       "      <td>BA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>&lt;line lineNo=\"014\"&gt;Когда же чорт возьмет теб&lt;s...</td>\n",
       "      <td>kagdaži Čirt vaZMit TiBA</td>\n",
       "      <td>TiBA</td>\n",
       "      <td>BA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    StanzaNo  LineNo                                               Text  \\\n",
       "0          1       1  <line lineNo=\"001\">\"Мой дядя самых честных пр<...   \n",
       "1          1       2  <line lineNo=\"002\">Когда не в шутку занем<stre...   \n",
       "2          1       3  <line lineNo=\"003\">Он уважать себя заст<stress...   \n",
       "3          1       4  <line lineNo=\"004\">И лучше выдумать не м<stres...   \n",
       "4          1       5  <line lineNo=\"005\">Его пример другим на<stress...   \n",
       "5          1       6  <line lineNo=\"006\">Но, боже мой, какая ск<stre...   \n",
       "6          1       7  <line lineNo=\"007\">С больным сидеть и день и н...   \n",
       "7          1       8  <line lineNo=\"008\">Не отходя ни шагу пр<stress...   \n",
       "8          1       9  <line lineNo=\"009\">Какое низкое ков<stress>а</...   \n",
       "9          1      10  <line lineNo=\"010\">Полу-живого забавл<stress>я...   \n",
       "10         1      11  <line lineNo=\"011\">Ему подушки поправл<stress>...   \n",
       "11         1      12  <line lineNo=\"012\">Печально подносить лек<stre...   \n",
       "12         1      13  <line lineNo=\"013\">Вздыхать и думать про себ<s...   \n",
       "13         1      14  <line lineNo=\"014\">Когда же чорт возьмет теб<s...   \n",
       "\n",
       "                         Phonetic  RhymeWord RhymeZone  \n",
       "0   maJ DiDi samix Čistnix prAVil     prAVil      AVil  \n",
       "1          kagda Nifšutku zaNimOk    zaNimOk        Ok  \n",
       "2         an uvažaT SiBi zastAVil   zastAVil      AVil  \n",
       "3            iluČši vidumaT NimOk      NimOk        Ok  \n",
       "4        Jiva pRiMir druGim naUka      naUka       Uka  \n",
       "5         nabaži maJ kakaJi skUka      skUka       Uka  \n",
       "6         zbaLnim SiDiT iDiN inOČ       inOČ        OČ  \n",
       "7            NiatxaDi Nišagu prOČ       prOČ        OČ  \n",
       "8        kakaJi NiskaJi kavArstva  kavArstva    Arstva  \n",
       "9            palu-živava zabavLAT   zabavLAT        AT  \n",
       "10         Jimu padušKi papravLAT  papravLAT        AT  \n",
       "11     PiČiLna padnaSiT LikArstva  LikArstva    Arstva  \n",
       "12         vzdixaT idumaT praSiBA    praSiBA        BA  \n",
       "13       kagdaži Čirt vaZMit TiBA       TiBA        BA  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rhymezonepat = re.compile(r'(.?[AEIOU]$)|([AEIOU].*$)')\n",
    "def remove_pretonic_segments(s: str) -> str: # removes segments in place\n",
    "    try:\n",
    "        return rhymezonepat.search(s).group(0)\n",
    "    except: # modify this to raise a real error, instead of just reporting\n",
    "        print(s)\n",
    "df[\"RhymeZone\"] = df[\"RhymeWord\"].apply(remove_pretonic_segments)\n",
    "df.head(14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize rhyme zone into C(C) and V\n",
    "\n",
    "We may have some null values in the rhyme zone for incomplete lines that have no final stressed syllable (*EO* 37: 13) and foreign words that we may not be handling properly (*EO* 23:2). Check for those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "      <th>Phonetic</th>\n",
       "      <th>RhymeWord</th>\n",
       "      <th>RhymeZone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2302</th>\n",
       "      <td>37</td>\n",
       "      <td>13</td>\n",
       "      <td>&lt;line lineNo=\"013\"&gt;И одевался...&lt;/line&gt;</td>\n",
       "      <td>iaDivalSi</td>\n",
       "      <td>iaDivalSi</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4702</th>\n",
       "      <td>23</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;line lineNo=\"002\"&gt;Сей неприятный tête-à-t&lt;str...</td>\n",
       "      <td>SiJ NipRiJitniJ têti-à-tÊti</td>\n",
       "      <td>têti-à-tÊti</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      StanzaNo  LineNo                                               Text  \\\n",
       "2302        37      13            <line lineNo=\"013\">И одевался...</line>   \n",
       "4702        23       2  <line lineNo=\"002\">Сей неприятный tête-à-t<str...   \n",
       "\n",
       "                         Phonetic    RhymeWord RhymeZone  \n",
       "2302                    iaDivalSi    iaDivalSi      None  \n",
       "4702  SiJ NipRiJitniJ têti-à-tÊti  têti-à-tÊti      None  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"RhymeZone\"].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter them out provisionally by writing in a placeholder value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>Text</th>\n",
       "      <th>Phonetic</th>\n",
       "      <th>RhymeWord</th>\n",
       "      <th>RhymeZone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [StanzaNo, LineNo, Text, Phonetic, RhymeWord, RhymeZone]\n",
       "Index: []"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df[\"RhymeZone\"].isnull(), \"RhymeZone\"] = \"Abcde\"\n",
    "df[df[\"RhymeZone\"] == \"Aplaceholder\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StanzaNo</th>\n",
       "      <th>LineNo</th>\n",
       "      <th>RhymeWord</th>\n",
       "      <th>token0</th>\n",
       "      <th>token1</th>\n",
       "      <th>token2</th>\n",
       "      <th>token3</th>\n",
       "      <th>token4</th>\n",
       "      <th>token5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>prAVil</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>V</td>\n",
       "      <td>i</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>zaNimOk</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>k</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>zastAVil</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>V</td>\n",
       "      <td>i</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>NimOk</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>k</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>naUka</td>\n",
       "      <td>missing</td>\n",
       "      <td>U</td>\n",
       "      <td>k</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>skUka</td>\n",
       "      <td>missing</td>\n",
       "      <td>U</td>\n",
       "      <td>k</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>inOČ</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>Č</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>prOČ</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>Č</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>kavArstva</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>rstv</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>zabavLAT</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>T</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>papravLAT</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>T</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>LikArstva</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>rstv</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>praSiBA</td>\n",
       "      <td>B</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>TiBA</td>\n",
       "      <td>B</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>paVEsa</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>napaČtavIx</td>\n",
       "      <td>missing</td>\n",
       "      <td>I</td>\n",
       "      <td>x</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>ZiVEsa</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>radnIx</td>\n",
       "      <td>missing</td>\n",
       "      <td>I</td>\n",
       "      <td>x</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>iruslAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>ramAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>ČAs</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>s</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>vAs</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>s</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>pRiJATiL</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>T</td>\n",
       "      <td>i</td>\n",
       "      <td>L</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>NivI</td>\n",
       "      <td>v</td>\n",
       "      <td>I</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>vI</td>\n",
       "      <td>v</td>\n",
       "      <td>I</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>ČitATiL</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>T</td>\n",
       "      <td>i</td>\n",
       "      <td>L</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>iJA</td>\n",
       "      <td>J</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>DLiMiNA</td>\n",
       "      <td>N</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>aTLiČna-blagarOdna</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>dn</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>aTEts</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>ts</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5071</th>\n",
       "      <td>49</td>\n",
       "      <td>13</td>\n",
       "      <td>naJTI</td>\n",
       "      <td>T</td>\n",
       "      <td>I</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5072</th>\n",
       "      <td>49</td>\n",
       "      <td>14</td>\n",
       "      <td>praSTI</td>\n",
       "      <td>T</td>\n",
       "      <td>I</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5073</th>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>strAniJ</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>i</td>\n",
       "      <td>J</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5074</th>\n",
       "      <td>50</td>\n",
       "      <td>2</td>\n",
       "      <td>iDiAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5075</th>\n",
       "      <td>50</td>\n",
       "      <td>3</td>\n",
       "      <td>ipastaJAniJ</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>i</td>\n",
       "      <td>J</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5076</th>\n",
       "      <td>50</td>\n",
       "      <td>4</td>\n",
       "      <td>znAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5077</th>\n",
       "      <td>50</td>\n",
       "      <td>5</td>\n",
       "      <td>DLipaEta</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>t</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5078</th>\n",
       "      <td>50</td>\n",
       "      <td>6</td>\n",
       "      <td>sVEta</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>t</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5079</th>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>druZEJ</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>J</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5080</th>\n",
       "      <td>50</td>\n",
       "      <td>8</td>\n",
       "      <td>DNEJ</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>J</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5081</th>\n",
       "      <td>50</td>\n",
       "      <td>9</td>\n",
       "      <td>taTJAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5082</th>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>SNE</td>\n",
       "      <td>N</td>\n",
       "      <td>E</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5083</th>\n",
       "      <td>50</td>\n",
       "      <td>11</td>\n",
       "      <td>mNE</td>\n",
       "      <td>N</td>\n",
       "      <td>E</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5084</th>\n",
       "      <td>50</td>\n",
       "      <td>12</td>\n",
       "      <td>ramAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5085</th>\n",
       "      <td>50</td>\n",
       "      <td>13</td>\n",
       "      <td>kRistAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5086</th>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>raZLiČAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5087</th>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "      <td>fstREČi</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>Č</td>\n",
       "      <td>i</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5088</th>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>ČitAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5089</th>\n",
       "      <td>51</td>\n",
       "      <td>3</td>\n",
       "      <td>daLEČi</td>\n",
       "      <td>missing</td>\n",
       "      <td>E</td>\n",
       "      <td>Č</td>\n",
       "      <td>i</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5090</th>\n",
       "      <td>51</td>\n",
       "      <td>4</td>\n",
       "      <td>skazAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5091</th>\n",
       "      <td>51</td>\n",
       "      <td>5</td>\n",
       "      <td>daRisOvan</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>v</td>\n",
       "      <td>a</td>\n",
       "      <td>n</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5092</th>\n",
       "      <td>51</td>\n",
       "      <td>6</td>\n",
       "      <td>abrazOvan</td>\n",
       "      <td>missing</td>\n",
       "      <td>O</td>\n",
       "      <td>v</td>\n",
       "      <td>a</td>\n",
       "      <td>n</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5093</th>\n",
       "      <td>51</td>\n",
       "      <td>7</td>\n",
       "      <td>iDiAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5094</th>\n",
       "      <td>51</td>\n",
       "      <td>8</td>\n",
       "      <td>aTJAl</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>l</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5095</th>\n",
       "      <td>51</td>\n",
       "      <td>9</td>\n",
       "      <td>rAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5096</th>\n",
       "      <td>51</td>\n",
       "      <td>10</td>\n",
       "      <td>dadnA</td>\n",
       "      <td>n</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5097</th>\n",
       "      <td>51</td>\n",
       "      <td>11</td>\n",
       "      <td>VinA</td>\n",
       "      <td>n</td>\n",
       "      <td>A</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5098</th>\n",
       "      <td>51</td>\n",
       "      <td>12</td>\n",
       "      <td>ramAna</td>\n",
       "      <td>missing</td>\n",
       "      <td>A</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5099</th>\n",
       "      <td>51</td>\n",
       "      <td>13</td>\n",
       "      <td>SNIm</td>\n",
       "      <td>missing</td>\n",
       "      <td>I</td>\n",
       "      <td>m</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5100</th>\n",
       "      <td>51</td>\n",
       "      <td>14</td>\n",
       "      <td>maJIm</td>\n",
       "      <td>missing</td>\n",
       "      <td>I</td>\n",
       "      <td>m</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "      <td>missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5101 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      StanzaNo  LineNo           RhymeWord   token0 token1   token2   token3  \\\n",
       "0            1       1              prAVil  missing      A        V        i   \n",
       "1            1       2             zaNimOk  missing      O        k  missing   \n",
       "2            1       3            zastAVil  missing      A        V        i   \n",
       "3            1       4               NimOk  missing      O        k  missing   \n",
       "4            1       5               naUka  missing      U        k        a   \n",
       "5            1       6               skUka  missing      U        k        a   \n",
       "6            1       7                inOČ  missing      O        Č  missing   \n",
       "7            1       8                prOČ  missing      O        Č  missing   \n",
       "8            1       9           kavArstva  missing      A     rstv        a   \n",
       "9            1      10            zabavLAT  missing      A        T  missing   \n",
       "10           1      11           papravLAT  missing      A        T  missing   \n",
       "11           1      12           LikArstva  missing      A     rstv        a   \n",
       "12           1      13             praSiBA        B      A  missing  missing   \n",
       "13           1      14                TiBA        B      A  missing  missing   \n",
       "14           2       1              paVEsa  missing      E        s        a   \n",
       "15           2       2          napaČtavIx  missing      I        x  missing   \n",
       "16           2       3              ZiVEsa  missing      E        s        a   \n",
       "17           2       4              radnIx  missing      I        x  missing   \n",
       "18           2       5            iruslAna  missing      A        n        a   \n",
       "19           2       6              ramAna  missing      A        n        a   \n",
       "20           2       7                 ČAs  missing      A        s  missing   \n",
       "21           2       8                 vAs  missing      A        s  missing   \n",
       "22           2       9            pRiJATiL  missing      A        T        i   \n",
       "23           2      10                NivI        v      I  missing  missing   \n",
       "24           2      11                  vI        v      I  missing  missing   \n",
       "25           2      12             ČitATiL  missing      A        T        i   \n",
       "26           2      13                 iJA        J      A  missing  missing   \n",
       "27           2      14             DLiMiNA        N      A  missing  missing   \n",
       "28           3       1  aTLiČna-blagarOdna  missing      O       dn        a   \n",
       "29           3       2               aTEts  missing      E       ts  missing   \n",
       "...        ...     ...                 ...      ...    ...      ...      ...   \n",
       "5071        49      13               naJTI        T      I  missing  missing   \n",
       "5072        49      14              praSTI        T      I  missing  missing   \n",
       "5073        50       1             strAniJ  missing      A        n        i   \n",
       "5074        50       2               iDiAl  missing      A        l  missing   \n",
       "5075        50       3         ipastaJAniJ  missing      A        n        i   \n",
       "5076        50       4                znAl  missing      A        l  missing   \n",
       "5077        50       5            DLipaEta  missing      E        t        a   \n",
       "5078        50       6               sVEta  missing      E        t        a   \n",
       "5079        50       7              druZEJ  missing      E        J  missing   \n",
       "5080        50       8                DNEJ  missing      E        J  missing   \n",
       "5081        50       9             taTJAna  missing      A        n        a   \n",
       "5082        50      10                 SNE        N      E  missing  missing   \n",
       "5083        50      11                 mNE        N      E  missing  missing   \n",
       "5084        50      12              ramAna  missing      A        n        a   \n",
       "5085        50      13             kRistAl  missing      A        l  missing   \n",
       "5086        50      14            raZLiČAl  missing      A        l  missing   \n",
       "5087        51       1             fstREČi  missing      E        Č        i   \n",
       "5088        51       2               ČitAl  missing      A        l  missing   \n",
       "5089        51       3              daLEČi  missing      E        Č        i   \n",
       "5090        51       4              skazAl  missing      A        l  missing   \n",
       "5091        51       5           daRisOvan  missing      O        v        a   \n",
       "5092        51       6           abrazOvan  missing      O        v        a   \n",
       "5093        51       7               iDiAl  missing      A        l  missing   \n",
       "5094        51       8               aTJAl  missing      A        l  missing   \n",
       "5095        51       9                rAna  missing      A        n        a   \n",
       "5096        51      10               dadnA        n      A  missing  missing   \n",
       "5097        51      11                VinA        n      A  missing  missing   \n",
       "5098        51      12              ramAna  missing      A        n        a   \n",
       "5099        51      13                SNIm  missing      I        m  missing   \n",
       "5100        51      14               maJIm  missing      I        m  missing   \n",
       "\n",
       "       token4   token5  \n",
       "0           l  missing  \n",
       "1     missing  missing  \n",
       "2           l  missing  \n",
       "3     missing  missing  \n",
       "4     missing  missing  \n",
       "5     missing  missing  \n",
       "6     missing  missing  \n",
       "7     missing  missing  \n",
       "8     missing  missing  \n",
       "9     missing  missing  \n",
       "10    missing  missing  \n",
       "11    missing  missing  \n",
       "12    missing  missing  \n",
       "13    missing  missing  \n",
       "14    missing  missing  \n",
       "15    missing  missing  \n",
       "16    missing  missing  \n",
       "17    missing  missing  \n",
       "18    missing  missing  \n",
       "19    missing  missing  \n",
       "20    missing  missing  \n",
       "21    missing  missing  \n",
       "22          L  missing  \n",
       "23    missing  missing  \n",
       "24    missing  missing  \n",
       "25          L  missing  \n",
       "26    missing  missing  \n",
       "27    missing  missing  \n",
       "28    missing  missing  \n",
       "29    missing  missing  \n",
       "...       ...      ...  \n",
       "5071  missing  missing  \n",
       "5072  missing  missing  \n",
       "5073        J  missing  \n",
       "5074  missing  missing  \n",
       "5075        J  missing  \n",
       "5076  missing  missing  \n",
       "5077  missing  missing  \n",
       "5078  missing  missing  \n",
       "5079  missing  missing  \n",
       "5080  missing  missing  \n",
       "5081  missing  missing  \n",
       "5082  missing  missing  \n",
       "5083  missing  missing  \n",
       "5084  missing  missing  \n",
       "5085  missing  missing  \n",
       "5086  missing  missing  \n",
       "5087  missing  missing  \n",
       "5088  missing  missing  \n",
       "5089  missing  missing  \n",
       "5090  missing  missing  \n",
       "5091        n  missing  \n",
       "5092        n  missing  \n",
       "5093  missing  missing  \n",
       "5094  missing  missing  \n",
       "5095  missing  missing  \n",
       "5096  missing  missing  \n",
       "5097  missing  missing  \n",
       "5098  missing  missing  \n",
       "5099  missing  missing  \n",
       "5100  missing  missing  \n",
       "\n",
       "[5101 rows x 9 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"tokenized\"] = [x[0] for x in df[\"RhymeZone\"].str.\n",
    "                   findall(r\"(.?)([AEIOU])([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)([^aeiou]*)([aeiou]?)\")]\n",
    "i = 0\n",
    "while pd.np.count_nonzero([item[i] for item in df[\"tokenized\"]]) > 0:\n",
    "    # print([item[i] for item in df[\"tokenized\"]]) # diagnostic\n",
    "    df[\"token\" + str(i)] = [item[i] for item in df[\"tokenized\"]]\n",
    "    i += 1\n",
    "tokenheaders = list([item for item in df.columns if re.match(r'token\\d', item)])\n",
    "df[tokenheaders] = df[tokenheaders].replace(r'^$', \"missing\", regex=True) # replace empty strings with specific value; inplace doesn't work (?)\n",
    "df.filter(regex=r\"StanzaNo|LineNo|RhymeWord|^token\\d\") # columns we care about"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A bit of exploration\n",
    "\n",
    "### What values appear in which columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token0\n",
      "missing    4207\n",
      "n           204\n",
      "J            88\n",
      "v            70\n",
      "N            62\n",
      "l            52\n",
      "r            50\n",
      "t            39\n",
      "s            33\n",
      "d            33\n",
      "m            32\n",
      "k            30\n",
      "K            28\n",
      "L            23\n",
      "T            20\n",
      "š            19\n",
      "B            19\n",
      "R            16\n",
      "g            12\n",
      "Č            10\n",
      "ž            10\n",
      "S             9\n",
      "V             8\n",
      "X             6\n",
      "M             5\n",
      "D             4\n",
      "Q             4\n",
      "x             4\n",
      "b             2\n",
      "z             2\n",
      "dtype: int64\n",
      "token1\n",
      "A    1482\n",
      "E    1265\n",
      "O    1184\n",
      "I     834\n",
      "U     336\n",
      "dtype: int64\n",
      "token2\n",
      "missing    900\n",
      "J          618\n",
      "t          431\n",
      "l          359\n",
      "n          314\n",
      "m          226\n",
      "N          178\n",
      "NJ         154\n",
      "T          154\n",
      "k          150\n",
      "L          111\n",
      "r          108\n",
      "f          106\n",
      "d           98\n",
      "v           90\n",
      "x           90\n",
      "s           55\n",
      "ts          54\n",
      "D           46\n",
      "g           44\n",
      "ž           42\n",
      "S           40\n",
      "M           40\n",
      "žn          38\n",
      "R           38\n",
      "rn          30\n",
      "Č           30\n",
      "dn          30\n",
      "V           28\n",
      "ST          26\n",
      "          ... \n",
      "fš           4\n",
      "ZN           4\n",
      "žD           4\n",
      "tstv         2\n",
      "zdn          2\n",
      "vN           2\n",
      "št           2\n",
      "žN           2\n",
      "RJ           2\n",
      "pn           2\n",
      "tr           2\n",
      "Jsk          2\n",
      "STJ          2\n",
      "SČ           2\n",
      "TJ           2\n",
      "VJ           2\n",
      "X            2\n",
      "bcd          2\n",
      "bn           2\n",
      "P            2\n",
      "fts          2\n",
      "LČ           2\n",
      "Lts          2\n",
      "lK           2\n",
      "ln           2\n",
      "Ld           2\n",
      "nsK          2\n",
      "pK           2\n",
      "rstv         2\n",
      "-s           1\n",
      "Length: 100, dtype: int64\n",
      "token3\n",
      "missing    2907\n",
      "i          1280\n",
      "a           759\n",
      "u           153\n",
      "e             2\n",
      "dtype: int64\n",
      "token4\n",
      "missing    4207\n",
      "J           391\n",
      "t           145\n",
      "m           124\n",
      "S            38\n",
      "x            32\n",
      "T            31\n",
      "ST           30\n",
      "n            30\n",
      "k            24\n",
      "L            20\n",
      "l            12\n",
      "š             8\n",
      "f             6\n",
      "N             2\n",
      "s             1\n",
      "dtype: int64\n",
      "token5\n",
      "missing    5100\n",
      "i             1\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "for i in range(int(df.columns[-1][-1]) + 1): # number value of last token column\n",
    "    print(df.groupby(\"token\" + str(i)).size().nlargest(1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is this what we expect?\n",
    "\n",
    "We can’t compare sound frequencies directly to letter frequencies and we don’t have fully stressed texts (either our own poetry or a standard reference corpus), so we can’t can’t derive the pronunciation from the orthography. But we can check whether the frequency of stressed *[U]* (or unstressed) in our poetry corpus matches the frequency of the letters **у** and __ю__ in the general reference corpus, since those vowel letters, uniquely, do not participate as either sources or targets of vowel reduction. Let’s checked stressed *[U]*\n",
    "\n",
    "#### Frequency of *[u]* in Russian in general"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06897369044797344"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from http://practicalcryptography.com/cryptanalysis/letter-frequencies-various-languages/russian-letter-frequencies/\n",
    "frequencies = pd.Series({\n",
    "'А' :  8.04,\n",
    "'О' : 10.61,\n",
    "'Е' :  8.21,\n",
    "'Ё' :  0.22,\n",
    "'Ы' :  1.91,\n",
    "'И' :  7.98,\n",
    "'Э' :  0.31,\n",
    "'У' :  2.28,\n",
    "'Ю' :  0.63,\n",
    "'Я' :  2.00\n",
    "})\n",
    "all_total = frequencies.sum()\n",
    "all_u = frequencies[['У', 'Ю']].sum()\n",
    "all_u_freq = all_u / all_total\n",
    "all_u_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Frequency in our corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06586943736522251"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_total = len(df) # all lines have stressed vowels\n",
    "corpus_u = len(df[df[\"token1\"].isin([\"U\"])])\n",
    "corpus_u_freq = corpus_u / corpus_total\n",
    "corpus_u_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Is it significant?\n",
    "\n",
    "Use a binomial test (<https://en.wikipedia.org/wiki/Binomial_test>, <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom_test.html>):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.39184733383988657"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.binom_test(corpus_u, corpus_total, all_u_freq) # successes, trials, probability of success "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks significant from here! The distribution in our corpus is not inconsistent with what we’d expect from a Russian text.\n",
    "\n",
    "We could similarly test the few other places where letters have consistent pronunciation: nasals and liquids. Oral obstruents are all subject to voicing adjustments, and we can’t do anything useful with vowels other than *[u]* because we don’t have stress information for a general corpus and vowels other than *[u]* may be either sources or targets of vowel reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}