english_level_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc1910fa-1797-40cd-ae7d-ca33c2916f2b",
   "metadata": {},
   "source": [
    "# English Level (dataset formation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e8b756-4423-4544-ace2-51bebe6fac14",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "111c7f3a-0bfd-4e7a-bea3-6f887febdffe",
   "metadata": {},
   "source": [
    "**Yandex.Practicum's English department** is a customer for this project https://practicum.yandex.ru/english/\n",
    "\n",
    "One of the most effective ways to study foreighn languages (including English) is to watch movies. It's considered it's best for the student to watch videos in which they can understand from 50% to 70% of all dialogs in order to maximise their learning rate. Thus it's vital how movie contents matches an English level. We will use **CEFR** to define English level.\n",
    "\n",
    "A dataset containig information on some movies' English level is provided by Yandex.Prackticum experts.\n",
    "\n",
    "**Objective** is to build a model that can evaluate English level of movies based on their subtitles content."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4bb5df5-63a7-4371-8b8b-89a6b9f8d40e",
   "metadata": {},
   "source": [
    "This project is devided on three notebooks:\n",
    "* `english_level_dataset.ipynb`: forms a dataset from all the data and saves it into `text_labels.csv` file\n",
    "* `english_level_modeling.ipynb`: takes `text_labels.csv` file, does text processing, modeling and saves model `english_labels_model.pkl` file\n",
    "* `english_level_servise.ipynb`: allows to label provided `.srt` file using the saved model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606b9822-0759-4b71-ae72-bb0774f63478",
   "metadata": {},
   "source": [
    "### Data provided"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135da4be-fe99-4a66-9ee8-ed39ba20a762",
   "metadata": {},
   "source": [
    "* A spreadsheet containig data on some movies' titles and English levels\n",
    "* Sets of labeled subtitles (`.srt` fromat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960d1568-ca74-48a0-9480-46ccca07d211",
   "metadata": {},
   "source": [
    "This notebook will process all the data to form a dataset that contains text data from `.srt` files and English level labels and save it into an `.csv` file for future training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df1350a-122f-4ef3-a807-b69523e17ff7",
   "metadata": {},
   "source": [
    "## Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45220ab7-6a0c-4b77-bbe1-017ccf79cf56",
   "metadata": {},
   "source": [
    "### Libraries and settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d90c6a3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries to work with data\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "import difflib\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "44f85caf-3743-4c7f-b2df-dd514e1f0936",
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries to work with files\n",
    "import os\n",
    "import pysrt # https://github.com/byroot/pysrt\n",
    "import chardet # detect encoding\n",
    "import codecs # decode files\n",
    "\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d9b2ec06-ea13-4827-99d7-3094a9fff220",
   "metadata": {},
   "outputs": [],
   "source": [
    "# global variables\n",
    "PATH_MOVIE_LABELS = r'./English_scores/movies_labels.xlsx' # movie titles and labels\n",
    "PATH_ALL_SUBS_FOLDER = r'./English_scores/Subtitles_all/Subtitles' # all unlabeled .srt files\n",
    "PATH_SUBS_A2 = r'./English_scores/Subtitles_all/A2' # .srt files labeled A2\n",
    "PATH_SUBS_B1 = r'./English_scores/Subtitles_all/B1' # .srt files labeled B1\n",
    "PATH_SUBS_B2 = r'./English_scores/Subtitles_all/B2' # .srt files labeled B2\n",
    "PATH_SUBS_C1 = r'./English_scores/Subtitles_all/C1' # .srt files labeld C1\n",
    "UTF8_SUBFOLDER = r'/utf-8' # subfolder for .srt files re-encoded to utf-8 \n",
    "PATH_TRAIN_DATA = r'./text_labels.csv' # file for dataset\n",
    "RND_STATE = 1337 # random state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2228e00c-6876-48d5-9a51-0b05679c355c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# regex for text processing\n",
    "HTML = re.compile(r'<.*?>')\n",
    "TAG = re.compile(r'{.*?}')\n",
    "COMMENTS = re.compile(r'[\\(\\[][A-Z ]+[\\)\\]]')\n",
    "LETTERS = re.compile(r'[^a-zA-Z\\'.,!? ]')\n",
    "SPACES = re.compile(r'([ ])\\1+')\n",
    "DOTS = re.compile(r'[\\.]+')\n",
    "\n",
    "ONLY_WORDS = re.compile(r'[.,!?]|(?:\\'[a-z]*)') # for BOW"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2600dae2-3b38-4f16-848c-0223df9750ac",
   "metadata": {},
   "source": [
    "### Loading dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aac7cc06-6b1b-47c9-a6f5-8d9d9db3de72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Movie</th>\n",
       "      <th>Level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>10_Cloverfield_lane(2016)</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>10_things_I_hate_about_you(1999)</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>A_knights_tale(2001)</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>A_star_is_born(2018)</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Aladdin(1992)</td>\n",
       "      <td>A2/A2+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>236</td>\n",
       "      <td>Matilda(2022)</td>\n",
       "      <td>C1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>237</td>\n",
       "      <td>Bullet train</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>238</td>\n",
       "      <td>Thor: love and thunder</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>239</td>\n",
       "      <td>Lightyear</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>240</td>\n",
       "      <td>The Grinch</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>241 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                             Movie   Level\n",
       "0      0         10_Cloverfield_lane(2016)      B1\n",
       "1      1  10_things_I_hate_about_you(1999)      B1\n",
       "2      2              A_knights_tale(2001)      B2\n",
       "3      3              A_star_is_born(2018)      B2\n",
       "4      4                     Aladdin(1992)  A2/A2+\n",
       "..   ...                               ...     ...\n",
       "236  236                     Matilda(2022)      C1\n",
       "237  237                      Bullet train      B1\n",
       "238  238            Thor: love and thunder      B2\n",
       "239  239                         Lightyear      B2\n",
       "240  240                        The Grinch      B1\n",
       "\n",
       "[241 rows x 3 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# loading dataset\n",
    "movies_labels_df = pd.read_excel(PATH_MOVIE_LABELS)\n",
    "movies_labels_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1cca5a02-f23b-4714-b324-1fb669cc7fba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean the index\n",
    "movies_labels_df = movies_labels_df.drop(columns=['id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f3a6e6c3-14f2-4af6-a693-daa2156ad420",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rename columns\n",
    "movies_labels_df.columns = ['movie', 'label']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37024fb-cb1b-4087-bd35-978b489b12c1",
   "metadata": {},
   "source": [
    "As we see, `movie` contains not only a movie title but also a release year. We will probably need that data so I will extract it. Column `label` is target data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "417a7942-9fd2-4246-ad9e-6cdc4a68a13a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# movie names to lowercase\n",
    "movies_labels_df['movie_title'] = movies_labels_df['movie'].str.casefold()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c399e8c6-ea50-4d64-9114-23ea94a07d64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract year\n",
    "movies_labels_df['year'] = movies_labels_df['movie_title'].str.extract('((?<=\\()\\d\\d\\d\\d(?=\\)))')\n",
    "movies_labels_df['year'] = pd.to_numeric(movies_labels_df['year'], errors='coerce').astype('Int32')\n",
    "# remove year\n",
    "movies_labels_df['movie_title'] = movies_labels_df['movie_title'].apply(lambda x: re.sub('\\(((19)|(20))\\d\\d\\)', '', x))\n",
    "# change underscore to space\n",
    "movies_labels_df['movie_title'] = movies_labels_df['movie_title'].apply(lambda x: re.sub('_', ' ', x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "790c8c53-f96b-4472-bd75-8ffbcf1ded86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie</th>\n",
       "      <th>label</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10_Cloverfield_lane(2016)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 cloverfield lane</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10_things_I_hate_about_you(1999)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 things i hate about you</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A_knights_tale(2001)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a knights tale</td>\n",
       "      <td>2001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A_star_is_born(2018)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a star is born</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aladdin(1992)</td>\n",
       "      <td>A2/A2+</td>\n",
       "      <td>aladdin</td>\n",
       "      <td>1992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>Matilda(2022)</td>\n",
       "      <td>C1</td>\n",
       "      <td>matilda</td>\n",
       "      <td>2022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>Bullet train</td>\n",
       "      <td>B1</td>\n",
       "      <td>bullet train</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>Thor: love and thunder</td>\n",
       "      <td>B2</td>\n",
       "      <td>thor: love and thunder</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>Lightyear</td>\n",
       "      <td>B2</td>\n",
       "      <td>lightyear</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>The Grinch</td>\n",
       "      <td>B1</td>\n",
       "      <td>the grinch</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>241 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                movie   label                 movie_title  \\\n",
       "0           10_Cloverfield_lane(2016)      B1         10 cloverfield lane   \n",
       "1    10_things_I_hate_about_you(1999)      B1  10 things i hate about you   \n",
       "2                A_knights_tale(2001)      B2              a knights tale   \n",
       "3                A_star_is_born(2018)      B2              a star is born   \n",
       "4                       Aladdin(1992)  A2/A2+                     aladdin   \n",
       "..                                ...     ...                         ...   \n",
       "236                     Matilda(2022)      C1                     matilda   \n",
       "237                      Bullet train      B1                bullet train   \n",
       "238            Thor: love and thunder      B2      thor: love and thunder   \n",
       "239                         Lightyear      B2                   lightyear   \n",
       "240                        The Grinch      B1                  the grinch   \n",
       "\n",
       "     year  \n",
       "0    2016  \n",
       "1    1999  \n",
       "2    2001  \n",
       "3    2018  \n",
       "4    1992  \n",
       "..    ...  \n",
       "236  2022  \n",
       "237  <NA>  \n",
       "238  <NA>  \n",
       "239  <NA>  \n",
       "240  <NA>  \n",
       "\n",
       "[241 rows x 4 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_labels_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4d5392e2-4ea4-4cd5-9deb-15bdd02fd6ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 241 entries, 0 to 240\n",
      "Data columns (total 4 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   movie        241 non-null    object\n",
      " 1   label        241 non-null    object\n",
      " 2   movie_title  241 non-null    object\n",
      " 3   year         108 non-null    Int32 \n",
      "dtypes: Int32(1), object(3)\n",
      "memory usage: 7.0+ KB\n"
     ]
    }
   ],
   "source": [
    "movies_labels_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b656bcfb-afc7-4ccc-a168-7e203025d436",
   "metadata": {},
   "source": [
    "Lets' check for duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "79ee3e4d-424d-49c2-acfe-9ce6cefdfbea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie</th>\n",
       "      <th>label</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Inside_out(2015)</td>\n",
       "      <td>B1</td>\n",
       "      <td>inside out</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Inside_out(2015)</td>\n",
       "      <td>B1</td>\n",
       "      <td>inside out</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>Matilda(1996)</td>\n",
       "      <td>B1</td>\n",
       "      <td>matilda</td>\n",
       "      <td>1996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>Matilda(2022)</td>\n",
       "      <td>C1</td>\n",
       "      <td>matilda</td>\n",
       "      <td>2022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Powder(1995)</td>\n",
       "      <td>B1</td>\n",
       "      <td>powder</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Powder(1995)</td>\n",
       "      <td>B1</td>\n",
       "      <td>powder</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>The_blind_side(2009)</td>\n",
       "      <td>B2</td>\n",
       "      <td>the blind side</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>The_blind_side(2009)</td>\n",
       "      <td>B1</td>\n",
       "      <td>the blind side</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>The_terminal(2004)</td>\n",
       "      <td>B1</td>\n",
       "      <td>the terminal</td>\n",
       "      <td>2004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>The_terminal(2004)</td>\n",
       "      <td>A2/A2+, B1</td>\n",
       "      <td>the terminal</td>\n",
       "      <td>2004</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    movie       label     movie_title  year\n",
       "43       Inside_out(2015)          B1      inside out  2015\n",
       "44       Inside_out(2015)          B1      inside out  2015\n",
       "56          Matilda(1996)          B1         matilda  1996\n",
       "236         Matilda(2022)          C1         matilda  2022\n",
       "38           Powder(1995)          B1          powder  1995\n",
       "68           Powder(1995)          B1          powder  1995\n",
       "75   The_blind_side(2009)          B2  the blind side  2009\n",
       "84   The_blind_side(2009)          B1  the blind side  2009\n",
       "83     The_terminal(2004)          B1    the terminal  2004\n",
       "99     The_terminal(2004)  A2/A2+, B1    the terminal  2004"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_labels_df[movies_labels_df['movie_title'].duplicated(keep=False)].sort_values(by='movie_title')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d83461-610a-468d-8367-ab95af27459d",
   "metadata": {},
   "source": [
    "Some movies are duplicated and have different English level rating. What should I do with them?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8377ba3-6684-4486-b886-cee452f537ff",
   "metadata": {},
   "source": [
    "## Forming a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c75382-6ca5-4079-a271-a99290238d0a",
   "metadata": {},
   "source": [
    "We need to form a dataset that contains text data and labels indicating English level determined by the experts. Let's overview all the data we have so far:\n",
    "* a `.xls` spreadsheet containing movie titles and labels\n",
    "* a bunch of `.srt` files piled in one folder\n",
    "* some `.srt` files sorted by English level in separate folders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02e4791b-8275-42fa-a2f7-9d917e4ae63d",
   "metadata": {},
   "source": [
    "We will extract text data using [pysrt](https://github.com/byroot/pysrt) library. This library can extract plain text from `.srt` files without timestamps but has troubles dealing with some encodings. Thus the first thing we do is to encode all the `.srt.` files to `UTF-8` and put them into `/utf-8` subfolder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162e189b-c24f-48a4-ab68-383011f67c32",
   "metadata": {},
   "source": [
    "### Encoding all files to UTF-8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ce1269a4-5e55-4b06-b9ea-beb334915655",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function returns encoding of the file\n",
    "def encoding_detector(file_path):\n",
    "    # read the first 1000 bytes of the file\n",
    "    with open(file_path, 'rb') as file:\n",
    "        raw_data = file.read(1000)\n",
    "    # detect the encoding of the file\n",
    "    result = chardet.detect(raw_data)\n",
    "    encoding = result['encoding']\n",
    "    file.close()\n",
    "    return encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "deb01d24-4bd4-48f7-89fe-f7158b5ef0d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function takes folder, gets all .srt files , encodes them to utf-8\n",
    "# and puts them into /utf-8 subfolder\n",
    "def folder_to_utf(folder_path):\n",
    "    # create a utf-8 subfolder\n",
    "    os.makedirs(os.path.join(folder_path, 'utf-8'), exist_ok=True)\n",
    "\n",
    "    # loop through all files in the folder\n",
    "    for filename in os.listdir(folder_path):\n",
    "        if filename.endswith('.srt'):\n",
    "            # define the file paths\n",
    "            file_path = os.path.join(folder_path, filename)\n",
    "            new_file_path = os.path.join(folder_path, 'utf-8', filename)\n",
    "\n",
    "            # open the file and read its contents\n",
    "            with codecs.open(file_path, 'r', encoding=encoding_detector(file_path), errors='replace') as file:\n",
    "                contents = file.read()\n",
    "                file.close()\n",
    "\n",
    "            # write the contents to a new file with UTF-8 encoding\n",
    "            with codecs.open(new_file_path, 'w', encoding='UTF-8', errors='replace') as new_file:\n",
    "                new_file.write(contents)\n",
    "                new_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d417d5f-858d-4ce4-ae97-b9f63f0d6512",
   "metadata": {},
   "source": [
    "Now we iterate through all the folders we have, encode files to `UTF-8` and put them in `/utf-8` subfolder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ee21c8f8-e82a-418d-9742-4aaedfdb9a5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for folder in [\n",
    "    PATH_ALL_SUBS_FOLDER,\n",
    "    PATH_SUBS_A2,\n",
    "    PATH_SUBS_B1,\n",
    "    PATH_SUBS_B2,\n",
    "    PATH_SUBS_C1\n",
    "]:\n",
    "    folder_to_utf(folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c50d7ff3-bb37-481c-a2ab-667cecd8d86b",
   "metadata": {},
   "source": [
    "Now we have a `/utf-8` subfolder with re-encoded `.srt` files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98e091a7-9fed-457c-ab12-45bc3b304087",
   "metadata": {},
   "source": [
    "### Adding subs to the spreadsheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "45c269ef-4e73-42ef-b99e-911fc0a9ae3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving path to the folder with reencoded .srt\n",
    "all_subs_path = Path(PATH_ALL_SUBS_FOLDER+UTF8_SUBFOLDER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "edf7805c-de99-47c3-94d7-4829ba987bd9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_name</th>\n",
       "      <th>file_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10_Cloverfield_lane(2016).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10_things_I_hate_about_you(1999).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aladdin(1992).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>All_dogs_go_to_heaven(1989).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>An_American_tail(1986).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              file_name  \\\n",
       "0         10_Cloverfield_lane(2016).srt   \n",
       "1  10_things_I_hate_about_you(1999).srt   \n",
       "2                     Aladdin(1992).srt   \n",
       "3       All_dogs_go_to_heaven(1989).srt   \n",
       "4            An_American_tail(1986).srt   \n",
       "\n",
       "                                           file_path  \n",
       "0  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...  \n",
       "1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...  \n",
       "2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  \n",
       "3  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  \n",
       "4  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 115 subtitle files\n"
     ]
    }
   ],
   "source": [
    "# getting df with file names and file paths\n",
    "all_subs_list = [p.name for p in all_subs_path.glob('*.srt')]\n",
    "all_subs_df = pd.DataFrame({'file_name': all_subs_list,\n",
    "                            'file_path': list(all_subs_path.glob('*.srt'))})\n",
    "display(all_subs_df.head())\n",
    "print(f'Found {all_subs_df.shape[0]} subtitle files')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df65710e-340d-4db0-a23f-ce5b3b9090eb",
   "metadata": {},
   "source": [
    "Let's add file names and file paths to the spreadsheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d0574c92-2d4c-4e50-9f01-f5869f1d14b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_labels_df['file_name'] = movies_labels_df.apply(\n",
    "    lambda x: difflib.get_close_matches(x['movie'], all_subs_list, n=1, cutoff=0.8), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "749a6d83-dcce-4fb7-9c62-66b19606c422",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function extracts lists after .get_close_matches method\n",
    "def list_extractor(list):\n",
    "    if list == []:\n",
    "        return np.nan\n",
    "    else:\n",
    "        return list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7afc03e9-12e3-42e7-b024-f114e2c0f82e",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_labels_df['file_name'] = movies_labels_df['file_name'].apply(list_extractor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9b46cb83-4011-4f88-b32f-d5de891a8a3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie</th>\n",
       "      <th>label</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>year</th>\n",
       "      <th>file_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10_Cloverfield_lane(2016)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 cloverfield lane</td>\n",
       "      <td>2016</td>\n",
       "      <td>10_Cloverfield_lane(2016).srt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10_things_I_hate_about_you(1999)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 things i hate about you</td>\n",
       "      <td>1999</td>\n",
       "      <td>10_things_I_hate_about_you(1999).srt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A_knights_tale(2001)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a knights tale</td>\n",
       "      <td>2001</td>\n",
       "      <td>A_knights_tale(2001).srt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A_star_is_born(2018)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a star is born</td>\n",
       "      <td>2018</td>\n",
       "      <td>A_star_is_born(2018).srt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aladdin(1992)</td>\n",
       "      <td>A2/A2+</td>\n",
       "      <td>aladdin</td>\n",
       "      <td>1992</td>\n",
       "      <td>Aladdin(1992).srt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              movie   label                 movie_title  year  \\\n",
       "0         10_Cloverfield_lane(2016)      B1         10 cloverfield lane  2016   \n",
       "1  10_things_I_hate_about_you(1999)      B1  10 things i hate about you  1999   \n",
       "2              A_knights_tale(2001)      B2              a knights tale  2001   \n",
       "3              A_star_is_born(2018)      B2              a star is born  2018   \n",
       "4                     Aladdin(1992)  A2/A2+                     aladdin  1992   \n",
       "\n",
       "                              file_name  \n",
       "0         10_Cloverfield_lane(2016).srt  \n",
       "1  10_things_I_hate_about_you(1999).srt  \n",
       "2              A_knights_tale(2001).srt  \n",
       "3              A_star_is_born(2018).srt  \n",
       "4                     Aladdin(1992).srt  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(movies_labels_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "df62fcb2-4547-4744-8175-89daeb61bf65",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 241 entries, 0 to 240\n",
      "Data columns (total 5 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   movie        241 non-null    object\n",
      " 1   label        241 non-null    object\n",
      " 2   movie_title  241 non-null    object\n",
      " 3   year         108 non-null    Int32 \n",
      " 4   file_name    110 non-null    object\n",
      "dtypes: Int32(1), object(4)\n",
      "memory usage: 8.8+ KB\n"
     ]
    }
   ],
   "source": [
    "# let's check df for nans\n",
    "movies_labels_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed57c89-bfa3-4a85-9b2a-be06646699b8",
   "metadata": {},
   "source": [
    "We will perform an inner merge on file names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "945b8d0e-10a0-4739-93e8-1fbc764c8194",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie</th>\n",
       "      <th>label</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>year</th>\n",
       "      <th>file_name</th>\n",
       "      <th>file_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10_Cloverfield_lane(2016)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 cloverfield lane</td>\n",
       "      <td>2016</td>\n",
       "      <td>10_Cloverfield_lane(2016).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10_things_I_hate_about_you(1999)</td>\n",
       "      <td>B1</td>\n",
       "      <td>10 things i hate about you</td>\n",
       "      <td>1999</td>\n",
       "      <td>10_things_I_hate_about_you(1999).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A_knights_tale(2001)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a knights tale</td>\n",
       "      <td>2001</td>\n",
       "      <td>A_knights_tale(2001).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A_star_is_born(2018)</td>\n",
       "      <td>B2</td>\n",
       "      <td>a star is born</td>\n",
       "      <td>2018</td>\n",
       "      <td>A_star_is_born(2018).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aladdin(1992)</td>\n",
       "      <td>A2/A2+</td>\n",
       "      <td>aladdin</td>\n",
       "      <td>1992</td>\n",
       "      <td>Aladdin(1992).srt</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              movie   label                 movie_title  year  \\\n",
       "0         10_Cloverfield_lane(2016)      B1         10 cloverfield lane  2016   \n",
       "1  10_things_I_hate_about_you(1999)      B1  10 things i hate about you  1999   \n",
       "2              A_knights_tale(2001)      B2              a knights tale  2001   \n",
       "3              A_star_is_born(2018)      B2              a star is born  2018   \n",
       "4                     Aladdin(1992)  A2/A2+                     aladdin  1992   \n",
       "\n",
       "                              file_name  \\\n",
       "0         10_Cloverfield_lane(2016).srt   \n",
       "1  10_things_I_hate_about_you(1999).srt   \n",
       "2              A_knights_tale(2001).srt   \n",
       "3              A_star_is_born(2018).srt   \n",
       "4                     Aladdin(1992).srt   \n",
       "\n",
       "                                           file_path  \n",
       "0  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...  \n",
       "1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...  \n",
       "2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  \n",
       "3  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  \n",
       "4  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched 110 files\n"
     ]
    }
   ],
   "source": [
    "# merging dfs\n",
    "subs_df = movies_labels_df.merge(all_subs_df, how='inner', left_on='file_name', right_on='file_name')\n",
    "display(subs_df.head())\n",
    "print(f'Matched {subs_df.shape[0]} files')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cc1498b-82ec-4b05-85fd-5e7ec4050a94",
   "metadata": {},
   "source": [
    "### Adding subs from other folders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be9980a3-14db-4f97-ac74-2f3cfdd3c05a",
   "metadata": {},
   "source": [
    "We will scan all other folders for `.srt` files, add their file pathes to the list and add labels based on folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "56f9364d-9c42-41ac-a729-1675fa3d8d24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>file_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A2/A2+</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\Z...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>110 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                          file_path\n",
       "0        B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...\n",
       "1        B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...\n",
       "2        B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "3        B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "4    A2/A2+  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "..      ...                                                ...\n",
       "105      B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\V...\n",
       "106      B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...\n",
       "107      B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...\n",
       "108      B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\W...\n",
       "109      B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\Z...\n",
       "\n",
       "[110 rows x 2 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subs_df = subs_df[['label', 'file_path']]\n",
    "subs_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d1ee8f24-80d2-43bc-8fcd-0dd6aa5aff74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 6 subs for A2 label...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...</td>\n",
       "      <td>A2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...</td>\n",
       "      <td>A2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...</td>\n",
       "      <td>A2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           file_path label\n",
       "0  English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...    A2\n",
       "1  English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...    A2\n",
       "2  English_scores\\Subtitles_all\\A2\\utf-8\\The Walk...    A2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 17 subs for B1 label...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>English_scores\\Subtitles_all\\B1\\utf-8\\American...</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>English_scores\\Subtitles_all\\B1\\utf-8\\Angelas....</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>English_scores\\Subtitles_all\\B1\\utf-8\\Indiana ...</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           file_path label\n",
       "0  English_scores\\Subtitles_all\\B1\\utf-8\\American...    B1\n",
       "1  English_scores\\Subtitles_all\\B1\\utf-8\\Angelas....    B1\n",
       "2  English_scores\\Subtitles_all\\B1\\utf-8\\Indiana ...    B1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 107 subs for B2 label...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>English_scores\\Subtitles_all\\B2\\utf-8\\Angela's...</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>English_scores\\Subtitles_all\\B2\\utf-8\\Collater...</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>English_scores\\Subtitles_all\\B2\\utf-8\\Crazy4TV...</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           file_path label\n",
       "0  English_scores\\Subtitles_all\\B2\\utf-8\\Angela's...    B2\n",
       "1  English_scores\\Subtitles_all\\B2\\utf-8\\Collater...    B2\n",
       "2  English_scores\\Subtitles_all\\B2\\utf-8\\Crazy4TV...    B2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 33 subs for C1 label...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...</td>\n",
       "      <td>C1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...</td>\n",
       "      <td>C1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...</td>\n",
       "      <td>C1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           file_path label\n",
       "0  English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...    C1\n",
       "1  English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...    C1\n",
       "2  English_scores\\Subtitles_all\\C1\\utf-8\\Downton ...    C1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All subs dataset\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>file_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A2/A2+</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>C1</td>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>C1</td>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>C1</td>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>C1</td>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>C1</td>\n",
       "      <td>English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>273 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                          file_path\n",
       "0        B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...\n",
       "1        B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...\n",
       "2        B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "3        B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "4    A2/A2+  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...\n",
       "..      ...                                                ...\n",
       "268      C1  English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...\n",
       "269      C1  English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...\n",
       "270      C1  English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...\n",
       "271      C1  English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...\n",
       "272      C1  English_scores\\Subtitles_all\\C1\\utf-8\\Suits.S0...\n",
       "\n",
       "[273 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# iterate through folders\n",
    "for folder, label in zip([PATH_SUBS_A2, PATH_SUBS_B1, PATH_SUBS_B2, PATH_SUBS_C1],\n",
    "                         ['A2', 'B1', 'B2', 'C1']):\n",
    "    folder_utf8 = folder + UTF8_SUBFOLDER\n",
    "    temp_df = pd.DataFrame({'file_path': list(Path(folder_utf8).glob('*.srt'))})\n",
    "    temp_df['label'] = label\n",
    "    print(f'Adding {temp_df.shape[0]} subs for {label} label...')\n",
    "    display(temp_df.head(3))\n",
    "    subs_df = pd.concat([subs_df, temp_df], axis=0, ignore_index=True)\n",
    "\n",
    "print('All subs dataset')\n",
    "display(subs_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "09bd6fd8-e817-4cb9-bda6-05133aa14ea1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "B2            136\n",
       "B1             54\n",
       "C1             39\n",
       "A2/A2+         25\n",
       "B1, B2          8\n",
       "A2              6\n",
       "A2/A2+, B1      5\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking all the classes\n",
    "subs_df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf5810a3-2b69-43bf-9af4-396a43741743",
   "metadata": {},
   "source": [
    "There are some objects, that were classified in multiple cleasses. Also there are no **A1** and **C2**.\n",
    "* **A1** level is too basic and it's not so easy to find movies that don't go beyond **A1** in terms of language (except ones that are made this way)\n",
    "* We will re-label **A2/A2+** to just **A2**\n",
    "* We will re-label **B1, B2** to **B1**. Since movies are used for language learning, we can choose lower skill level\n",
    "* We will re-label **A2/A2+, B1** to **A2** following the same motive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9e20804b-0f18-4766-af6c-41235e20f956",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "B2    136\n",
       "B1     62\n",
       "C1     39\n",
       "A2     36\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename classes\n",
    "subs_df.loc[(subs_df['label'] == 'A2/A2+')|(subs_df['label'] == 'A2/A2+, B1'), 'label'] = 'A2'\n",
    "subs_df.loc[subs_df['label'] == 'B1, B2', 'label'] = 'B1'\n",
    "subs_df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adae08b2-a6c7-497b-a85d-003334f75613",
   "metadata": {},
   "source": [
    "### Adding text data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eebc63d-acd6-4138-898a-6dc9014d2829",
   "metadata": {},
   "source": [
    "Now we add subtitle text to every DataFrame object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "09ac5e46-22d3-4698-ba26-996d355d2be9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function extracts raw text from .srt file\n",
    "def srt_raw_text(file_path):\n",
    "    try:\n",
    "        subs = pysrt.open(file_path)\n",
    "        return subs.text\n",
    "    except:\n",
    "        return np.NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "eacb01ce-ca00-471e-ad6a-ccf0eceb457d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function extracts full text from .srt file\n",
    "def srt_full_subs(file_path):\n",
    "    try:\n",
    "        with open(file_path) as file:\n",
    "            full_text = file.read()\n",
    "            file.close()\n",
    "        return full_text\n",
    "    except:\n",
    "        return np.NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "301020ea-027f-43cf-a5e6-632debffa093",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>file_path</th>\n",
       "      <th>raw_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "      <td>&lt;font color=\"#ffff80\"&gt;&lt;b&gt;Fixed &amp; Synced by boz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "      <td>Hey!\\nI'll be right with you.\\nSo, Cameron. He...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>Resync: Xenzai[NEF]\\nRETAIL\\nShould we help hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>- &lt;i&gt;&lt;font color=\"#ffffff\"&gt; Synced and correct...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>&lt;i&gt;Oh, I come from a land\\nFrom a faraway plac...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                          file_path  \\\n",
       "0    B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...   \n",
       "1    B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...   \n",
       "2    B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "3    B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "4    A2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "\n",
       "                                            raw_text  \n",
       "0  <font color=\"#ffff80\"><b>Fixed & Synced by boz...  \n",
       "1  Hey!\\nI'll be right with you.\\nSo, Cameron. He...  \n",
       "2  Resync: Xenzai[NEF]\\nRETAIL\\nShould we help hi...  \n",
       "3  - <i><font color=\"#ffffff\"> Synced and correct...  \n",
       "4  <i>Oh, I come from a land\\nFrom a faraway plac...  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# applying text extraction function to df\n",
    "subs_df['raw_text'] = subs_df['file_path'].apply(srt_raw_text)\n",
    "subs_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e4036726-e233-4f24-bf9f-ca56fce8a56d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>file_path</th>\n",
       "      <th>raw_text</th>\n",
       "      <th>full_subs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "      <td>&lt;font color=\"#ffff80\"&gt;&lt;b&gt;Fixed &amp; Synced by boz...</td>\n",
       "      <td>1\\n00:00:55,279 --&gt; 00:01:07,279\\n&lt;font color=...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B1</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...</td>\n",
       "      <td>Hey!\\nI'll be right with you.\\nSo, Cameron. He...</td>\n",
       "      <td>1\\n00:01:54,281 --&gt; 00:01:55,698\\nHey!\\n\\n2\\n0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>Resync: Xenzai[NEF]\\nRETAIL\\nShould we help hi...</td>\n",
       "      <td>1\\n00:00:15,089 --&gt; 00:00:21,229\\nResync: Xenz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>- &lt;i&gt;&lt;font color=\"#ffffff\"&gt; Synced and correct...</td>\n",
       "      <td>1\\n00:00:17,610 --&gt; 00:00:22,610\\n- &lt;i&gt;&lt;font c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A2</td>\n",
       "      <td>English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...</td>\n",
       "      <td>&lt;i&gt;Oh, I come from a land\\nFrom a faraway plac...</td>\n",
       "      <td>1\\n00:00:27,240 --&gt; 00:00:30,879\\n&lt;i&gt;Oh, I com...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                          file_path  \\\n",
       "0    B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...   \n",
       "1    B1  English_scores\\Subtitles_all\\Subtitles\\utf-8\\1...   \n",
       "2    B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "3    B2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "4    A2  English_scores\\Subtitles_all\\Subtitles\\utf-8\\A...   \n",
       "\n",
       "                                            raw_text  \\\n",
       "0  <font color=\"#ffff80\"><b>Fixed & Synced by boz...   \n",
       "1  Hey!\\nI'll be right with you.\\nSo, Cameron. He...   \n",
       "2  Resync: Xenzai[NEF]\\nRETAIL\\nShould we help hi...   \n",
       "3  - <i><font color=\"#ffffff\"> Synced and correct...   \n",
       "4  <i>Oh, I come from a land\\nFrom a faraway plac...   \n",
       "\n",
       "                                           full_subs  \n",
       "0  1\\n00:00:55,279 --> 00:01:07,279\\n<font color=...  \n",
       "1  1\\n00:01:54,281 --> 00:01:55,698\\nHey!\\n\\n2\\n0...  \n",
       "2  1\\n00:00:15,089 --> 00:00:21,229\\nResync: Xenz...  \n",
       "3  1\\n00:00:17,610 --> 00:00:22,610\\n- <i><font c...  \n",
       "4  1\\n00:00:27,240 --> 00:00:30,879\\n<i>Oh, I com...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add full text to dataset\n",
    "subs_df['full_subs'] = subs_df['file_path'].apply(srt_full_subs)\n",
    "subs_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "3ede2b93-e07d-434e-a033-b24fc3a84808",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 273 entries, 0 to 272\n",
      "Data columns (total 4 columns):\n",
      " #   Column     Non-Null Count  Dtype \n",
      "---  ------     --------------  ----- \n",
      " 0   label      273 non-null    object\n",
      " 1   file_path  273 non-null    object\n",
      " 2   raw_text   273 non-null    object\n",
      " 3   full_subs  273 non-null    object\n",
      "dtypes: object(4)\n",
      "memory usage: 8.7+ KB\n"
     ]
    }
   ],
   "source": [
    "subs_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61785b29-3e02-4ac6-bf1e-489c680441aa",
   "metadata": {},
   "source": [
    "### Saving train data to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6aabd3b1-070b-4120-88c1-e7efa5ca2c02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# leave only labels and text and save\n",
    "subs_df[['label', 'raw_text', 'full_subs']].to_csv(path_or_buf=PATH_TRAIN_DATA, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25afc201-f1dd-4096-8eb0-3e4546806ae4",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78cc0f79-93f8-4dc6-9910-938b5280f3b1",
   "metadata": {},
   "source": [
    "The main objective here was to get all the data together and form a solid dataset. All files are encoded differently so we had to preprocess them in on encoding. The main problem is there's not so much labeled data collected"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e351445-929f-4f08-9fec-e82b99a413ea",
   "metadata": {},
   "source": [
    "We created a dataset that contains:\n",
    "* raw text for avery subtitle file\n",
    "* full subtutile data with timestamps\n",
    "* lebels for subtitles\n",
    "\n",
    "Now we can use this file for training"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}