dota2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Это моё решение задачи в качестве сабмишена для финального проекта специализации \n",
    "на курсере по введение в машинное обучение от вышки и Яндекса**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для начала просто извлечём данные: сделаем это при помощи ранее выбранных признаков. Поскольку цели показать  большой балл на лидерборде у меня нет, то обойдёмся этим неполным набором признаков."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_time</th>\n",
       "      <th>lobby_type</th>\n",
       "      <th>r1_hero</th>\n",
       "      <th>r1_level</th>\n",
       "      <th>r1_xp</th>\n",
       "      <th>r1_gold</th>\n",
       "      <th>r1_lh</th>\n",
       "      <th>r1_kills</th>\n",
       "      <th>r1_deaths</th>\n",
       "      <th>r1_items</th>\n",
       "      <th>...</th>\n",
       "      <th>dire_boots_count</th>\n",
       "      <th>dire_ward_observer_count</th>\n",
       "      <th>dire_ward_sentry_count</th>\n",
       "      <th>dire_first_ward_time</th>\n",
       "      <th>duration</th>\n",
       "      <th>radiant_win</th>\n",
       "      <th>tower_status_radiant</th>\n",
       "      <th>tower_status_dire</th>\n",
       "      <th>barracks_status_radiant</th>\n",
       "      <th>barracks_status_dire</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>match_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430198770</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>2098</td>\n",
       "      <td>1489</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-52.0</td>\n",
       "      <td>2874</td>\n",
       "      <td>1</td>\n",
       "      <td>1796</td>\n",
       "      <td>0</td>\n",
       "      <td>51</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1430220345</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>4</td>\n",
       "      <td>1188</td>\n",
       "      <td>1033</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>2463</td>\n",
       "      <td>1</td>\n",
       "      <td>1974</td>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1430227081</td>\n",
       "      <td>7</td>\n",
       "      <td>33</td>\n",
       "      <td>4</td>\n",
       "      <td>1319</td>\n",
       "      <td>1270</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1830</td>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1430263531</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "      <td>1779</td>\n",
       "      <td>1056</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1459</td>\n",
       "      <td>0</td>\n",
       "      <td>1920</td>\n",
       "      <td>2047</td>\n",
       "      <td>50</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1430282290</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>1431</td>\n",
       "      <td>1090</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>-16.0</td>\n",
       "      <td>2449</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1974</td>\n",
       "      <td>3</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 108 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          start_time  lobby_type  r1_hero  r1_level  r1_xp  r1_gold  r1_lh  \\\n",
       "match_id                                                                     \n",
       "0         1430198770           7       11         5   2098     1489     20   \n",
       "1         1430220345           0       42         4   1188     1033      9   \n",
       "2         1430227081           7       33         4   1319     1270     22   \n",
       "3         1430263531           1       29         4   1779     1056     14   \n",
       "4         1430282290           7       13         4   1431     1090      8   \n",
       "\n",
       "          r1_kills  r1_deaths  r1_items          ...           \\\n",
       "match_id                                         ...            \n",
       "0                0          0         7          ...            \n",
       "1                0          1        12          ...            \n",
       "2                0          0        12          ...            \n",
       "3                0          0         5          ...            \n",
       "4                1          0         8          ...            \n",
       "\n",
       "          dire_boots_count  dire_ward_observer_count  dire_ward_sentry_count  \\\n",
       "match_id                                                                       \n",
       "0                        4                         2                       2   \n",
       "1                        4                         3                       1   \n",
       "2                        4                         3                       1   \n",
       "3                        4                         2                       0   \n",
       "4                        3                         3                       0   \n",
       "\n",
       "          dire_first_ward_time  duration  radiant_win  tower_status_radiant  \\\n",
       "match_id                                                                      \n",
       "0                        -52.0      2874            1                  1796   \n",
       "1                         -5.0      2463            1                  1974   \n",
       "2                         13.0      2130            0                     0   \n",
       "3                         27.0      1459            0                  1920   \n",
       "4                        -16.0      2449            0                     4   \n",
       "\n",
       "          tower_status_dire  barracks_status_radiant  barracks_status_dire  \n",
       "match_id                                                                    \n",
       "0                         0                       51                     0  \n",
       "1                         0                       63                     1  \n",
       "2                      1830                        0                    63  \n",
       "3                      2047                       50                    63  \n",
       "4                      1974                        3                    63  \n",
       "\n",
       "[5 rows x 108 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "train_data = pd.read_csv('./features.csv', index_col='match_id')\n",
    "test_data = pd.read_csv('./features_test.csv', index_col='match_id')\n",
    "train_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предлагаю посмотреть более детально какие признаки мы имеем"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['start_time', 'lobby_type', 'r1_hero', 'r1_level', 'r1_xp', 'r1_gold', 'r1_lh', 'r1_kills', 'r1_deaths', 'r1_items', 'r2_hero', 'r2_level', 'r2_xp', 'r2_gold', 'r2_lh', 'r2_kills', 'r2_deaths', 'r2_items', 'r3_hero', 'r3_level', 'r3_xp', 'r3_gold', 'r3_lh', 'r3_kills', 'r3_deaths', 'r3_items', 'r4_hero', 'r4_level', 'r4_xp', 'r4_gold', 'r4_lh', 'r4_kills', 'r4_deaths', 'r4_items', 'r5_hero', 'r5_level', 'r5_xp', 'r5_gold', 'r5_lh', 'r5_kills', 'r5_deaths', 'r5_items', 'd1_hero', 'd1_level', 'd1_xp', 'd1_gold', 'd1_lh', 'd1_kills', 'd1_deaths', 'd1_items', 'd2_hero', 'd2_level', 'd2_xp', 'd2_gold', 'd2_lh', 'd2_kills', 'd2_deaths', 'd2_items', 'd3_hero', 'd3_level', 'd3_xp', 'd3_gold', 'd3_lh', 'd3_kills', 'd3_deaths', 'd3_items', 'd4_hero', 'd4_level', 'd4_xp', 'd4_gold', 'd4_lh', 'd4_kills', 'd4_deaths', 'd4_items', 'd5_hero', 'd5_level', 'd5_xp', 'd5_gold', 'd5_lh', 'd5_kills', 'd5_deaths', 'd5_items', 'first_blood_time', 'first_blood_team', 'first_blood_player1', 'first_blood_player2', 'radiant_bottle_time', 'radiant_courier_time', 'radiant_flying_courier_time', 'radiant_tpscroll_count', 'radiant_boots_count', 'radiant_ward_observer_count', 'radiant_ward_sentry_count', 'radiant_first_ward_time', 'dire_bottle_time', 'dire_courier_time', 'dire_flying_courier_time', 'dire_tpscroll_count', 'dire_boots_count', 'dire_ward_observer_count', 'dire_ward_sentry_count', 'dire_first_ward_time', 'duration', 'radiant_win', 'tower_status_radiant', 'tower_status_dire', 'barracks_status_radiant', 'barracks_status_dire']\n"
     ]
    }
   ],
   "source": [
    "features = []\n",
    "for label in train_data.columns:\n",
    "    features.append(label)\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Отлично! Теперь мы видим перед глазами набор признаков, которые у нас есть. Сразу же хочется понять как их можно объединить... С одной стороны мы видим всевозможные статистики по каждому из героев. Много ли от этого толка? Не уверен, потому что игра командная...\n",
    "<br>На самом деле надо немного объянить типичную ситауцию в доте - есть мидер, керри, саппорты и оффлейнер. Как правило саппорты не имеют ни денег, ни опыта, покупают вещи на команду(варды, хилки, телепорты и так далее). Мидеры стоят 1 на 1, имеют много опыта и их задача на первые 5 минут обыграть вражеского мидера, получить много опыта и задоминировать линию и как правило, мидеры задают динамику боя - быстрее всех получают сильную способность(ультимейт) и гангают другие лайны, убивая врагов. Задача керри - фармить деньги, убивая крипов, чтобы в конце боя убить всех остальных - как правило такие герои критичны к крипстату. Ну и наконец есть оффлейнеры - врываторы, инициаторы -  они стоят против 2-3 героев... Их задача получить как можно больше опыта, при этом желательно не умирая.\n",
    "<br>Этакие правило дотера - ботинки на первой минуте - плохо, нет куры - плохо, варды не стоят - саппорты раки. Фид врагов тоже понятен.\n",
    "<br>Понятно, что в целом, из-за командности игры можно ввести признак, который будет показывать общие показатели команды... Хотя он и будет линейно зависим от остальных.\n",
    "<br>Теперь что касается результатов матча - у нас есть статистики по зданиям и победитель встречи... Целевой переменно тут конечно является \"radiant_win\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь давайте уберём лишние признаки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_time</th>\n",
       "      <th>lobby_type</th>\n",
       "      <th>r1_hero</th>\n",
       "      <th>r1_level</th>\n",
       "      <th>r1_xp</th>\n",
       "      <th>r1_gold</th>\n",
       "      <th>r1_lh</th>\n",
       "      <th>r1_kills</th>\n",
       "      <th>r1_deaths</th>\n",
       "      <th>r1_items</th>\n",
       "      <th>...</th>\n",
       "      <th>radiant_ward_sentry_count</th>\n",
       "      <th>radiant_first_ward_time</th>\n",
       "      <th>dire_bottle_time</th>\n",
       "      <th>dire_courier_time</th>\n",
       "      <th>dire_flying_courier_time</th>\n",
       "      <th>dire_tpscroll_count</th>\n",
       "      <th>dire_boots_count</th>\n",
       "      <th>dire_ward_observer_count</th>\n",
       "      <th>dire_ward_sentry_count</th>\n",
       "      <th>dire_first_ward_time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>match_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430198770</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>2098</td>\n",
       "      <td>1489</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>-84.0</td>\n",
       "      <td>221.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-52.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1430220345</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>4</td>\n",
       "      <td>1188</td>\n",
       "      <td>1033</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>-84.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1430227081</td>\n",
       "      <td>7</td>\n",
       "      <td>33</td>\n",
       "      <td>4</td>\n",
       "      <td>1319</td>\n",
       "      <td>1270</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>-39.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>-77.0</td>\n",
       "      <td>221.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1430263531</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "      <td>1779</td>\n",
       "      <td>1056</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>-30.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>-80.0</td>\n",
       "      <td>184.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1430282290</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>1431</td>\n",
       "      <td>1090</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>-80.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          start_time  lobby_type  r1_hero  r1_level  r1_xp  r1_gold  r1_lh  \\\n",
       "match_id                                                                     \n",
       "0         1430198770           7       11         5   2098     1489     20   \n",
       "1         1430220345           0       42         4   1188     1033      9   \n",
       "2         1430227081           7       33         4   1319     1270     22   \n",
       "3         1430263531           1       29         4   1779     1056     14   \n",
       "4         1430282290           7       13         4   1431     1090      8   \n",
       "\n",
       "          r1_kills  r1_deaths  r1_items          ...           \\\n",
       "match_id                                         ...            \n",
       "0                0          0         7          ...            \n",
       "1                0          1        12          ...            \n",
       "2                0          0        12          ...            \n",
       "3                0          0         5          ...            \n",
       "4                1          0         8          ...            \n",
       "\n",
       "          radiant_ward_sentry_count  radiant_first_ward_time  \\\n",
       "match_id                                                       \n",
       "0                                 0                     35.0   \n",
       "1                                 0                    -20.0   \n",
       "2                                 1                    -39.0   \n",
       "3                                 0                    -30.0   \n",
       "4                                 0                     46.0   \n",
       "\n",
       "          dire_bottle_time  dire_courier_time  dire_flying_courier_time  \\\n",
       "match_id                                                                  \n",
       "0                    103.0              -84.0                     221.0   \n",
       "1                    149.0              -84.0                     195.0   \n",
       "2                     45.0              -77.0                     221.0   \n",
       "3                    124.0              -80.0                     184.0   \n",
       "4                    182.0              -80.0                     225.0   \n",
       "\n",
       "          dire_tpscroll_count  dire_boots_count  dire_ward_observer_count  \\\n",
       "match_id                                                                    \n",
       "0                           3                 4                         2   \n",
       "1                           5                 4                         3   \n",
       "2                           3                 4                         3   \n",
       "3                           0                 4                         2   \n",
       "4                           6                 3                         3   \n",
       "\n",
       "          dire_ward_sentry_count  dire_first_ward_time  \n",
       "match_id                                                \n",
       "0                              2                 -52.0  \n",
       "1                              1                  -5.0  \n",
       "2                              1                  13.0  \n",
       "3                              0                  27.0  \n",
       "4                              0                 -16.0  \n",
       "\n",
       "[5 rows x 102 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#запишем нашу целевую переменную\n",
    "y_train = train_data['radiant_win']\n",
    "#запомним лишние данные\n",
    "towers_dict = ['duration', 'tower_status_radiant', 'tower_status_dire', 'barracks_status_radiant', \n",
    "               'barracks_status_dire', 'radiant_win']\n",
    "towers_df = train_data[towers_dict]\n",
    "\n",
    "#Да, мы записали эти данные отдельно, но ещё не убрали их из трейна\n",
    "train_df = train_data.drop(towers_dict, axis=1)\n",
    "train_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь всё хорошо, но по факту наши признаки нужно обрабатывать и обрабатывать. Да, большинство из них числовые, но не все, да и значения  некоторых остутствуют"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "97230\n",
      "{'radiant_flying_courier_time': [69751, 12292], 'first_blood_time': [77677, 13625], 'first_blood_player1': [77677, 13625], 'first_blood_player2': [53243, 9411], 'first_blood_team': [77677, 13625], 'dire_courier_time': [96554, 17047], 'radiant_courier_time': [96538, 17050], 'radiant_first_ward_time': [95394, 16847], 'radiant_bottle_time': [81539, 14282], 'dire_bottle_time': [81087, 14335], 'dire_first_ward_time': [95404, 16914], 'dire_flying_courier_time': [71132, 12653]}\n"
     ]
    }
   ],
   "source": [
    "#Первым делом посчитаем в каких столбцах у нас сколько значений\n",
    "data_nulls = {}\n",
    "print(len(y_train))\n",
    "for label in train_df.columns:\n",
    "    if train_df[label].count() < 97230:\n",
    "        data_nulls[label]=[train_df[label].count(), test_data[label].count()]\n",
    "print(data_nulls) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте возьмём 2 признака с наибольшим числом пропусков и попробуем про них что-то сказать:\n",
    "<br>1)<b>\"first_blood_player2\"</b> - первый килл был сделан без ассистов \n",
    "<br>2)<b>\"radiant_flying_courier_time\"</b> - кура не нужна =D Учитывая что доступ к ней с 3+ минуты"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Уберём пропуски!\n",
    "train_df = train_df.fillna(0).astype('int')\n",
    "test_df = test_data.fillna(0).astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь самое весёлое - время <b>Обучения</b>!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/3 [00:00<?, ?it/s] 33%|███▎      | 1/3 [00:33<01:06, 33.20s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.664647480884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 2/3 [01:31<00:40, 40.75s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.682183356598\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 3/3 [02:55<00:00, 53.65s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.689532166304\n",
      "CPU times: user 2min 53s, sys: 1.29 s, total: 2min 54s\n",
      "Wall time: 2min 55s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from tqdm import tqdm\n",
    "kf = KFold(n_splits=5, shuffle=True)\n",
    "tree_amount = [10, 20, 30]\n",
    "for n_tree in tqdm(tree_amount):\n",
    "    clf = GradientBoostingClassifier(n_estimators=n_tree)\n",
    "    print(cross_val_score(clf, train_df, y_train, scoring='roc_auc', cv=kf).mean())\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Итак, что делать, если число деревьев растёт? Ответ как мне кажется тут довольно прост. Как мы видим из результатов обучения над прошлым множеством, то можно сделать вывод, что максимум пока не достигнут, соотвественно можно продолжать тюнить нашу модель. Поскольку мы используем кросс-валидацию, то переобучения быть не должно. Вопрос о том, как ускорить наш алгоритм? Мне кажется, что одна минута - не такой долгий срок. Рост времени от кол-ва деревьев является линейным.\n",
    "<br>Ответы на вопрос про ускорение уже есть внизу - можно уменьшить обучающую выборку - деревья малочувствительны к размеру, можно уменьшить глубину, поскольку в градиентном бустинге в отличие от случайного леса нам важна ширина. Есть стохастический градиентный бустинг. Там случайные подвыборки... Наверное можно посмотреть на фичи и выкинуть признаки, которые слабо коррелируют с целевой переменной."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Часть вторая. Логистическая регрессия</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Итак, у нас есть уже предобработанный сет. Теперь его надо нормализовать и применить к нему логистическую регрессию. Давайте это и сделаем!\n",
    "<br>Чтобы подобрать параметр C можно конечно воспользоваться гридом, но поскольку от нас требуют повторить прошлую задачу, то давайте сделаем это вручную"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s] 10%|█         | 1/10 [00:08<01:19,  8.82s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.001 0.716127417478\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 2/10 [00:19<01:14,  9.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0035938136638 0.716316177719\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 3/10 [00:31<01:10, 10.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0129154966501 0.716454190211\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 4/10 [00:43<01:05, 10.86s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0464158883361 0.716504462033\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 5/10 [00:55<00:56, 11.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.16681005372 0.716432792802\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 6/10 [01:07<00:45, 11.41s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.599484250319 0.716455769659\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 7/10 [01:19<00:34, 11.56s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.15443469003 0.716392906434\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 8/10 [01:31<00:23, 11.77s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.74263682681 0.716396268218\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 9/10 [01:44<00:11, 11.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27.8255940221 0.716433010332\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 10/10 [01:57<00:00, 12.29s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100.0 0.716433957776\n",
      "CPU times: user 1min 56s, sys: 15.7 s, total: 2min 12s\n",
      "Wall time: 1min 58s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import numpy as np\n",
    "\n",
    "X_train = StandardScaler().fit_transform(train_df)\n",
    "C_space = np.logspace(-3, 2, 10)\n",
    "\n",
    "for c in tqdm(C_space):\n",
    "    clf = LogisticRegression(C=c)\n",
    "    print(c, cross_val_score(clf, X_train, y_train, scoring='roc_auc', cv=kf).mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s] 10%|█         | 1/10 [00:12<01:50, 12.24s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1 0.71640275448\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 2/10 [00:24<01:37, 12.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2 0.716263802391\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 3/10 [00:36<01:25, 12.16s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3 0.716445369363\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 4/10 [00:48<01:12, 12.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4 0.716477783942\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 5/10 [01:00<01:00, 12.00s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5 0.716369475468\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 6/10 [01:12<00:48, 12.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6 0.716665381643\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 7/10 [01:23<00:35, 11.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7 0.716434567137\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 8/10 [01:35<00:23, 11.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8 0.716150439988\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 9/10 [01:47<00:11, 11.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9 0.716396289048\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 10/10 [02:04<00:00, 13.33s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0 0.716348643726\n",
      "CPU times: user 2min 2s, sys: 15.2 s, total: 2min 18s\n",
      "Wall time: 2min 4s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#попробуем найти более точное значение C\n",
    "C_space = np.linspace(0.1, 1, 10)\n",
    "\n",
    "for c in tqdm(C_space):\n",
    "    clf = LogisticRegression(C=c)\n",
    "    print(c, cross_val_score(clf, X_train, y_train, scoring='roc_auc', cv=kf).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В целом можно увидеть, что искать С более точно нет смысла. Причиной, почему наш скор так скачет, скорее всего является то, что не ни фиксировали разбиение(Random State) при кросс-валидации... Поэтому без зазрения совести можно взять C = 0.5.\n",
    "<br>Соотвественно качество при нём примерно 0.716. Надо заметить, что оно площадь под рок-кривой получилась больше, чем при градиентном бустинге, хотя по идее деревья дольжны давать большую точность. Скорее всего это обусловлено тем, что в случае Градиентного бустинга мы просто недообучились и можно было увеличивать число деревьев(разница между 20 и 30 была порядка 0.01, что довольно много)\n",
    "<br>И если говорить про скорость - регрессия безусловно быстрее"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Окей, в прошлый раз мы поняли, что регрессия работает хорошо.. Однако с некоторыми переменнами мы поступали неправильно. Считали категориальные за числовые."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "categorial_features = ['lobby_type', 'r1_hero', 'r2_hero', 'r3_hero', 'r4_hero', 'r5_hero',\n",
    "                      'd1_hero', 'd2_hero', 'd3_hero', 'd4_hero', 'd5_hero']\n",
    "decimal_df = train_df.drop(categorial_features, axis=1)\n",
    "X_decimal_train = StandardScaler().fit_transform(decimal_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Итак, мы выкинули категориальные признаки... Подбор С про обучении будет немного другим, чем в прошлый раз - отптимальное значение понятно, что лежит примерно от 0.1 до 10, да и точность не слишком важна"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s] 10%|█         | 1/10 [00:10<01:32, 10.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.01 0.716294508529\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 2/10 [00:22<01:25, 10.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0215443469003 0.716612076807\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 3/10 [00:34<01:17, 11.12s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0464158883361 0.716465655655\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 4/10 [00:46<01:08, 11.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1 0.716412155744\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 5/10 [00:56<00:55, 11.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.215443469003 0.716225690512\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 6/10 [01:07<00:43, 10.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.464158883361 0.716597275565\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 7/10 [01:18<00:33, 11.01s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0 0.716325823045\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 8/10 [01:29<00:22, 11.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.15443469003 0.716775732218\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 9/10 [01:40<00:11, 11.05s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.64158883361 0.716403485595\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 10/10 [01:51<00:00, 11.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.0 0.716485323868\n",
      "CPU times: user 1min 51s, sys: 15.1 s, total: 2min 6s\n",
      "Wall time: 1min 51s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "C_space = np.logspace(-2, 1, 10)\n",
    "\n",
    "for c in tqdm(C_space):\n",
    "    clf = LogisticRegression(C=c)\n",
    "    print(c, cross_val_score(clf, X_decimal_train, y_train, scoring='roc_auc', cv=kf).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Как можно увидеть, качество практически не меняется - на четвёртый знак после запятой больше всего слияет random_seed. Причиной по которой такое может быть - наверное потому что в случае логистической регресии эти признаки слабо коррелировали с целевой переменной, по сути они были шумовые. Поэтому классификатор присвоил им малый вес - они сносили практически никакой вклад в ответ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Кстати, про подсчёт героев... В pandas 0.20.0 будет функция nunique, применяемая ко всему датафрейму.\n",
    "У нас есть есть пока только возможность работать с Серией.\n",
    "Не совсем ясно про распределение героев по позициям, поэтому создадим супер-серию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "108\n",
      "112\n"
     ]
    }
   ],
   "source": [
    "hero_features = ['r2_hero', 'r3_hero', 'r4_hero', 'r5_hero',\n",
    "                'd1_hero', 'd2_hero', 'd3_hero', 'd4_hero', 'd5_hero']\n",
    "hero_sr = train_df['r1_hero']\n",
    "for label in hero_features:\n",
    "    hero_sr.append(train_df[label])\n",
    "print(hero_sr.nunique())\n",
    "print(hero_sr.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Забавный момент... в обучающей выборке оказались не ве герои, из-за чегоя не смог воспользоваться мешком слов. Но выход легко нашёлся - можно было просто взять максимальный индекс героя.. И тогда он скорее всего совпадает с их количеством"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь надо добавить эти признаки обратно, вопспользуемся предложенным кодом"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_time</th>\n",
       "      <th>r1_level</th>\n",
       "      <th>r1_xp</th>\n",
       "      <th>r1_gold</th>\n",
       "      <th>r1_lh</th>\n",
       "      <th>r1_kills</th>\n",
       "      <th>r1_deaths</th>\n",
       "      <th>r1_items</th>\n",
       "      <th>r2_level</th>\n",
       "      <th>r2_xp</th>\n",
       "      <th>...</th>\n",
       "      <th>102</th>\n",
       "      <th>103</th>\n",
       "      <th>104</th>\n",
       "      <th>105</th>\n",
       "      <th>106</th>\n",
       "      <th>107</th>\n",
       "      <th>108</th>\n",
       "      <th>109</th>\n",
       "      <th>110</th>\n",
       "      <th>111</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>match_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430198770</td>\n",
       "      <td>5</td>\n",
       "      <td>2098</td>\n",
       "      <td>1489</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>842</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1430220345</td>\n",
       "      <td>4</td>\n",
       "      <td>1188</td>\n",
       "      <td>1033</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>1596</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1430227081</td>\n",
       "      <td>4</td>\n",
       "      <td>1319</td>\n",
       "      <td>1270</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>1314</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1430263531</td>\n",
       "      <td>4</td>\n",
       "      <td>1779</td>\n",
       "      <td>1056</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>539</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1430282290</td>\n",
       "      <td>4</td>\n",
       "      <td>1431</td>\n",
       "      <td>1090</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>629</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 203 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          start_time  r1_level  r1_xp  r1_gold  r1_lh  r1_kills  r1_deaths  \\\n",
       "match_id                                                                     \n",
       "0         1430198770         5   2098     1489     20         0          0   \n",
       "1         1430220345         4   1188     1033      9         0          1   \n",
       "2         1430227081         4   1319     1270     22         0          0   \n",
       "3         1430263531         4   1779     1056     14         0          0   \n",
       "4         1430282290         4   1431     1090      8         1          0   \n",
       "\n",
       "          r1_items  r2_level  r2_xp ...   102  103  104  105  106  107  108  \\\n",
       "match_id                            ...                                       \n",
       "0                7         3    842 ...     0    0    1    0    0    0    0   \n",
       "1               12         4   1596 ...     0    0    0    0    0    0    0   \n",
       "2               12         3   1314 ...     0    0    0    0    0    0    0   \n",
       "3                5         2    539 ...     0    0    0    0    0    0    0   \n",
       "4                8         2    629 ...     0    0    0    0    0    0    0   \n",
       "\n",
       "          109  110  111  \n",
       "match_id                 \n",
       "0           0    0    0  \n",
       "1           0    0    0  \n",
       "2           0    0    0  \n",
       "3           0    0    0  \n",
       "4           0    0    0  \n",
       "\n",
       "[5 rows x 203 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pick = np.zeros((train_df.shape[0], 112))\n",
    "\n",
    "for i, match_id in enumerate(train_df.index):\n",
    "    for p in range(5):\n",
    "        X_pick[i, train_df.ix[match_id, 'r%d_hero' % (p+1)]-1] = 1\n",
    "        X_pick[i, train_df.ix[match_id, 'd%d_hero' % (p+1)]-1] = -1\n",
    "        \n",
    "#преобразуем нашу матрицу к датафрейму и склеим\n",
    "pick_df = pd.DataFrame(X_pick, index=train_df.index).fillna(0).astype('int')\n",
    "full_train_df = decimal_df.join(pick_df)\n",
    "full_train_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s] 10%|█         | 1/10 [00:19<02:54, 19.38s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.01 0.7520283409\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 2/10 [00:41<02:40, 20.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0215443469003 0.751742631573\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 3/10 [01:02<02:23, 20.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0464158883361 0.751948956695\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 4/10 [01:23<02:03, 20.63s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1 0.751850101381\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 5/10 [01:44<01:44, 20.82s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.215443469003 0.751918201533\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 6/10 [02:05<01:23, 20.81s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.464158883361 0.751955120318\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 7/10 [02:27<01:03, 21.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0 0.752010772839\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 8/10 [02:48<00:42, 21.19s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.15443469003 0.751781101435\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 9/10 [03:09<00:21, 21.20s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.64158883361 0.75193296372\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 10/10 [03:30<00:00, 21.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.0 0.7515363446\n",
      "CPU times: user 3min 22s, sys: 23.6 s, total: 3min 46s\n",
      "Wall time: 3min 31s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X_full_train = StandardScaler().fit_transform(full_train_df)\n",
    "\n",
    "for c in tqdm(C_space):\n",
    "    clf = LogisticRegression(C=c)\n",
    "    print(c, cross_val_score(clf, X_full_train, y_train, scoring='roc_auc', cv=kf).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как видно отсюда, у нас удалось улучшить качество на кросс валидации до 0.7519 Это произошло ввиду того, что признаки оказались сильными:) И теперь они не случайные, а коррелируют с целевой переменной"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Грустный финал</b>\n",
    "<br><br> Как оказалось зря с самого начала я не вставил тестовую часть:) Придётся быстренько её видоизменить"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_time</th>\n",
       "      <th>r1_level</th>\n",
       "      <th>r1_xp</th>\n",
       "      <th>r1_gold</th>\n",
       "      <th>r1_lh</th>\n",
       "      <th>r1_kills</th>\n",
       "      <th>r1_deaths</th>\n",
       "      <th>r1_items</th>\n",
       "      <th>r2_level</th>\n",
       "      <th>r2_xp</th>\n",
       "      <th>...</th>\n",
       "      <th>102</th>\n",
       "      <th>103</th>\n",
       "      <th>104</th>\n",
       "      <th>105</th>\n",
       "      <th>106</th>\n",
       "      <th>107</th>\n",
       "      <th>108</th>\n",
       "      <th>109</th>\n",
       "      <th>110</th>\n",
       "      <th>111</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>match_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1430287923</td>\n",
       "      <td>4</td>\n",
       "      <td>1103</td>\n",
       "      <td>1089</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>1183</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1430293357</td>\n",
       "      <td>2</td>\n",
       "      <td>556</td>\n",
       "      <td>570</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>1194</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1430301774</td>\n",
       "      <td>2</td>\n",
       "      <td>751</td>\n",
       "      <td>808</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>421</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1430323933</td>\n",
       "      <td>3</td>\n",
       "      <td>708</td>\n",
       "      <td>903</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>672</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1430331112</td>\n",
       "      <td>4</td>\n",
       "      <td>1259</td>\n",
       "      <td>661</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>1703</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 203 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          start_time  r1_level  r1_xp  r1_gold  r1_lh  r1_kills  r1_deaths  \\\n",
       "match_id                                                                     \n",
       "6         1430287923         4   1103     1089      8         0          1   \n",
       "7         1430293357         2    556      570      1         0          0   \n",
       "10        1430301774         2    751      808      1         0          0   \n",
       "13        1430323933         3    708      903      1         1          1   \n",
       "16        1430331112         4   1259      661      4         0          0   \n",
       "\n",
       "          r1_items  r2_level  r2_xp ...   102  103  104  105  106  107  108  \\\n",
       "match_id                            ...                                       \n",
       "6                9         3   1183 ...     0    0    0    0    0    0    0   \n",
       "7                9         4   1194 ...     0    0    0    0    0    0    0   \n",
       "10              13         2    421 ...     0    0    0    0    0    0    0   \n",
       "13              11         2    672 ...     0    0    0    0    0    0    0   \n",
       "16               9         5   1703 ...     0    0    0    0    0    0    0   \n",
       "\n",
       "          109  110  111  \n",
       "match_id                 \n",
       "6           0    0    0  \n",
       "7           0    0    0  \n",
       "10          0    0    1  \n",
       "13         -1    0    0  \n",
       "16          0    0    0  \n",
       "\n",
       "[5 rows x 203 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = test_df.drop(categorial_features, axis=1)\n",
    "X_test_pick = np.zeros((test_df.shape[0], 112))\n",
    "\n",
    "for i, match_id in enumerate(test_df.index):\n",
    "    for p in range(5):\n",
    "        X_test_pick[i, test_data.ix[match_id, 'r%d_hero' % (p+1)]-1] = 1\n",
    "        X_test_pick[i, test_data.ix[match_id, 'd%d_hero' % (p+1)]-1] = -1\n",
    "        \n",
    "#преобразуем нашу матрицу к датафрейму и склеим\n",
    "pick_test_df = pd.DataFrame(X_test_pick, index=test_df.index).fillna(0).astype('int')\n",
    "full_test_df = test_df.join(pick_test_df)\n",
    "full_test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.82494991,  0.75850176,  0.18702219,  0.85934659,  0.23762798])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_full_test = StandardScaler().fit_transform(full_test_df)\n",
    "clf = LogisticRegression(C=0.5)\n",
    "clf.fit(X_full_train, y_train)\n",
    "pred = clf.predict_proba(X_full_test)[:, 1]\n",
    "pred[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_submission = pd.read_csv('data/sample_submission.csv', index_col='match_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>radiant_win</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>match_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.518503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.518503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.518503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.518503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.518503</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          radiant_win\n",
       "match_id             \n",
       "6            0.518503\n",
       "7            0.518503\n",
       "10           0.518503\n",
       "13           0.518503\n",
       "16           0.518503"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_submission.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submission = sample_submission.copy()\n",
    "submission['radiant_win'] = pred\n",
    "submission.to_csv('baseline.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дмитрий Торшин\t0.75522\t- Mon, 27 Mar 2017 11:54:49"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.00868961197807 0.996484254327\n"
     ]
    }
   ],
   "source": [
    "print(pred.min(), pred.max())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}