1 - TikTok Claims Exploration.ipynb

{
    "metadata": {
        "kernelspec": {
            "name": "python3",
            "display_name": "Python 3",
            "language": "python"
        },
        "language_info": {
            "name": "python",
            "version": "3.10.4",
            "mimetype": "text/x-python",
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "pygments_lexer": "ipython3",
            "nbconvert_exporter": "python",
            "file_extension": ".py"
        },
        "colab": {
            "provenance": []
        }
    },
    "nbformat_minor": 2,
    "nbformat": 4,
    "cells": [
        {
            "cell_type": "markdown",
            "source": [
                "# **TikTok Claims Exploration**\n",
                "**The goal** of this first notebook is to explore and understand the 2019 TikTok Claims data aand present our findings.\n",
                "**Part 1:** Load the Data\n",
                "- Create a pandas dataframe for data learning, future exploratory data analysis (EDA), and statistical analysis.\n",
                "**Part 2:** Inspect and Analyze the Data\n",
                "- Compile summary information about the data to inform next steps.\n",
                "    \n",
                "- Use insights from our examination of the summary data to guide deeper investigation into specific variables.\n",
                "    \n",
                "**Part 3:** Interpret the Data\n",
                "- Summarize our examination of the summary data and propose deeper investigation into specific variables."
            ],
            "metadata": {
                "id": "rgSbVJvomcVa",
                "azdata_cell_guid": "e793d795-8632-4a30-b7f4-b59ce4720461"
            },
            "attachments": {}
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **1: Load the Data**\n",
                "\n",
                "### **Build dataframe**"
            ],
            "metadata": {
                "id": "Uqtf9i8bOYwO",
                "azdata_cell_guid": "f4c839da-17c3-4c17-9064-c005ba6fcd68"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Import packages\n",
                "import pandas as pd\n",
                "import numpy as np\n",
                "\n",
                "# Load data\n",
                "data = pd.read_csv(\"tiktok_dataset.csv\")\n",
                ""
            ],
            "metadata": {
                "id": "0mlHltbrOYjf",
                "azdata_cell_guid": "413428bf-fd83-4044-8749-9fd9218204b0",
                "language": "python"
            },
            "outputs": [],
            "execution_count": 1
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **2: Inspect and Analyze the Data**\n",
                "\n",
                "### **Task 2a. Compile summary info about the dataframe using:**\n",
                "\n",
                "1. df.head()\n",
                "2. df.info()\n",
                "3. df.describe()"
            ],
            "metadata": {
                "id": "-V80mmNnOYUM",
                "azdata_cell_guid": "f21fcce6-4e5c-4c84-827c-774a46340b20"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Display and examine the first 10 rows of the dataframe\r\n",
                "data.head(10)\r\n",
                ""
            ],
            "metadata": {
                "id": "rKZWwAu2OYEe",
                "azdata_cell_guid": "22d02ede-30ec-48e4-a20f-c373444c8dc9",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "    # claim_status    video_id  video_duration_sec   \n0   1        claim  7017666017                  59  \\\n1   2        claim  4014381136                  32   \n2   3        claim  9859838091                  31   \n3   4        claim  1866847991                  25   \n4   5        claim  7105231098                  19   \n5   6        claim  8972200955                  35   \n6   7        claim  4958886992                  16   \n7   8        claim  2270982263                  41   \n8   9        claim  5235769692                  50   \n9  10        claim  4660861094                  45   \n\n                            video_transcription_text verified_status   \n0  someone shared with me that drone deliveries a...    not verified  \\\n1  someone shared with me that there are more mic...    not verified   \n2  someone shared with me that american industria...    not verified   \n3  someone shared with me that the metro of st. p...    not verified   \n4  someone shared with me that the number of busi...    not verified   \n5  someone shared with me that gross domestic pro...    not verified   \n6  someone shared with me that elvis presley has ...    not verified   \n7  someone shared with me that the best selling s...    not verified   \n8  someone shared with me that about half of the ...    not verified   \n9  someone shared with me that it would take a 50...        verified   \n\n  author_ban_status  video_view_count  video_like_count  video_share_count   \n0      under review          343296.0           19425.0              241.0  \\\n1            active          140877.0           77355.0            19034.0   \n2            active          902185.0           97690.0             2858.0   \n3            active          437506.0          239954.0            34812.0   \n4            active           56167.0           34987.0             4110.0   \n5      under review          336647.0          175546.0            62303.0   \n6            active          750345.0          486192.0           193911.0   \n7            active          547532.0            1072.0               50.0   \n8            active           24819.0           10160.0             1050.0   \n9            active          931587.0          171051.0            67739.0   \n\n   video_download_count  video_comment_count  \n0                   1.0                  0.0  \n1                1161.0                684.0  \n2                 833.0                329.0  \n3                1234.0                584.0  \n4                 547.0                152.0  \n5                4293.0               1857.0  \n6                8616.0               5446.0  \n7                  22.0                 11.0  \n8                  53.0                 27.0  \n9                4104.0               2540.0  ",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>#</th>\n      <th>claim_status</th>\n      <th>video_id</th>\n      <th>video_duration_sec</th>\n      <th>video_transcription_text</th>\n      <th>verified_status</th>\n      <th>author_ban_status</th>\n      <th>video_view_count</th>\n      <th>video_like_count</th>\n      <th>video_share_count</th>\n      <th>video_download_count</th>\n      <th>video_comment_count</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>claim</td>\n      <td>7017666017</td>\n      <td>59</td>\n      <td>someone shared with me that drone deliveries a...</td>\n      <td>not verified</td>\n      <td>under review</td>\n      <td>343296.0</td>\n      <td>19425.0</td>\n      <td>241.0</td>\n      <td>1.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>claim</td>\n      <td>4014381136</td>\n      <td>32</td>\n      <td>someone shared with me that there are more mic...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>140877.0</td>\n      <td>77355.0</td>\n      <td>19034.0</td>\n      <td>1161.0</td>\n      <td>684.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>claim</td>\n      <td>9859838091</td>\n      <td>31</td>\n      <td>someone shared with me that american industria...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>902185.0</td>\n      <td>97690.0</td>\n      <td>2858.0</td>\n      <td>833.0</td>\n      <td>329.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4</td>\n      <td>claim</td>\n      <td>1866847991</td>\n      <td>25</td>\n      <td>someone shared with me that the metro of st. p...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>437506.0</td>\n      <td>239954.0</td>\n      <td>34812.0</td>\n      <td>1234.0</td>\n      <td>584.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5</td>\n      <td>claim</td>\n      <td>7105231098</td>\n      <td>19</td>\n      <td>someone shared with me that the number of busi...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>56167.0</td>\n      <td>34987.0</td>\n      <td>4110.0</td>\n      <td>547.0</td>\n      <td>152.0</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>6</td>\n      <td>claim</td>\n      <td>8972200955</td>\n      <td>35</td>\n      <td>someone shared with me that gross domestic pro...</td>\n      <td>not verified</td>\n      <td>under review</td>\n      <td>336647.0</td>\n      <td>175546.0</td>\n      <td>62303.0</td>\n      <td>4293.0</td>\n      <td>1857.0</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>7</td>\n      <td>claim</td>\n      <td>4958886992</td>\n      <td>16</td>\n      <td>someone shared with me that elvis presley has ...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>750345.0</td>\n      <td>486192.0</td>\n      <td>193911.0</td>\n      <td>8616.0</td>\n      <td>5446.0</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>8</td>\n      <td>claim</td>\n      <td>2270982263</td>\n      <td>41</td>\n      <td>someone shared with me that the best selling s...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>547532.0</td>\n      <td>1072.0</td>\n      <td>50.0</td>\n      <td>22.0</td>\n      <td>11.0</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>9</td>\n      <td>claim</td>\n      <td>5235769692</td>\n      <td>50</td>\n      <td>someone shared with me that about half of the ...</td>\n      <td>not verified</td>\n      <td>active</td>\n      <td>24819.0</td>\n      <td>10160.0</td>\n      <td>1050.0</td>\n      <td>53.0</td>\n      <td>27.0</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>10</td>\n      <td>claim</td>\n      <td>4660861094</td>\n      <td>45</td>\n      <td>someone shared with me that it would take a 50...</td>\n      <td>verified</td>\n      <td>active</td>\n      <td>931587.0</td>\n      <td>171051.0</td>\n      <td>67739.0</td>\n      <td>4104.0</td>\n      <td>2540.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 4,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 4
        },
        {
            "cell_type": "code",
            "source": [
                "# Get summary info\n",
                "data.info()\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/"
                },
                "executionInfo": {
                    "elapsed": 10,
                    "status": "ok",
                    "timestamp": 1683669872548,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "TWbYPEFR8N39",
                "outputId": "8df0e29a-bb29-4f4d-d748-15ac5a1ae174",
                "azdata_cell_guid": "56b97de5-1e7c-4b59-8909-039051eb99b9",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "text": "<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 19382 entries, 0 to 19381\nData columns (total 12 columns):\n #   Column                    Non-Null Count  Dtype  \n---  ------                    --------------  -----  \n 0   #                         19382 non-null  int64  \n 1   claim_status              19084 non-null  object \n 2   video_id                  19382 non-null  int64  \n 3   video_duration_sec        19382 non-null  int64  \n 4   video_transcription_text  19084 non-null  object \n 5   verified_status           19382 non-null  object \n 6   author_ban_status         19382 non-null  object \n 7   video_view_count          19084 non-null  float64\n 8   video_like_count          19084 non-null  float64\n 9   video_share_count         19084 non-null  float64\n 10  video_download_count      19084 non-null  float64\n 11  video_comment_count       19084 non-null  float64\ndtypes: float64(5), int64(3), object(4)\nmemory usage: 1.8+ MB\n",
                    "output_type": "stream"
                }
            ],
            "execution_count": 5
        },
        {
            "cell_type": "code",
            "source": [
                "# Get summary statistics\n",
                "data.describe()\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 364
                },
                "executionInfo": {
                    "elapsed": 15,
                    "status": "ok",
                    "timestamp": 1683669875217,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "m_qVdttU8Ofj",
                "outputId": "8d76544e-ecb7-41de-80e3-d9a4b5bfe01e",
                "azdata_cell_guid": "c041d10e-3495-42a3-9256-03bde1f6bed0",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "                  #      video_id  video_duration_sec  video_view_count   \ncount  19382.000000  1.938200e+04        19382.000000      19084.000000  \\\nmean    9691.500000  5.627454e+09           32.421732     254708.558688   \nstd     5595.245794  2.536440e+09           16.229967     322893.280814   \nmin        1.000000  1.234959e+09            5.000000         20.000000   \n25%     4846.250000  3.430417e+09           18.000000       4942.500000   \n50%     9691.500000  5.618664e+09           32.000000       9954.500000   \n75%    14536.750000  7.843960e+09           47.000000     504327.000000   \nmax    19382.000000  9.999873e+09           60.000000     999817.000000   \n\n       video_like_count  video_share_count  video_download_count   \ncount      19084.000000       19084.000000          19084.000000  \\\nmean       84304.636030       16735.248323           1049.429627   \nstd       133420.546814       32036.174350           2004.299894   \nmin            0.000000           0.000000              0.000000   \n25%          810.750000         115.000000              7.000000   \n50%         3403.500000         717.000000             46.000000   \n75%       125020.000000       18222.000000           1156.250000   \nmax       657830.000000      256130.000000          14994.000000   \n\n       video_comment_count  \ncount         19084.000000  \nmean            349.312146  \nstd             799.638865  \nmin               0.000000  \n25%               1.000000  \n50%               9.000000  \n75%             292.000000  \nmax            9599.000000  ",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>#</th>\n      <th>video_id</th>\n      <th>video_duration_sec</th>\n      <th>video_view_count</th>\n      <th>video_like_count</th>\n      <th>video_share_count</th>\n      <th>video_download_count</th>\n      <th>video_comment_count</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>19382.000000</td>\n      <td>1.938200e+04</td>\n      <td>19382.000000</td>\n      <td>19084.000000</td>\n      <td>19084.000000</td>\n      <td>19084.000000</td>\n      <td>19084.000000</td>\n      <td>19084.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>9691.500000</td>\n      <td>5.627454e+09</td>\n      <td>32.421732</td>\n      <td>254708.558688</td>\n      <td>84304.636030</td>\n      <td>16735.248323</td>\n      <td>1049.429627</td>\n      <td>349.312146</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>5595.245794</td>\n      <td>2.536440e+09</td>\n      <td>16.229967</td>\n      <td>322893.280814</td>\n      <td>133420.546814</td>\n      <td>32036.174350</td>\n      <td>2004.299894</td>\n      <td>799.638865</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>1.000000</td>\n      <td>1.234959e+09</td>\n      <td>5.000000</td>\n      <td>20.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>4846.250000</td>\n      <td>3.430417e+09</td>\n      <td>18.000000</td>\n      <td>4942.500000</td>\n      <td>810.750000</td>\n      <td>115.000000</td>\n      <td>7.000000</td>\n      <td>1.000000</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>9691.500000</td>\n      <td>5.618664e+09</td>\n      <td>32.000000</td>\n      <td>9954.500000</td>\n      <td>3403.500000</td>\n      <td>717.000000</td>\n      <td>46.000000</td>\n      <td>9.000000</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>14536.750000</td>\n      <td>7.843960e+09</td>\n      <td>47.000000</td>\n      <td>504327.000000</td>\n      <td>125020.000000</td>\n      <td>18222.000000</td>\n      <td>1156.250000</td>\n      <td>292.000000</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>19382.000000</td>\n      <td>9.999873e+09</td>\n      <td>60.000000</td>\n      <td>999817.000000</td>\n      <td>657830.000000</td>\n      <td>256130.000000</td>\n      <td>14994.000000</td>\n      <td>9599.000000</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 6,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 6
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observations:**\n",
                "\n",
                "1. The dataframe contains a collection of categorical, text, and numerical data. Each row represents a distinct TikTok video that presents either a claim or an opinion and the accompanying metadata about that video.\n",
                "\n",
                "1. The dataframe contains five float64s, three int64s, and four objects. There are 19,382 observations, but some of the variables are missing values, including claim status, the video transcripton, and all of the count variables.\n",
                "\n",
                "1. Many of the count variables seem to have outliers at the high end of the distribution. They have very large standard deviations and maximum values that are very high compared to their quartile values."
            ],
            "metadata": {
                "id": "b9jWzZWImETT",
                "azdata_cell_guid": "eaa34117-3793-4e91-83b3-ba8ec8f68046"
            },
            "attachments": {}
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Task 2b. Investigate the Primary Variables**\n",
                "\n",
                "### <span style=\"font-size: 14px; color: var(--vscode-foreground);\">Our ultimate objective is to use machine learning to classify videos as either <i>claims </i>or <i>opinions</i>. Therefore we start by examining the </span> `claim_status` <span style=\"font-size: 14px; color: var(--vscode-foreground);\"> variable, specifically determining how many videos are in each different claim status.</span>"
            ],
            "metadata": {
                "id": "gYx1emvno7U_",
                "azdata_cell_guid": "c9d3b055-51cf-4bd0-acc3-a2ad46c85ee2"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Return the different values and counts for the claim_status column\n",
                "data['claim_status'].value_counts()\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/"
                },
                "executionInfo": {
                    "elapsed": 226,
                    "status": "ok",
                    "timestamp": 1683669879919,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "Qy3rkaCLCEUy",
                "outputId": "c32a26b1-6a3d-4850-c33d-f0e48ba70f67",
                "azdata_cell_guid": "f3b58409-a1ab-4472-9f0a-12580d9f1e85",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "claim_status\nclaim      9608\nopinion    9476\nName: count, dtype: int64"
                    },
                    "metadata": {},
                    "execution_count": 7,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 7
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observation****:** The counts of each claim status are quite balanced."
            ],
            "metadata": {
                "id": "ThFgtYRWCSiz",
                "azdata_cell_guid": "e31e14fd-2832-4a25-9abb-22bf6b3f044a"
            },
            "attachments": {}
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Task 2c. Deeper Investigation of Primary Variables**\n",
                "\n",
                "Next, we examine the _engagement trends_ associated with each different claim status.\n",
                "\n",
                "We apply a Boolean mask to filter the data according to claim status, then calculate the mean and median view counts for each claim status."
            ],
            "metadata": {
                "id": "UUQEcbsXCXfq",
                "azdata_cell_guid": "74eb9036-5daa-4249-9342-ad960978d2cf"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# What is the average view count of videos with \"claim\" status?\n",
                "claims = data[data['claim_status'] == 'claim']\n",
                "print('Mean view count claims:', claims['video_view_count'].mean())\n",
                "print('Median view count claims:', claims['video_view_count'].median())\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/"
                },
                "executionInfo": {
                    "elapsed": 211,
                    "status": "ok",
                    "timestamp": 1683669882093,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "Bn80TFLxCdfw",
                "outputId": "4eee58a5-78d7-4833-a4b8-af1d306efadb",
                "azdata_cell_guid": "ec471844-45df-42af-85ea-8e7f738976a6",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "text": "Mean view count claims: 501029.4527477102\nMedian view count claims: 501555.0\n",
                    "output_type": "stream"
                }
            ],
            "execution_count": 9
        },
        {
            "cell_type": "code",
            "source": [
                "# What is the average view count of videos with \"opinion\" status?\n",
                "opinions = data[data['claim_status'] == 'opinion']\n",
                "print('Mean view count opinions:', opinions['video_view_count'].mean())\n",
                "print('Median view count opinions:', opinions['video_view_count'].median())\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/"
                },
                "executionInfo": {
                    "elapsed": 163,
                    "status": "ok",
                    "timestamp": 1683669883382,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "8WeKRj3GCdPZ",
                "outputId": "47c74bf2-4f59-4cd0-ea6c-ec0883f32087",
                "azdata_cell_guid": "dff0c9e3-50a4-4623-b859-4c01fab98a50",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "text": "Mean view count opinions: 4956.43224989447\nMedian view count opinions: 4953.0\n",
                    "output_type": "stream"
                }
            ],
            "execution_count": 10
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observation****:** The mean and the median within each claim category are close to one another, but there is a vast discrepancy between view counts for videos labeled as claims and videos labeled as opinions.\n",
                "\n",
                "### **Now, we examine trends associated with the ban status of the author.**\n",
                "\n",
                "We use `groupby()` to calculate how many videos are in each combination of categories of claim status and author ban status."
            ],
            "metadata": {
                "id": "oGsRYN18DBFt",
                "azdata_cell_guid": "179259f0-590c-46fa-827d-7b9ee2f066ba"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Get counts for each group combination of claim status and author ban status\n",
                "data.groupby(['claim_status', 'author_ban_status']).count()[['#']]\n",
                ""
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 269
                },
                "executionInfo": {
                    "elapsed": 205,
                    "status": "ok",
                    "timestamp": 1683669885198,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "WkqyKV4LDJZb",
                "outputId": "5b3417d7-c15d-46aa-b5cf-dc245bbc2417",
                "azdata_cell_guid": "0018029c-ed3e-4b67-8f38-dec2bcaf3264",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "                                   #\nclaim_status author_ban_status      \nclaim        active             6566\n             banned             1439\n             under review       1603\nopinion      active             8817\n             banned              196\n             under review        463",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th></th>\n      <th>#</th>\n    </tr>\n    <tr>\n      <th>claim_status</th>\n      <th>author_ban_status</th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th rowspan=\"3\" valign=\"top\">claim</th>\n      <th>active</th>\n      <td>6566</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>1439</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>1603</td>\n    </tr>\n    <tr>\n      <th rowspan=\"3\" valign=\"top\">opinion</th>\n      <th>active</th>\n      <td>8817</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>196</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>463</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 11,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 11
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observations:**\n",
                "\n",
                "1. There are many more claim videos with banned authors than there are opinion videos with banned authors. This could mean a number of things, including the possibilities that:\n",
                "    \n",
                "    - Claim videos are more strictly policed than opinion videos\n",
                "    - Authors must comply with a stricter set of rules if they post a claim than if they post an opinion\n",
                "2. There's no way of knowing if claim videos are inherently more likely than opinion videos to result in author bans, or if authors who post claim videos are more likely to post videos that violate terms of service.\n",
                "    \n",
                "3. While we can draw conclusions about banned/active authors, we **cannot** draw conclusions about banned videos. There's no way of determining whether a particular video _caused_ the ban, and banned authors could have posted videos that complied with the terms of service.\n",
                "    \n",
                "\n",
                "### **Next, we look at `author_ban_status` to continue investigating engagement levels**"
            ],
            "metadata": {
                "id": "XYAQB4Q-DJ7Y",
                "azdata_cell_guid": "7f16e79d-9cb3-4df4-8f14-989ca2a2476e"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Calculate the median video share count of each author ban status\n",
                "data.groupby(['author_ban_status']).agg(\n",
                "    {'video_view_count': ['mean', 'median'],\n",
                "     'video_like_count': ['mean', 'median'],\n",
                "     'video_share_count': ['mean', 'median']})\n",
                "     "
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 206
                },
                "executionInfo": {
                    "elapsed": 152,
                    "status": "ok",
                    "timestamp": 1683669887816,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "MwBNYOe2DnMb",
                "outputId": "89dd73f9-4308-4a8b-c0e2-14a8be251033",
                "azdata_cell_guid": "cd2359c8-1969-4764-b342-12ae2abfb605",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "                  video_view_count           video_like_count             \n                              mean    median             mean    median   \nauthor_ban_status                                                         \nactive               215927.039524    8616.0     71036.533836    2222.0  \\\nbanned               445845.439144  448201.0    153017.236697  105573.0   \nunder review         392204.836399  365245.5    128718.050339   71204.5   \n\n                  video_share_count           \n                               mean   median  \nauthor_ban_status                             \nactive                 14111.466164    437.0  \nbanned                 29998.942508  14468.0  \nunder review           25774.696999   9444.0  ",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead tr th {\n        text-align: left;\n    }\n\n    .dataframe thead tr:last-of-type th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr>\n      <th></th>\n      <th colspan=\"2\" halign=\"left\">video_view_count</th>\n      <th colspan=\"2\" halign=\"left\">video_like_count</th>\n      <th colspan=\"2\" halign=\"left\">video_share_count</th>\n    </tr>\n    <tr>\n      <th></th>\n      <th>mean</th>\n      <th>median</th>\n      <th>mean</th>\n      <th>median</th>\n      <th>mean</th>\n      <th>median</th>\n    </tr>\n    <tr>\n      <th>author_ban_status</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>active</th>\n      <td>215927.039524</td>\n      <td>8616.0</td>\n      <td>71036.533836</td>\n      <td>2222.0</td>\n      <td>14111.466164</td>\n      <td>437.0</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>445845.439144</td>\n      <td>448201.0</td>\n      <td>153017.236697</td>\n      <td>105573.0</td>\n      <td>29998.942508</td>\n      <td>14468.0</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>392204.836399</td>\n      <td>365245.5</td>\n      <td>128718.050339</td>\n      <td>71204.5</td>\n      <td>25774.696999</td>\n      <td>9444.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 12,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 12
        },
        {
            "cell_type": "code",
            "source": [
                "# What's the median video share count of each author ban status?\n",
                "data.groupby(['author_ban_status']).median(numeric_only=True)[\n",
                "    ['video_share_count']]\n",
                "    "
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 175
                },
                "executionInfo": {
                    "elapsed": 165,
                    "status": "ok",
                    "timestamp": 1683669889718,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "eGBp2UpFDm6p",
                "outputId": "56f0274a-618f-44ae-9d1f-346a3920d8a3",
                "azdata_cell_guid": "0e7875c0-76d6-4c22-bd2c-80ed15e05451",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "                   video_share_count\nauthor_ban_status                   \nactive                         437.0\nbanned                       14468.0\nunder review                  9444.0",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>video_share_count</th>\n    </tr>\n    <tr>\n      <th>author_ban_status</th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>active</th>\n      <td>437.0</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>14468.0</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>9444.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 13,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 13
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observation:** Banned authors have a median share count that's 33 times the median share count of active authors!\n",
                "\n",
                "### **We explore this in more depth. . .**\n",
                "\n",
                "Use `groupby()` to group the data by `author_ban_status`, then use `agg()` to get the count, mean, and median of each of the following columns:\n",
                "\n",
                "- `video_view_count`\n",
                "- `video_like_count`\n",
                "- `video_share_count`\n",
                "\n",
                "Remember, the argument for the `agg()` function is a dictionary whose keys are columns. The values for each column are a list of the calculations you want to perform."
            ],
            "metadata": {
                "id": "FtZV-FmRD-NU",
                "azdata_cell_guid": "70616f69-26b1-4288-aefe-0307a1a4d621"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "\n",
                "data.groupby(['author_ban_status']).agg(\n",
                "    {'video_view_count': ['count', 'mean', 'median'],\n",
                "     'video_like_count': ['count', 'mean', 'median'],\n",
                "     'video_share_count': ['count', 'mean', 'median']\n",
                "     })"
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 206
                },
                "executionInfo": {
                    "elapsed": 217,
                    "status": "ok",
                    "timestamp": 1683669892303,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "X6o-LkIWEngt",
                "outputId": "5e6e1d44-c3ff-4474-db07-50c0ddc78ddf",
                "azdata_cell_guid": "008eca20-6840-4804-8c44-e2654bf59041",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": "                  video_view_count                          video_like_count   \n                             count           mean    median            count   \nauthor_ban_status                                                              \nactive                       15383  215927.039524    8616.0            15383  \\\nbanned                        1635  445845.439144  448201.0             1635   \nunder review                  2066  392204.836399  365245.5             2066   \n\n                                           video_share_count                 \n                            mean    median             count          mean   \nauthor_ban_status                                                            \nactive              71036.533836    2222.0             15383  14111.466164  \\\nbanned             153017.236697  105573.0              1635  29998.942508   \nunder review       128718.050339   71204.5              2066  25774.696999   \n\n                            \n                    median  \nauthor_ban_status           \nactive               437.0  \nbanned             14468.0  \nunder review        9444.0  ",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead tr th {\n        text-align: left;\n    }\n\n    .dataframe thead tr:last-of-type th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr>\n      <th></th>\n      <th colspan=\"3\" halign=\"left\">video_view_count</th>\n      <th colspan=\"3\" halign=\"left\">video_like_count</th>\n      <th colspan=\"3\" halign=\"left\">video_share_count</th>\n    </tr>\n    <tr>\n      <th></th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n    </tr>\n    <tr>\n      <th>author_ban_status</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>active</th>\n      <td>15383</td>\n      <td>215927.039524</td>\n      <td>8616.0</td>\n      <td>15383</td>\n      <td>71036.533836</td>\n      <td>2222.0</td>\n      <td>15383</td>\n      <td>14111.466164</td>\n      <td>437.0</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>1635</td>\n      <td>445845.439144</td>\n      <td>448201.0</td>\n      <td>1635</td>\n      <td>153017.236697</td>\n      <td>105573.0</td>\n      <td>1635</td>\n      <td>29998.942508</td>\n      <td>14468.0</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>2066</td>\n      <td>392204.836399</td>\n      <td>365245.5</td>\n      <td>2066</td>\n      <td>128718.050339</td>\n      <td>71204.5</td>\n      <td>2066</td>\n      <td>25774.696999</td>\n      <td>9444.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {},
                    "execution_count": 14,
                    "output_type": "execute_result"
                }
            ],
            "execution_count": 14
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observations****:**\n",
                "\n",
                "1. Banned authors and those under review get far more views, likes, and shares than active authors.\n",
                "1. In most groups, the mean is much greater than the median, which indicates that there are some videos with very high engagement counts.\n",
                "\n",
                "### **Now, we create three new columns to help better understand engagement rates:**\n",
                "### \n",
                "- `likes_per_view`: represents the number of likes divided by the number of views for each video\n",
                "- `comments_per_view`: represents the number of comments divided by the number of views for each video\n",
                "- `shares_per_view`: represents the number of shares divided by the number of views for each video"
            ],
            "metadata": {
                "id": "tqKMv2P1EuDh",
                "azdata_cell_guid": "8a42f0d7-13ca-4ebe-9a33-ff6afaf86892"
            },
            "attachments": {}
        },
        {
            "cell_type": "code",
            "source": [
                "# Create a likes_per_view column\n",
                "data['likes_per_view'] = data['video_like_count'] / data['video_view_count']\n",
                "\n",
                "# Create a comments_per_view column\n",
                "data['comments_per_view'] = data['video_comment_count'] / data['video_view_count']\n",
                "\n",
                "# Create a shares_per_view column\n",
                "data['shares_per_view'] = data['video_share_count'] / data['video_view_count']\n",
                ""
            ],
            "metadata": {
                "id": "JT6a39aXEymQ",
                "azdata_cell_guid": "c7d05fe2-998f-466a-b802-cf57ef4086c3",
                "language": "python"
            },
            "outputs": [],
            "execution_count": 16
        },
        {
            "cell_type": "markdown",
            "source": [
                "Use `groupby()` to compile the information in each of the three newly created columns for each combination of categories of claim status and author ban status, then use `agg()` to calculate the count, the mean, and the median of each group."
            ],
            "metadata": {
                "id": "8zgvBukjFbWp",
                "azdata_cell_guid": "b238ac02-dcf6-46ba-a946-8067f309b03c"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "data.groupby(['claim_status', 'author_ban_status']).agg(\n",
                "    {'likes_per_view': ['count', 'mean', 'median'],\n",
                "     'comments_per_view': ['count', 'mean', 'median'],\n",
                "     'shares_per_view': ['count', 'mean', 'median']})"
            ],
            "metadata": {
                "colab": {
                    "base_uri": "https://localhost:8080/",
                    "height": 300
                },
                "executionInfo": {
                    "elapsed": 235,
                    "status": "ok",
                    "timestamp": 1683669897536,
                    "user": {
                        "displayName": "Victoria Lansdown",
                        "userId": "02417259790472296449"
                    },
                    "user_tz": 420
                },
                "id": "ySVc5qk5Fb55",
                "outputId": "4fc753c6-927a-41e4-df9e-5575dc47e510",
                "azdata_cell_guid": "713f5d60-b323-4888-8516-2f977eaf9a22",
                "language": "python"
            },
            "outputs": [
                {
                    "output_type": "execute_result",
                    "execution_count": 14,
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead tr th {\n        text-align: left;\n    }\n\n    .dataframe thead tr:last-of-type th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr>\n      <th></th>\n      <th></th>\n      <th colspan=\"3\" halign=\"left\">likes_per_view</th>\n      <th colspan=\"3\" halign=\"left\">comments_per_view</th>\n      <th colspan=\"3\" halign=\"left\">shares_per_view</th>\n    </tr>\n    <tr>\n      <th></th>\n      <th></th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n      <th>count</th>\n      <th>mean</th>\n      <th>median</th>\n    </tr>\n    <tr>\n      <th>claim_status</th>\n      <th>author_ban_status</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th rowspan=\"3\" valign=\"top\">claim</th>\n      <th>active</th>\n      <td>6566</td>\n      <td>0.329542</td>\n      <td>0.326538</td>\n      <td>6566</td>\n      <td>0.001393</td>\n      <td>0.000776</td>\n      <td>6566</td>\n      <td>0.065456</td>\n      <td>0.049279</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>1439</td>\n      <td>0.345071</td>\n      <td>0.358909</td>\n      <td>1439</td>\n      <td>0.001377</td>\n      <td>0.000746</td>\n      <td>1439</td>\n      <td>0.067893</td>\n      <td>0.051606</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>1603</td>\n      <td>0.327997</td>\n      <td>0.320867</td>\n      <td>1603</td>\n      <td>0.001367</td>\n      <td>0.000789</td>\n      <td>1603</td>\n      <td>0.065733</td>\n      <td>0.049967</td>\n    </tr>\n    <tr>\n      <th rowspan=\"3\" valign=\"top\">opinion</th>\n      <th>active</th>\n      <td>8817</td>\n      <td>0.219744</td>\n      <td>0.218330</td>\n      <td>8817</td>\n      <td>0.000517</td>\n      <td>0.000252</td>\n      <td>8817</td>\n      <td>0.043729</td>\n      <td>0.032405</td>\n    </tr>\n    <tr>\n      <th>banned</th>\n      <td>196</td>\n      <td>0.206868</td>\n      <td>0.198483</td>\n      <td>196</td>\n      <td>0.000434</td>\n      <td>0.000193</td>\n      <td>196</td>\n      <td>0.040531</td>\n      <td>0.030728</td>\n    </tr>\n    <tr>\n      <th>under review</th>\n      <td>463</td>\n      <td>0.226394</td>\n      <td>0.228051</td>\n      <td>463</td>\n      <td>0.000536</td>\n      <td>0.000293</td>\n      <td>463</td>\n      <td>0.044472</td>\n      <td>0.035027</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "                               likes_per_view                      \\\n                                        count      mean    median   \nclaim_status author_ban_status                                      \nclaim        active                      6566  0.329542  0.326538   \n             banned                      1439  0.345071  0.358909   \n             under review                1603  0.327997  0.320867   \nopinion      active                      8817  0.219744  0.218330   \n             banned                       196  0.206868  0.198483   \n             under review                 463  0.226394  0.228051   \n\n                               comments_per_view                      \\\n                                           count      mean    median   \nclaim_status author_ban_status                                         \nclaim        active                         6566  0.001393  0.000776   \n             banned                         1439  0.001377  0.000746   \n             under review                   1603  0.001367  0.000789   \nopinion      active                         8817  0.000517  0.000252   \n             banned                          196  0.000434  0.000193   \n             under review                    463  0.000536  0.000293   \n\n                               shares_per_view                      \n                                         count      mean    median  \nclaim_status author_ban_status                                      \nclaim        active                       6566  0.065456  0.049279  \n             banned                       1439  0.067893  0.051606  \n             under review                 1603  0.065733  0.049967  \nopinion      active                       8817  0.043729  0.032405  \n             banned                        196  0.040531  0.030728  \n             under review                  463  0.044472  0.035027  "
                    },
                    "metadata": {}
                }
            ],
            "execution_count": 14
        },
        {
            "cell_type": "markdown",
            "source": [
                "**Observations:** \n",
                "\n",
                "1. We see that videos by banned authors and those under review tend to get **far** more views, likes, and shares than videos by non-banned authors. \n",
                "    \n",
                "    - However, _when a video does get viewed_, its engagement rate is _less related_ to author **ban status** and _more related_ to its **claim status**.\n",
                "2. Also, we see that claim videos have a higher view rate than opinion videos, but this tells us that claim videos also have a higher rate of likes on average, so they are more favorably received as well.\n",
                "    \n",
                "\n",
                "    - Furthermore, they receive **more** engagement via comments and shares than opinion videos.\n",
                "\n",
                "3. Note that for claim videos, banned authors have slightly higher likes/view and shares/view rates than active authors or those under review. \n",
                "\n",
                "    - However, for opinion videos, active authors and those under review _both_ get **higher engagement rates** than banned authors in all categories."
            ],
            "metadata": {
                "id": "6brwGCMcFoNY",
                "azdata_cell_guid": "04de0694-7196-4fb5-8352-83448b7312e5"
            },
            "attachments": {}
        },
        {
            "cell_type": "markdown",
            "source": [
                "# **Preliminary Conclusions**\n",
                "\n",
                "- Of the 19,382 samples in this dataset, just over 50% (9,608 items) are claims.\n",
                "- Engagement level is strongly correlated with claim status. This should be a focus of further inquiry.\n",
                "- Videos with banned authors have significantly higher engagement than videos with active authors. Videos with authors under review fall between these two categories in terms of engagement levels."
            ],
            "metadata": {
                "id": "WS52UKcGz-Bx",
                "azdata_cell_guid": "dc0cd876-d463-499a-a399-60baa9b40d43"
            },
            "attachments": {}
        }
    ]
}