video_to_anime.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "LRS2_to_Anime.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cranberrymuffin/anime-dub/blob/main/video_to_anime.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "color-singer"
      },
      "source": [
        "# Talking Head Anime from a Single Image (Human Video to Anime)\n",
        "\n",
        "**Instruction**\n",
        "\n",
        "1. From the main menu, click \"Runtime > Change runtime type.\" \n",
        "2. Change \"Hardware accelerator\" to \"GPU,\" and click \"Save.\"\n",
        "3. Place your LRS2 data in the `data/` folder\n",
        "3. Place your anime characters in the `data/illust/` folder\n",
        "3. Run the four cells below, one by one, in order by clicking the \"Play\" button to the left of it. Wait for each cell to finish before going to the next one.\n",
        "4. Once the last cell has played (to create a .zip with your converted data, download `anime_lrs2.zip` to your local machine.\n",
        "\n",
        "**Constraints on Images**\n",
        "\n",
        "1. Must be an image of a single humanoid anime character.\n",
        "2. Must be of size 256x256.\n",
        "3. The head must be roughly contained in the middle 128x128 middle box.\n",
        "4. Must have PNG format.\n",
        "5. Must have an alpha channel.\n",
        "6. Background pixels must have RGBA=(0,0,0,0). See [this link](https://github.com/pkhungurn/talking-head-anime-demo/issues/1) if you do not get clean results.\n",
        "\n",
        "**Links**\n",
        "\n",
        "* Human Video to Anime Video Fork: https://github.com/cranberrymuffin/talking-head-anime-demo\n",
        "* Origional Github repository: http://github.com/pkhungurn/talking-head-anime-demo\n",
        "* Origional Project writeup: http://pkhungurn.github.io/talking-head-anime"
      ],
      "id": "color-singer"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bottom-tuner",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c85c51c8-ab58-4780-fb22-47ab8bf893de"
      },
      "source": [
        "# Clone the repository\n",
        "%cd /content\n",
        "!git clone https://github.com/cranberrymuffin/talking-head-anime-demo.git"
      ],
      "id": "bottom-tuner",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "Cloning into 'talking-head-anime-demo'...\n",
            "remote: Enumerating objects: 130, done.\u001b[K\n",
            "remote: Counting objects: 100% (130/130), done.\u001b[K\n",
            "remote: Compressing objects: 100% (96/96), done.\u001b[K\n",
            "remote: Total 130 (delta 66), reused 90 (delta 33), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (130/130), 385.09 KiB | 2.64 MiB/s, done.\n",
            "Resolving deltas: 100% (66/66), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "significant-burlington",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0de222a-0a1a-4cc6-c6e7-13ef7e0fbd44"
      },
      "source": [
        "# CD into the repository directory.\n",
        "%cd /content/talking-head-anime-demo"
      ],
      "id": "significant-burlington",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/talking-head-anime-demo\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "federal-youth",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24da9b74-7d83-48b5-fd4d-724e8ed01976"
      },
      "source": [
        "# Download model files\n",
        "!wget -O data/combiner.pt https://www.dropbox.com/s/p220v9rmbjmqien/combiner.pt?dl=0\n",
        "!wget -O data/face_morpher.pt https://www.dropbox.com/s/oukbnofkffc2bis/face_morpher.pt?dl=0\n",
        "!wget -O data/two_algo_face_rotator.pt https://www.dropbox.com/s/o78wzc5cpxnxggr/two_algo_face_rotator.pt?dl=0"
      ],
      "id": "federal-youth",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-01 01:19:38--  https://www.dropbox.com/s/p220v9rmbjmqien/combiner.pt?dl=0\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.18, 2620:100:601a:18::a27d:712\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/p220v9rmbjmqien/combiner.pt [following]\n",
            "--2021-04-01 01:19:38--  https://www.dropbox.com/s/raw/p220v9rmbjmqien/combiner.pt\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com/cd/0/inline/BLz8ZK4pW1VI239yatIChm71LcHoODgMAMfAUtqvyZRjXXIUIQ3daV2wFbGZzdt2rDaYEPG_FqmCHwHS8sJ4IIfyQp4y32gj114U_mXn36uU5m5TOHXAavVvyz1zfCpXPTsoR0LT10whPfdVTV42npYF/file# [following]\n",
            "--2021-04-01 01:19:38--  https://uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com/cd/0/inline/BLz8ZK4pW1VI239yatIChm71LcHoODgMAMfAUtqvyZRjXXIUIQ3daV2wFbGZzdt2rDaYEPG_FqmCHwHS8sJ4IIfyQp4y32gj114U_mXn36uU5m5TOHXAavVvyz1zfCpXPTsoR0LT10whPfdVTV42npYF/file\n",
            "Resolving uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com (uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com)... 162.125.1.15, 2620:100:601a:15::a27d:70f\n",
            "Connecting to uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com (uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com)|162.125.1.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /cd/0/inline2/BLwyvpvEecHIgnGRox6BQsu4qhCpv-aQwyL6y5y4Aa3Om7z3oh5JNDxSoiA-id6EYUrphyEVlENvhCFPMBC843w6Io2-8skumsFJVMG4G8tHRdectHXSiMMvGixL4s-sBkWd2x4r6NKjSdPIyrP2y6PumhwYYmr6FDcZrP9j809zBLTyFrMb6jirn5q-Q8aTcU6gFfnbZ0jrKKkPAQOlw3ilIfRlfWrY4D6_0zISCI1mKLYQsQHpj4cCMFJzjUvrBZ5IVuzAmKReCNA15F86j2nz-XIWB0IbwtTS-wYkhu6lCtga6hSfTaYCbRK6Zfps23-ZG0T2UbUkhptOIY7VxCDoM8MjwGOjJzDOj2wpLp6PZfHZ4JKie7MRa-B-TTTZR9c/file [following]\n",
            "--2021-04-01 01:19:39--  https://uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com/cd/0/inline2/BLwyvpvEecHIgnGRox6BQsu4qhCpv-aQwyL6y5y4Aa3Om7z3oh5JNDxSoiA-id6EYUrphyEVlENvhCFPMBC843w6Io2-8skumsFJVMG4G8tHRdectHXSiMMvGixL4s-sBkWd2x4r6NKjSdPIyrP2y6PumhwYYmr6FDcZrP9j809zBLTyFrMb6jirn5q-Q8aTcU6gFfnbZ0jrKKkPAQOlw3ilIfRlfWrY4D6_0zISCI1mKLYQsQHpj4cCMFJzjUvrBZ5IVuzAmKReCNA15F86j2nz-XIWB0IbwtTS-wYkhu6lCtga6hSfTaYCbRK6Zfps23-ZG0T2UbUkhptOIY7VxCDoM8MjwGOjJzDOj2wpLp6PZfHZ4JKie7MRa-B-TTTZR9c/file\n",
            "Reusing existing connection to uc74a601c100340eb4b2c0451a5a.dl.dropboxusercontent.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 139853811 (133M) [application/octet-stream]\n",
            "Saving to: ‘data/combiner.pt’\n",
            "\n",
            "data/combiner.pt    100%[===================>] 133.37M  70.4MB/s    in 1.9s    \n",
            "\n",
            "2021-04-01 01:19:41 (70.4 MB/s) - ‘data/combiner.pt’ saved [139853811/139853811]\n",
            "\n",
            "--2021-04-01 01:19:41--  https://www.dropbox.com/s/oukbnofkffc2bis/face_morpher.pt?dl=0\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.18, 2620:100:601a:18::a27d:712\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/oukbnofkffc2bis/face_morpher.pt [following]\n",
            "--2021-04-01 01:19:41--  https://www.dropbox.com/s/raw/oukbnofkffc2bis/face_morpher.pt\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com/cd/0/inline/BLyWD31iVJKcTUCHNZq-GNWl-iBp5JLQkbSo60CiuF3s9FcggCDu1eWdCoCPFvUfocXDHSb4Fcp8LtCsZ4ldh6LUFLkBFGat01ZjFNNbEOtr7QzB3wzhBXQkraEL7_ZT8WUeLWqQ4dycmNljeAyEwoTi/file# [following]\n",
            "--2021-04-01 01:19:42--  https://uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com/cd/0/inline/BLyWD31iVJKcTUCHNZq-GNWl-iBp5JLQkbSo60CiuF3s9FcggCDu1eWdCoCPFvUfocXDHSb4Fcp8LtCsZ4ldh6LUFLkBFGat01ZjFNNbEOtr7QzB3wzhBXQkraEL7_ZT8WUeLWqQ4dycmNljeAyEwoTi/file\n",
            "Resolving uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com (uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com)... 162.125.1.15, 2620:100:6016:15::a27d:10f\n",
            "Connecting to uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com (uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com)|162.125.1.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /cd/0/inline2/BLyzfBxWxMvtQjCJar7hkSZb8ntb49A8wK4BwiqYBC5j29PTipX5sDoSkZKN6Dcs_aHaK0Z1GYV78RsR7rXXKyX3hU_zG96HQ2VR297A3XlPdX3FsSmAoixL7GiiUnHQlXzUCQSNu4W8-a2Ujp_58WRYsxYvlxYY4TkcSSp2MEqv8I64FM1Ye_7oIbuBz1mQ1RvDyZg6vcwNNBuZEIKhBqpnjJ77zZkrBKY97FACISd5e6CCKQ3JUgLyOmiTzL7E3h1-sjM45qeq8r2NBgKBCCWMdzMMhHEvAIEJ9HK428cItqXVdSzvzpyzIBXHEkKGaP9cyzri2ucWV4miNoKVWpqZjWvjZKstQ2_wDk5nQTscQWphJ1-0g3J8Qag0GvoAQ2s/file [following]\n",
            "--2021-04-01 01:19:42--  https://uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com/cd/0/inline2/BLyzfBxWxMvtQjCJar7hkSZb8ntb49A8wK4BwiqYBC5j29PTipX5sDoSkZKN6Dcs_aHaK0Z1GYV78RsR7rXXKyX3hU_zG96HQ2VR297A3XlPdX3FsSmAoixL7GiiUnHQlXzUCQSNu4W8-a2Ujp_58WRYsxYvlxYY4TkcSSp2MEqv8I64FM1Ye_7oIbuBz1mQ1RvDyZg6vcwNNBuZEIKhBqpnjJ77zZkrBKY97FACISd5e6CCKQ3JUgLyOmiTzL7E3h1-sjM45qeq8r2NBgKBCCWMdzMMhHEvAIEJ9HK428cItqXVdSzvzpyzIBXHEkKGaP9cyzri2ucWV4miNoKVWpqZjWvjZKstQ2_wDk5nQTscQWphJ1-0g3J8Qag0GvoAQ2s/file\n",
            "Reusing existing connection to uc4dc0bda23e777ff2c12fccaedb.dl.dropboxusercontent.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 135524776 (129M) [application/octet-stream]\n",
            "Saving to: ‘data/face_morpher.pt’\n",
            "\n",
            "data/face_morpher.p 100%[===================>] 129.25M   151MB/s    in 0.9s    \n",
            "\n",
            "2021-04-01 01:19:44 (151 MB/s) - ‘data/face_morpher.pt’ saved [135524776/135524776]\n",
            "\n",
            "--2021-04-01 01:19:44--  https://www.dropbox.com/s/o78wzc5cpxnxggr/two_algo_face_rotator.pt?dl=0\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/o78wzc5cpxnxggr/two_algo_face_rotator.pt [following]\n",
            "--2021-04-01 01:19:44--  https://www.dropbox.com/s/raw/o78wzc5cpxnxggr/two_algo_face_rotator.pt\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com/cd/0/inline/BLywLmbidJaFwNnXsTzJq57jKlm80IZnEF1MQfKyRdf0-yfHUNc6sgKmyBqoOmUXsk32FlXxvc6vHNXVsqO-uMjmqngIALWZxv-AgjASX66UzrF1hXPPYGs3QxsNA5Vnrdy6kxetPhojR7hIxiZyRNBK/file# [following]\n",
            "--2021-04-01 01:19:44--  https://ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com/cd/0/inline/BLywLmbidJaFwNnXsTzJq57jKlm80IZnEF1MQfKyRdf0-yfHUNc6sgKmyBqoOmUXsk32FlXxvc6vHNXVsqO-uMjmqngIALWZxv-AgjASX66UzrF1hXPPYGs3QxsNA5Vnrdy6kxetPhojR7hIxiZyRNBK/file\n",
            "Resolving ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com (ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com)... 162.125.1.15, 2620:100:601a:15::a27d:70f\n",
            "Connecting to ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com (ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com)|162.125.1.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /cd/0/inline2/BLxsh00RsPXdKs74JR9oHsEQ58bWneTL1ADFIwD-IrwUfSRoyFphDakmkS-yFFG9optLRQAF2kkGqxh5Poovnd-bkg_blgkhGe3WbEe6_d-a3glsPzxluiG-4sbd-3DOd0Dd5ttD35AQn7DHoVEoYf3OvvDahVnB8-B8hEBrXguLvOScZQEs4NzcV0tyLRBv1ppg1pRkXgwgCSQrgJixbPmTBbMUvFAul-j7JHKry1ZgViRRU9r4a4LgndrGAxZD7ercU6mzN3BJCbTMy_pekvuTXw9ufRcQKX4N-ttwaPT3FpPNXB9nTymWJT2-qftzRNNJNDQGvm_RUesUyMVBKDyOOXVS9ysYUr0P6Mpomny-xmyYnZwyRSBhZ_p3nR2Hos0/file [following]\n",
            "--2021-04-01 01:19:45--  https://ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com/cd/0/inline2/BLxsh00RsPXdKs74JR9oHsEQ58bWneTL1ADFIwD-IrwUfSRoyFphDakmkS-yFFG9optLRQAF2kkGqxh5Poovnd-bkg_blgkhGe3WbEe6_d-a3glsPzxluiG-4sbd-3DOd0Dd5ttD35AQn7DHoVEoYf3OvvDahVnB8-B8hEBrXguLvOScZQEs4NzcV0tyLRBv1ppg1pRkXgwgCSQrgJixbPmTBbMUvFAul-j7JHKry1ZgViRRU9r4a4LgndrGAxZD7ercU6mzN3BJCbTMy_pekvuTXw9ufRcQKX4N-ttwaPT3FpPNXB9nTymWJT2-qftzRNNJNDQGvm_RUesUyMVBKDyOOXVS9ysYUr0P6Mpomny-xmyYnZwyRSBhZ_p3nR2Hos0/file\n",
            "Reusing existing connection to ucd1170b25823950d59d3bb78956.dl.dropboxusercontent.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 135550056 (129M) [application/octet-stream]\n",
            "Saving to: ‘data/two_algo_face_rotator.pt’\n",
            "\n",
            "data/two_algo_face_ 100%[===================>] 129.27M   162MB/s    in 0.8s    \n",
            "\n",
            "2021-04-01 01:19:46 (162 MB/s) - ‘data/two_algo_face_rotator.pt’ saved [135550056/135550056]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zeps4S3-_JTw"
      },
      "source": [
        "# **Before proceeding, make sure you download shape_predictor_68_face_landmarks.dat and save it to the data directory. You can download the bzip archive from [here](https://github.com/davisking/dlib-models/blob/master/shape_predictor_68_face_landmarks.dat.bz2). Do not forget to uncompress.**"
      ],
      "id": "Zeps4S3-_JTw"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ic0PxZClFFQA"
      },
      "source": [
        "import glob\n",
        "import os\n",
        "for anime_character_file in glob.iglob('data/illust/*.png'):\n",
        "  for video_file in glob.iglob('data/000*.mp4'):\n",
        "    os.system(\"python app/video_puppeteer.py --character \" + str(anime_character_file) + \" --video  \" + str(video_file))"
      ],
      "id": "ic0PxZClFFQA",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GaffENqvK1tR",
        "outputId": "d9812e40-aef7-4f58-f556-bda6aba24ab1"
      },
      "source": [
        "!zip -r anime_lrs2.zip results/converted_dataset"
      ],
      "id": "GaffENqvK1tR",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  adding: results/converted_dataset/ (stored 0%)\n",
            "  adding: results/converted_dataset/00019.txt (deflated 41%)\n",
            "  adding: results/converted_dataset/00011.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00011.txt (deflated 39%)\n",
            "  adding: results/converted_dataset/00020.txt (deflated 49%)\n",
            "  adding: results/converted_dataset/00012.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00016.mp4 (deflated 1%)\n",
            "  adding: results/converted_dataset/00028.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00028.txt (deflated 45%)\n",
            "  adding: results/converted_dataset/00014.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00020_anime_waifu_04_256.mp4 (deflated 3%)\n",
            "  adding: results/converted_dataset/00012_anime_waifu_04_256.mp4 (deflated 3%)\n",
            "  adding: results/converted_dataset/00019_anime_waifu_04_256.mp4 (deflated 4%)\n",
            "  adding: results/converted_dataset/00013.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00027.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00027.txt (deflated 35%)\n",
            "  adding: results/converted_dataset/00013.txt (deflated 34%)\n",
            "  adding: results/converted_dataset/00020.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00016.txt (deflated 39%)\n",
            "  adding: results/converted_dataset/00014_anime_waifu_04_256.mp4 (deflated 3%)\n",
            "  adding: results/converted_dataset/00028_anime_waifu_04_256.mp4 (deflated 4%)\n",
            "  adding: results/converted_dataset/00011_anime_waifu_04_256.mp4 (deflated 4%)\n",
            "  adding: results/converted_dataset/00016_anime_waifu_04_256.mp4 (deflated 4%)\n",
            "  adding: results/converted_dataset/00019.mp4 (deflated 0%)\n",
            "  adding: results/converted_dataset/00012.txt (deflated 41%)\n",
            "  adding: results/converted_dataset/00027_anime_waifu_04_256.mp4 (deflated 4%)\n",
            "  adding: results/converted_dataset/00014.txt (deflated 42%)\n",
            "  adding: results/converted_dataset/00013_anime_waifu_04_256.mp4 (deflated 3%)\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}