chapter12_part03_neural-style-transfer.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P83bO4GUagpq"
      },
      "source": [
        "이 노트북은 [케라스 창시자에게 배우는 딥러닝 2판](https://tensorflow.blog/kerasdl2/)의 예제 코드를 담고 있습니다.\n",
        "\n",
        "<table align=\"left\">\n",
        "    <tr>\n",
        "        <td>\n",
        "            <a href=\"https://colab.research.google.com/github/rickiepark/deep-learning-with-python-2nd/blob/main/chapter12_part03_neural-style-transfer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "        </td>\n",
        "    </tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aB73qx2Lagpt"
      },
      "source": [
        "## 뉴럴 스타일 트랜스퍼"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzKwawEtagpu"
      },
      "source": [
        "### 콘텐츠 손실"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7EAmjBVmagpu"
      },
      "source": [
        "### 스타일 손실"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXuvL7ipagpu"
      },
      "source": [
        "### 케라스로 뉴럴 스타일 트랜스퍼 구현하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CE3vZQbPagpu"
      },
      "source": [
        "**스타일 이미지와 콘텐츠 이미지 준비하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "e8TXkoZiagpv",
        "outputId": "8ce80e1d-49e1-4899-e773-908494c6d4d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://img-datasets.s3.amazonaws.com/sf.jpg\n",
            "575046/575046 [==============================] - 1s 2us/step\n",
            "Downloading data from https://img-datasets.s3.amazonaws.com/starry_night.jpg\n",
            "943128/943128 [==============================] - 1s 1us/step\n"
          ]
        }
      ],
      "source": [
        "from tensorflow import keras\n",
        "\n",
        "base_image_path = keras.utils.get_file(\n",
        "    \"sf.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/sf.jpg\")\n",
        "style_reference_image_path = keras.utils.get_file(\n",
        "    \"starry_night.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/starry_night.jpg\")\n",
        "\n",
        "original_width, original_height = keras.utils.load_img(base_image_path).size\n",
        "img_height = 400\n",
        "img_width = round(original_width * img_height / original_height)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0P3cTrgCagpw"
      },
      "source": [
        "**유틸리티 함수**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "kuJBC13Oagpw"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def preprocess_image(image_path):\n",
        "    img = keras.utils.load_img(\n",
        "        image_path, target_size=(img_height, img_width))\n",
        "    img = keras.utils.img_to_array(img)\n",
        "    img = np.expand_dims(img, axis=0)\n",
        "    img = keras.applications.vgg19.preprocess_input(img)\n",
        "    return img\n",
        "\n",
        "def deprocess_image(img):\n",
        "    img = img.reshape((img_height, img_width, 3))\n",
        "    img[:, :, 0] += 103.939\n",
        "    img[:, :, 1] += 116.779\n",
        "    img[:, :, 2] += 123.68\n",
        "    img = img[:, :, ::-1]\n",
        "    img = np.clip(img, 0, 255).astype(\"uint8\")\n",
        "    return img"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UHOtTa2zagpx"
      },
      "source": [
        "**사전 훈련된 VGG19 모델을 사용해 특성 추출기 만들기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "ALBqRpdfagpx",
        "outputId": "a1fb21c6-0ef1-4bd5-81e6-dd41c5532967",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg19/vgg19_weights_tf_dim_ordering_tf_kernels_notop.h5\n",
            "80134624/80134624 [==============================] - 5s 0us/step\n"
          ]
        }
      ],
      "source": [
        "model = keras.applications.vgg19.VGG19(weights=\"imagenet\", include_top=False)\n",
        "\n",
        "outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])\n",
        "feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6pPDIK5agpy"
      },
      "source": [
        "**콘텐츠 손실**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "U5IKJ7lragpy"
      },
      "outputs": [],
      "source": [
        "def content_loss(base_img, combination_img):\n",
        "    return tf.reduce_sum(tf.square(combination_img - base_img))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ErdX5lJZagpy"
      },
      "source": [
        "**스타일 손실**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "-3hG8Zshagpy"
      },
      "outputs": [],
      "source": [
        "def gram_matrix(x):\n",
        "    x = tf.transpose(x, (2, 0, 1))\n",
        "    features = tf.reshape(x, (tf.shape(x)[0], -1))\n",
        "    gram = tf.matmul(features, tf.transpose(features))\n",
        "    return gram\n",
        "\n",
        "def style_loss(style_img, combination_img):\n",
        "    S = gram_matrix(style_img)\n",
        "    C = gram_matrix(combination_img)\n",
        "    channels = 3\n",
        "    size = img_height * img_width\n",
        "    return tf.reduce_sum(tf.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xskhdIfOagpz"
      },
      "source": [
        "**총 변위 손실**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "jhiOGPOCagpz"
      },
      "outputs": [],
      "source": [
        "def total_variation_loss(x):\n",
        "    a = tf.square(\n",
        "        x[:, : img_height - 1, : img_width - 1, :] - x[:, 1:, : img_width - 1, :]\n",
        "    )\n",
        "    b = tf.square(\n",
        "        x[:, : img_height - 1, : img_width - 1, :] - x[:, : img_height - 1, 1:, :]\n",
        "    )\n",
        "    return tf.reduce_sum(tf.pow(a + b, 1.25))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t48S0czKagpz"
      },
      "source": [
        "**최소화할 최종 손실 정의하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "hoGzAe9_agpz"
      },
      "outputs": [],
      "source": [
        "style_layer_names = [\n",
        "    \"block1_conv1\",\n",
        "    \"block2_conv1\",\n",
        "    \"block3_conv1\",\n",
        "    \"block4_conv1\",\n",
        "    \"block5_conv1\",\n",
        "]\n",
        "content_layer_name = \"block5_conv2\"\n",
        "total_variation_weight = 1e-6\n",
        "style_weight = 1e-6\n",
        "content_weight = 2.5e-8\n",
        "\n",
        "def compute_loss(combination_image, base_image, style_reference_image):\n",
        "    input_tensor = tf.concat(\n",
        "        [base_image, style_reference_image, combination_image], axis=0\n",
        "    )\n",
        "    features = feature_extractor(input_tensor)\n",
        "    loss = tf.zeros(shape=())\n",
        "    layer_features = features[content_layer_name]\n",
        "    base_image_features = layer_features[0, :, :, :]\n",
        "    combination_features = layer_features[2, :, :, :]\n",
        "    loss = loss + content_weight * content_loss(\n",
        "        base_image_features, combination_features\n",
        "    )\n",
        "    for layer_name in style_layer_names:\n",
        "        layer_features = features[layer_name]\n",
        "        style_reference_features = layer_features[1, :, :, :]\n",
        "        combination_features = layer_features[2, :, :, :]\n",
        "        style_loss_value = style_loss(\n",
        "          style_reference_features, combination_features)\n",
        "        loss += (style_weight / len(style_layer_names)) * style_loss_value\n",
        "\n",
        "    loss += total_variation_weight * total_variation_loss(combination_image)\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R68qRidVagp0"
      },
      "source": [
        "**경사 하강법 단계 설정하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "MRB3CAMUagp0",
        "outputId": "b3e79e80-346e-47b6-aaf2-24436de1ea51",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100번째 반복: loss=8136.60\n",
            "200번째 반복: loss=6652.54\n",
            "300번째 반복: loss=6062.91\n",
            "400번째 반복: loss=5741.78\n",
            "500번째 반복: loss=5535.58\n",
            "600번째 반복: loss=5389.96\n",
            "700번째 반복: loss=5280.65\n",
            "800번째 반복: loss=5195.29\n",
            "900번째 반복: loss=5126.43\n",
            "1000번째 반복: loss=5069.65\n",
            "1100번째 반복: loss=5021.93\n",
            "1200번째 반복: loss=4981.17\n",
            "1300번째 반복: loss=4945.86\n",
            "1400번째 반복: loss=4914.91\n",
            "1500번째 반복: loss=4887.63\n",
            "1600번째 반복: loss=4863.49\n",
            "1700번째 반복: loss=4841.96\n",
            "1800번째 반복: loss=4822.71\n",
            "1900번째 반복: loss=4805.31\n",
            "2000번째 반복: loss=4789.50\n",
            "2100번째 반복: loss=4775.13\n",
            "2200번째 반복: loss=4762.00\n",
            "2300번째 반복: loss=4749.93\n",
            "2400번째 반복: loss=4738.82\n",
            "2500번째 반복: loss=4728.57\n",
            "2600번째 반복: loss=4719.05\n",
            "2700번째 반복: loss=4710.29\n",
            "2800번째 반복: loss=4702.19\n",
            "2900번째 반복: loss=4694.63\n",
            "3000번째 반복: loss=4687.60\n",
            "3100번째 반복: loss=4681.03\n",
            "3200번째 반복: loss=4674.88\n",
            "3300번째 반복: loss=4669.12\n",
            "3400번째 반복: loss=4663.73\n",
            "3500번째 반복: loss=4658.68\n",
            "3600번째 반복: loss=4653.93\n",
            "3700번째 반복: loss=4649.46\n",
            "3800번째 반복: loss=4645.25\n",
            "3900번째 반복: loss=4641.29\n",
            "4000번째 반복: loss=4637.54\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "@tf.function\n",
        "def compute_loss_and_grads(combination_image, base_image, style_reference_image):\n",
        "    with tf.GradientTape() as tape:\n",
        "        loss = compute_loss(combination_image, base_image, style_reference_image)\n",
        "    grads = tape.gradient(loss, combination_image)\n",
        "    return loss, grads\n",
        "\n",
        "optimizer = keras.optimizers.SGD(\n",
        "    keras.optimizers.schedules.ExponentialDecay(\n",
        "        initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96\n",
        "    )\n",
        ")\n",
        "\n",
        "base_image = preprocess_image(base_image_path)\n",
        "style_reference_image = preprocess_image(style_reference_image_path)\n",
        "combination_image = tf.Variable(preprocess_image(base_image_path))\n",
        "\n",
        "iterations = 4000\n",
        "for i in range(1, iterations + 1):\n",
        "    loss, grads = compute_loss_and_grads(\n",
        "        combination_image, base_image, style_reference_image\n",
        "    )\n",
        "    optimizer.apply_gradients([(grads, combination_image)])\n",
        "    if i % 100 == 0:\n",
        "        print(f\"{i}번째 반복: loss={loss:.2f}\")\n",
        "        img = deprocess_image(combination_image.numpy())\n",
        "        fname = f\"combination_image_at_iteration_{i}.png\"\n",
        "        keras.utils.save_img(fname, img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbwjfI_9agp0"
      },
      "source": [
        "![combination_image_at_iteration_4000.png](https://github.com/rickiepark/deep-learning-with-python-2nd/blob/main/combination_image_at_iteration_4000.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oU71z4i-agp0"
      },
      "source": [
        "### 정리"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "chapter12_part03_neural-style-transfer.i",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "default:Python",
      "language": "python",
      "name": "conda-env-default-py"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.10"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}