notebooks/dynamicrafter-animating-images/dynamicrafter-animating-images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a30812de-c46e-44a3-8194-b7f6f0fd4707",
   "metadata": {},
   "source": [
    "# Animating Open-domain Images with DynamiCrafter and OpenVINO\n",
    "\n",
    "Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, [DynamiCrafter team](https://doubiiu.github.io/projects/DynamiCrafter/) explores the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, DynamiCrafter team first projects it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, DynamiCrafter team further feeds the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that the proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image.\n",
    "\n",
    "In this tutorial, we consider how to use DynamiCrafter with OpenVINO. An additional part demonstrates how to run optimization with [NNCF](https://github.com/openvinotoolkit/nncf/) to speed up model.\n",
    "\n",
    "<table class=\"center\">\n",
    "  <tr>\n",
    "    <td colspan=\"2\">\"bear playing guitar happily, snowing\"</td>\n",
    "    <td colspan=\"2\">\"boy walking on the street\"</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.jpeg_00.png?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.gif?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/walk0.png_00.png?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/walk0.gif?raw=True width=\"170\">\n",
    "  </td>\n",
    "  </tr>\n",
    "</table >\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Load the original model](#Load-the-original-model)\n",
    "- [Convert the model to OpenVINO IR](#Convert-the-model-to-OpenVINO-IR)\n",
    "  - [Convert CLIP text encoder](#Convert-CLIP-text-encoder)\n",
    "  - [Convert CLIP image encoder](#Convert-CLIP-image-encoder)\n",
    "  - [Convert AE encoder](#Convert-AE-encoder)\n",
    "  - [Convert Diffusion U-Net model](#Convert-Diffusion-U-Net-model)\n",
    "  - [Convert AE decoder](#Convert-AE-decoder)\n",
    "- [Compiling models](#Compiling-models)\n",
    "- [Building the pipeline](#Building-the-pipeline)\n",
    "- [Run OpenVINO pipeline inference](#Run-OpenVINO-pipeline-inference)\n",
    "- [Quantization](#Quantization)\n",
    "    - [Prepare calibration dataset](#Prepare-calibration-dataset)\n",
    "    - [Run Quantization](#Run-Quantization)\n",
    "    - [Run Weights Compression](#Run-Weights-Compression)\n",
    "    - [Compare model file sizes](#Compare-model-file-sizes)\n",
    "    - [Compare inference time of the FP32 and INT8 pipelines](#Compare-inference-time-of-the-FP32-and-INT8-pipelines)\n",
    "- [Interactive inference](#Interactive-inference)\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/dynamicrafter-animating-images/dynamicrafter-animating-images.ipynb\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9dc9580-da81-47dd-b5d3-3cafa8f5a4b5",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "eac97b7e-2db7-41b3-8dc4-488c5b5cd275",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Error parsing dependencies of torchsde: .* suffix can only be used with `==` or `!=` operators\n",
      "    numpy (>=1.19.*) ; python_version >= \"3.7\"\n",
      "           ~~~~~~~^\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "\u001b[33mWARNING: Error parsing dependencies of torchsde: .* suffix can only be used with `==` or `!=` operators\n",
      "    numpy (>=1.19.*) ; python_version >= \"3.7\"\n",
      "           ~~~~~~~^\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -q \"openvino>=2024.2.0\" \"nncf>=2.11.0\" \"datasets>=2.20.0\"\n",
    "%pip install -q \"gradio>=4.19\" omegaconf einops pytorch_lightning kornia \"open_clip_torch==2.22.0\" transformers av opencv-python \"torch==2.2.2\" --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4c8de050-19e7-42a2-bf5a-98ca5eef050e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "dynamicrafter_path = Path(\"dynamicrafter\")\n",
    "\n",
    "if not dynamicrafter_path.exists():\n",
    "    dynamicrafter_path.mkdir(parents=True, exist_ok=True)\n",
    "    !git clone https://github.com/Doubiiu/DynamiCrafter.git dynamicrafter\n",
    "    %cd dynamicrafter\n",
    "    !git checkout 26e665cd6c174234238d2ded661e2e56f875d360 -q  # to avoid breaking changes\n",
    "    %cd ..\n",
    "\n",
    "sys.path.append(str(dynamicrafter_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c0c659-aad3-4962-8db7-7b123379f01a",
   "metadata": {},
   "source": [
    "## Load and run the original pipeline\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We will use model for 256x256 resolution as example. Also, models for 320x512 and 576x1024 are [available](https://github.com/Doubiiu/DynamiCrafter?tab=readme-ov-file#-models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3ce0481-d7de-4d37-9414-c72dc6488f8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.\n",
      ">>> model checkpoint loaded.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from collections import OrderedDict\n",
    "\n",
    "import torch\n",
    "from huggingface_hub import hf_hub_download\n",
    "from omegaconf import OmegaConf\n",
    "\n",
    "from dynamicrafter.utils.utils import instantiate_from_config\n",
    "\n",
    "\n",
    "def load_model_checkpoint(model, ckpt):\n",
    "    def load_checkpoint(model, ckpt, full_strict):\n",
    "        state_dict = torch.load(ckpt, map_location=\"cpu\")\n",
    "        if \"state_dict\" in list(state_dict.keys()):\n",
    "            state_dict = state_dict[\"state_dict\"]\n",
    "            try:\n",
    "                model.load_state_dict(state_dict, strict=full_strict)\n",
    "            except Exception:\n",
    "                ## rename the keys for 256x256 model\n",
    "                new_pl_sd = OrderedDict()\n",
    "                for k, v in state_dict.items():\n",
    "                    new_pl_sd[k] = v\n",
    "\n",
    "                for k in list(new_pl_sd.keys()):\n",
    "                    if \"framestride_embed\" in k:\n",
    "                        new_key = k.replace(\"framestride_embed\", \"fps_embedding\")\n",
    "                        new_pl_sd[new_key] = new_pl_sd[k]\n",
    "                        del new_pl_sd[k]\n",
    "                model.load_state_dict(new_pl_sd, strict=full_strict)\n",
    "        else:\n",
    "            ## deepspeed\n",
    "            new_pl_sd = OrderedDict()\n",
    "            for key in state_dict[\"module\"].keys():\n",
    "                new_pl_sd[key[16:]] = state_dict[\"module\"][key]\n",
    "            model.load_state_dict(new_pl_sd, strict=full_strict)\n",
    "\n",
    "        return model\n",
    "\n",
    "    load_checkpoint(model, ckpt, full_strict=True)\n",
    "    print(\">>> model checkpoint loaded.\")\n",
    "    return model\n",
    "\n",
    "\n",
    "def download_model():\n",
    "    REPO_ID = \"Doubiiu/DynamiCrafter\"\n",
    "    if not os.path.exists(\"./checkpoints/dynamicrafter_256_v1/\"):\n",
    "        os.makedirs(\"./checkpoints/dynamicrafter_256_v1/\")\n",
    "    local_file = os.path.join(\"./checkpoints/dynamicrafter_256_v1/model.ckpt\")\n",
    "    if not os.path.exists(local_file):\n",
    "        hf_hub_download(repo_id=REPO_ID, filename=\"model.ckpt\", local_dir=\"./checkpoints/dynamicrafter_256_v1/\", local_dir_use_symlinks=False)\n",
    "\n",
    "    ckpt_path = \"checkpoints/dynamicrafter_256_v1/model.ckpt\"\n",
    "    config_file = \"dynamicrafter/configs/inference_256_v1.0.yaml\"\n",
    "    config = OmegaConf.load(config_file)\n",
    "    model_config = config.pop(\"model\", OmegaConf.create())\n",
    "    model_config[\"params\"][\"unet_config\"][\"params\"][\"use_checkpoint\"] = False\n",
    "    model = instantiate_from_config(model_config)\n",
    "    model = load_model_checkpoint(model, ckpt_path)\n",
    "    model.eval()\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "model = download_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be9643c8-a70c-4dba-8259-d4467ae82949",
   "metadata": {},
   "source": [
    "## Convert the model to OpenVINO IR\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's define the conversion function for PyTorch modules. We use `ov.convert_model` function to obtain OpenVINO Intermediate Representation object and `ov.save_model` function to save it as XML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1bf5ffa9-c7d5-4485-915c-48ed18f657dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "\n",
    "def convert(model: torch.nn.Module, xml_path: str, example_input, input_shape=None):\n",
    "    xml_path = Path(xml_path)\n",
    "    if not xml_path.exists():\n",
    "        xml_path.parent.mkdir(parents=True, exist_ok=True)\n",
    "        with torch.no_grad():\n",
    "            if not input_shape:\n",
    "                converted_model = ov.convert_model(model, example_input=example_input)\n",
    "            else:\n",
    "                converted_model = ov.convert_model(model, example_input=example_input, input=input_shape)\n",
    "        ov.save_model(converted_model, xml_path, compress_to_fp16=False)\n",
    "\n",
    "        # cleanup memory\n",
    "        torch._C._jit_clear_class_registry()\n",
    "        torch.jit._recursive.concrete_type_store = torch.jit._recursive.ConcreteTypeStore()\n",
    "        torch.jit._state._clear_class_state()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c63518d-957d-4358-8711-cf6fb935d8be",
   "metadata": {},
   "source": [
    "Flowchart of DynamiCrafter proposed in [the paper](https://arxiv.org/abs/2310.12190):\n",
    "\n",
    "![schema](https://github.com/openvinotoolkit/openvino_notebooks/assets/76171391/d1033876-c664-4345-a254-0649edbf1906)\n",
    "Description:\n",
    "> During inference, our model can generate animation clips from noise conditioned on the input still image.\n",
    "\n",
    "Let's convert models from the pipeline one by one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9824415cd5b0ffd",
   "metadata": {},
   "source": [
    "### Convert CLIP text encoder\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e0b1637-7757-49bb-b60f-175f7d77b470",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dynamicrafter.lvdm.modules.encoders.condition import FrozenOpenCLIPEmbedder\n",
    "\n",
    "\n",
    "COND_STAGE_MODEL_OV_PATH = Path(\"models/cond_stage_model.xml\")\n",
    "\n",
    "\n",
    "class FrozenOpenCLIPEmbedderWrapper(FrozenOpenCLIPEmbedder):\n",
    "    def forward(self, tokens):\n",
    "        z = self.encode_with_transformer(tokens.to(self.device))\n",
    "        return z\n",
    "\n",
    "\n",
    "cond_stage_model = FrozenOpenCLIPEmbedderWrapper(device=\"cpu\")\n",
    "\n",
    "if not COND_STAGE_MODEL_OV_PATH.exists():\n",
    "    convert(\n",
    "        cond_stage_model,\n",
    "        COND_STAGE_MODEL_OV_PATH,\n",
    "        example_input=torch.ones([1, 77], dtype=torch.long),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b361937c948711",
   "metadata": {},
   "source": [
    "### Convert CLIP image encoder\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "`FrozenOpenCLIPImageEmbedderV2` model accepts images of various resolutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c6aa0a3f-4093-44c7-b7b0-ca830abc4826",
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDER_OV_PATH = Path(\"models/embedder_ir.xml\")\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([1, 3, 767, 767], dtype=torch.float32)\n",
    "\n",
    "model.embedder.model.visual.input_patchnorm = None  # fix error: visual model has not  attribute 'input_patchnorm'\n",
    "if not EMBEDDER_OV_PATH.exists():\n",
    "    convert(model.embedder, EMBEDDER_OV_PATH, example_input=dummy_input, input_shape=[1, 3, -1, -1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef65d17fec62fa",
   "metadata": {},
   "source": [
    "### Convert AE encoder\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "41434d51-07da-4688-b0af-be6271fef54f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ENCODER_FIRST_STAGE_OV_PATH = Path(\"models/encoder_first_stage_ir.xml\")\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([1, 3, 256, 256], dtype=torch.float32)\n",
    "\n",
    "if not ENCODER_FIRST_STAGE_OV_PATH.exists():\n",
    "    convert(\n",
    "        model.first_stage_model.encoder,\n",
    "        ENCODER_FIRST_STAGE_OV_PATH,\n",
    "        example_input=dummy_input,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec5ee02317d8e77",
   "metadata": {},
   "source": [
    "### Convert Diffusion U-Net model\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ca501d47-3f65-4d58-804d-d8ed20a761af",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_OV_PATH = Path(\"models/model_ir.xml\")\n",
    "\n",
    "\n",
    "class ModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, diffusion_model):\n",
    "        super().__init__()\n",
    "        self.diffusion_model = diffusion_model\n",
    "\n",
    "    def forward(self, xc, t, context=None, fs=None, temporal_length=None):\n",
    "        outputs = self.diffusion_model(xc, t, context=context, fs=fs, temporal_length=temporal_length)\n",
    "        return outputs\n",
    "\n",
    "\n",
    "if not MODEL_OV_PATH.exists():\n",
    "    convert(\n",
    "        ModelWrapper(model.model.diffusion_model),\n",
    "        MODEL_OV_PATH,\n",
    "        example_input={\n",
    "            \"xc\": torch.rand([1, 8, 16, 32, 32], dtype=torch.float32),\n",
    "            \"t\": torch.tensor([1]),\n",
    "            \"context\": torch.rand([1, 333, 1024], dtype=torch.float32),\n",
    "            \"fs\": torch.tensor([3]),\n",
    "            \"temporal_length\": torch.tensor([16]),\n",
    "        },\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d5430af-12b4-4a15-bb7c-c9300f824431",
   "metadata": {},
   "source": [
    "### Convert AE decoder\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "`Decoder` receives a `bfloat16` tensor. numpy doesn't support this type. To avoid problems with the conversion lets replace `decode` method to convert bfloat16 to float32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "db8394e7-d60b-4dbd-a94a-65e4f21a959b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import types\n",
    "\n",
    "\n",
    "def decode(self, z, **kwargs):\n",
    "    z = self.post_quant_conv(z)\n",
    "    z = z.float()\n",
    "    dec = self.decoder(z)\n",
    "    return dec\n",
    "\n",
    "\n",
    "model.first_stage_model.decode = types.MethodType(decode, model.first_stage_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d7c6879b-8b51-4d76-81e1-669378c7c4e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ea/work/openvino_notebooks_new_clone/openvino_notebooks/notebooks/dynamicrafter-animating-images/dynamicrafter/lvdm/modules/networks/ae_modules.py:67: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  w_ = w_ * (int(c)**(-0.5))\n"
     ]
    }
   ],
   "source": [
    "DECODER_FIRST_STAGE_OV_PATH = Path(\"models/decoder_first_stage_ir.xml\")\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([16, 4, 32, 32], dtype=torch.float32)\n",
    "\n",
    "if not DECODER_FIRST_STAGE_OV_PATH.exists():\n",
    "    convert(\n",
    "        model.first_stage_model.decoder,\n",
    "        DECODER_FIRST_STAGE_OV_PATH,\n",
    "        example_input=dummy_input,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ff6eb8-dd58-4820-aae3-85c0b4e487a8",
   "metadata": {},
   "source": [
    "## Compiling models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Select device from dropdown list for running inference using OpenVINO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f052ebf-dabe-4161-bee7-4a9d55b9b69a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35243ae51b58460e900598c06ae92547",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "core = ov.Core()\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value=\"AUTO\",\n",
    "    description=\"Device:\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "410e05b4-11ed-4d85-ab8d-5d1557c4c36e",
   "metadata": {},
   "outputs": [],
   "source": [
    "compiled_cond_stage_model = core.compile_model(COND_STAGE_MODEL_OV_PATH, device.value)\n",
    "compiled_encode_first_stage = core.compile_model(ENCODER_FIRST_STAGE_OV_PATH, device.value)\n",
    "compiled_embedder = core.compile_model(EMBEDDER_OV_PATH, device.value)\n",
    "compiled_model = core.compile_model(MODEL_OV_PATH, device.value)\n",
    "compiled_decoder_first_stage = core.compile_model(DECODER_FIRST_STAGE_OV_PATH, device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f2c95b-e872-458b-a6f8-448f8124ffe6",
   "metadata": {},
   "source": [
    "## Building the pipeline\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's create callable wrapper classes for compiled models to allow interaction with original pipelines. Note that all of wrapper classes return `torch.Tensor`s instead of `np.array`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9d50a153-f71d-4364-9114-14bc25e239d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import open_clip\n",
    "\n",
    "\n",
    "class CondStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, cond_stage_model):\n",
    "        super().__init__()\n",
    "        self.cond_stage_model = cond_stage_model\n",
    "\n",
    "    def encode(self, tokens):\n",
    "        if isinstance(tokens, list):\n",
    "            tokens = open_clip.tokenize(tokens[0])\n",
    "        outs = self.cond_stage_model(tokens)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "\n",
    "class EncoderFirstStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, encode_first_stage):\n",
    "        super().__init__()\n",
    "        self.encode_first_stage = encode_first_stage\n",
    "\n",
    "    def forward(self, x):\n",
    "        outs = self.encode_first_stage(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "\n",
    "class EmbedderWrapper(torch.nn.Module):\n",
    "    def __init__(self, embedder):\n",
    "        super().__init__()\n",
    "        self.embedder = embedder\n",
    "\n",
    "    def forward(self, x):\n",
    "        outs = self.embedder(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "\n",
    "class CModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, diffusion_model, out_channels):\n",
    "        super().__init__()\n",
    "        self.diffusion_model = diffusion_model\n",
    "        self.out_channels = out_channels\n",
    "\n",
    "    def forward(self, xc, t, context, fs, temporal_length):\n",
    "        inputs = {\n",
    "            \"xc\": xc,\n",
    "            \"t\": t,\n",
    "            \"context\": context,\n",
    "            \"fs\": fs,\n",
    "        }\n",
    "        outs = self.diffusion_model(inputs)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "\n",
    "class DecoderFirstStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, decoder_first_stage):\n",
    "        super().__init__()\n",
    "        self.decoder_first_stage = decoder_first_stage\n",
    "\n",
    "    def forward(self, x):\n",
    "        x.float()\n",
    "        outs = self.decoder_first_stage(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1178a847-eb14-419b-815e-c47628aa6868",
   "metadata": {},
   "source": [
    "And insert wrappers instances in the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f45d5dc3-6d17-408e-826d-b8525f461e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.cond_stage_model = CondStageModelWrapper(compiled_cond_stage_model)\n",
    "model.first_stage_model.encoder = EncoderFirstStageModelWrapper(compiled_encode_first_stage)\n",
    "model.embedder = EmbedderWrapper(compiled_embedder)\n",
    "model.model.diffusion_model = CModelWrapper(compiled_model, model.model.diffusion_model.out_channels)\n",
    "model.first_stage_model.decoder = DecoderFirstStageModelWrapper(compiled_decoder_first_stage)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "953f6d14",
   "metadata": {},
   "source": [
    "## Run OpenVINO pipeline inference\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9e7729a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from einops import repeat, rearrange\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "\n",
    "transform = transforms.Compose(\n",
    "    [\n",
    "        transforms.Resize(min((256, 256))),\n",
    "        transforms.CenterCrop((256, 256)),\n",
    "    ]\n",
    ")\n",
    "\n",
    "\n",
    "def get_latent_z(model, videos):\n",
    "    b, c, t, h, w = videos.shape\n",
    "    x = rearrange(videos, \"b c t h w -> (b t) c h w\")\n",
    "    z = model.encode_first_stage(x)\n",
    "    z = rearrange(z, \"(b t) c h w -> b c t h w\", b=b, t=t)\n",
    "    return z\n",
    "\n",
    "\n",
    "def process_input(model, prompt, image, transform=transform, fs=3):\n",
    "    text_emb = model.get_learned_conditioning([prompt])\n",
    "\n",
    "    # img cond\n",
    "    img_tensor = torch.from_numpy(image).permute(2, 0, 1).float().to(model.device)\n",
    "    img_tensor = (img_tensor / 255.0 - 0.5) * 2\n",
    "\n",
    "    image_tensor_resized = transform(img_tensor)  # 3,h,w\n",
    "    videos = image_tensor_resized.unsqueeze(0)  # bchw\n",
    "\n",
    "    z = get_latent_z(model, videos.unsqueeze(2))  # bc,1,hw\n",
    "    frames = model.temporal_length\n",
    "    img_tensor_repeat = repeat(z, \"b c t h w -> b c (repeat t) h w\", repeat=frames)\n",
    "\n",
    "    cond_images = model.embedder(img_tensor.unsqueeze(0))  # blc\n",
    "    img_emb = model.image_proj_model(cond_images)\n",
    "    imtext_cond = torch.cat([text_emb, img_emb], dim=1)\n",
    "\n",
    "    fs = torch.tensor([fs], dtype=torch.long, device=model.device)\n",
    "    cond = {\"c_crossattn\": [imtext_cond], \"fs\": fs, \"c_concat\": [img_tensor_repeat]}\n",
    "    return cond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "52b6ff5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "from lvdm.models.samplers.ddim import DDIMSampler\n",
    "from pytorch_lightning import seed_everything\n",
    "import torchvision\n",
    "\n",
    "\n",
    "def register_buffer(self, name, attr):\n",
    "    if isinstance(attr, torch.Tensor):\n",
    "        if attr.device != torch.device(\"cpu\"):\n",
    "            attr = attr.to(torch.device(\"cpu\"))\n",
    "    setattr(self, name, attr)\n",
    "\n",
    "\n",
    "def batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=50, ddim_eta=1.0, cfg_scale=1.0, temporal_cfg_scale=None, **kwargs):\n",
    "    ddim_sampler = DDIMSampler(model)\n",
    "    uncond_type = model.uncond_type\n",
    "    batch_size = noise_shape[0]\n",
    "    fs = cond[\"fs\"]\n",
    "    del cond[\"fs\"]\n",
    "    if noise_shape[-1] == 32:\n",
    "        timestep_spacing = \"uniform\"\n",
    "        guidance_rescale = 0.0\n",
    "    else:\n",
    "        timestep_spacing = \"uniform_trailing\"\n",
    "        guidance_rescale = 0.7\n",
    "    # construct unconditional guidance\n",
    "    if cfg_scale != 1.0:\n",
    "        if uncond_type == \"empty_seq\":\n",
    "            prompts = batch_size * [\"\"]\n",
    "            # prompts = N * T * [\"\"]  ## if is_imgbatch=True\n",
    "            uc_emb = model.get_learned_conditioning(prompts)\n",
    "        elif uncond_type == \"zero_embed\":\n",
    "            c_emb = cond[\"c_crossattn\"][0] if isinstance(cond, dict) else cond\n",
    "            uc_emb = torch.zeros_like(c_emb)\n",
    "\n",
    "        # process image embedding token\n",
    "        if hasattr(model, \"embedder\"):\n",
    "            uc_img = torch.zeros(noise_shape[0], 3, 224, 224).to(model.device)\n",
    "            ## img: b c h w >> b l c\n",
    "            uc_img = model.embedder(uc_img)\n",
    "            uc_img = model.image_proj_model(uc_img)\n",
    "            uc_emb = torch.cat([uc_emb, uc_img], dim=1)\n",
    "\n",
    "        if isinstance(cond, dict):\n",
    "            uc = {key: cond[key] for key in cond.keys()}\n",
    "            uc.update({\"c_crossattn\": [uc_emb]})\n",
    "        else:\n",
    "            uc = uc_emb\n",
    "    else:\n",
    "        uc = None\n",
    "\n",
    "    x_T = None\n",
    "    batch_variants = []\n",
    "\n",
    "    for _ in range(n_samples):\n",
    "        if ddim_sampler is not None:\n",
    "            kwargs.update({\"clean_cond\": True})\n",
    "            samples, _ = ddim_sampler.sample(\n",
    "                S=ddim_steps,\n",
    "                conditioning=cond,\n",
    "                batch_size=noise_shape[0],\n",
    "                shape=noise_shape[1:],\n",
    "                verbose=False,\n",
    "                unconditional_guidance_scale=cfg_scale,\n",
    "                unconditional_conditioning=uc,\n",
    "                eta=ddim_eta,\n",
    "                temporal_length=noise_shape[2],\n",
    "                conditional_guidance_scale_temporal=temporal_cfg_scale,\n",
    "                x_T=x_T,\n",
    "                fs=fs,\n",
    "                timestep_spacing=timestep_spacing,\n",
    "                guidance_rescale=guidance_rescale,\n",
    "                **kwargs,\n",
    "            )\n",
    "        # reconstruct from latent to pixel space\n",
    "        batch_images = model.decode_first_stage(samples)\n",
    "        batch_variants.append(batch_images)\n",
    "    # batch, <samples>, c, t, h, w\n",
    "    batch_variants = torch.stack(batch_variants, dim=1)\n",
    "    return batch_variants\n",
    "\n",
    "\n",
    "# monkey patching to replace the original method 'register_buffer' that uses CUDA\n",
    "DDIMSampler.register_buffer = types.MethodType(register_buffer, DDIMSampler)\n",
    "\n",
    "\n",
    "def save_videos(batch_tensors, savedir, filenames, fps=10):\n",
    "    # b,samples,c,t,h,w\n",
    "    n_samples = batch_tensors.shape[1]\n",
    "    for idx, vid_tensor in enumerate(batch_tensors):\n",
    "        video = vid_tensor.detach().cpu()\n",
    "        video = torch.clamp(video.float(), -1.0, 1.0)\n",
    "        video = video.permute(2, 0, 1, 3, 4)  # t,n,c,h,w\n",
    "        frame_grids = [torchvision.utils.make_grid(framesheet, nrow=int(n_samples)) for framesheet in video]  # [3, 1*h, n*w]\n",
    "        grid = torch.stack(frame_grids, dim=0)  # stack in temporal dim [t, 3, n*h, w]\n",
    "        grid = (grid + 1.0) / 2.0\n",
    "        grid = (grid * 255).to(torch.uint8).permute(0, 2, 3, 1)\n",
    "        savepath = os.path.join(savedir, f\"{filenames[idx]}.mp4\")\n",
    "        torchvision.io.write_video(savepath, grid, fps=fps, video_codec=\"h264\", options={\"crf\": \"10\"})\n",
    "\n",
    "\n",
    "def get_image(image, prompt, steps=5, cfg_scale=7.5, eta=1.0, fs=3, seed=123, model=model, result_dir=\"results\"):\n",
    "    if not os.path.exists(result_dir):\n",
    "        os.mkdir(result_dir)\n",
    "\n",
    "    seed_everything(seed)\n",
    "\n",
    "    # torch.cuda.empty_cache()\n",
    "    print(\"start:\", prompt, time.strftime(\"%Y-%m-%d %H:%M:%S\", time.localtime(time.time())))\n",
    "    start = time.time()\n",
    "    if steps > 60:\n",
    "        steps = 60\n",
    "    model = model.cpu()\n",
    "    batch_size = 1\n",
    "    channels = model.model.diffusion_model.out_channels\n",
    "    frames = model.temporal_length\n",
    "    h, w = 256 // 8, 256 // 8\n",
    "    noise_shape = [batch_size, channels, frames, h, w]\n",
    "\n",
    "    # text cond\n",
    "    with torch.no_grad(), torch.cpu.amp.autocast():\n",
    "        cond = process_input(model, prompt, image, transform, fs=3)\n",
    "\n",
    "        ## inference\n",
    "        batch_samples = batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=steps, ddim_eta=eta, cfg_scale=cfg_scale)\n",
    "        ## b,samples,c,t,h,w\n",
    "        prompt_str = prompt.replace(\"/\", \"_slash_\") if \"/\" in prompt else prompt\n",
    "        prompt_str = prompt_str.replace(\" \", \"_\") if \" \" in prompt else prompt_str\n",
    "        prompt_str = prompt_str[:40]\n",
    "        if len(prompt_str) == 0:\n",
    "            prompt_str = \"empty_prompt\"\n",
    "\n",
    "    save_videos(batch_samples, result_dir, filenames=[prompt_str], fps=8)\n",
    "    print(f\"Saved in {prompt_str}.mp4. Time used: {(time.time() - start):.2f} seconds\")\n",
    "\n",
    "    return os.path.join(result_dir, f\"{prompt_str}.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e9a0137a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Seed set to 234\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start: man fishing in a boat at sunset 2024-07-25 11:30:59\n",
      "Saved in man_fishing_in_a_boat_at_sunset.mp4. Time used: 183.19 seconds\n"
     ]
    }
   ],
   "source": [
    "image_path = \"dynamicrafter/prompts/256/art.png\"\n",
    "prompt = \"man fishing in a boat at sunset\"\n",
    "seed = 234\n",
    "image = Image.open(image_path)\n",
    "image = np.asarray(image)\n",
    "result_dir = \"results\"\n",
    "video_path = get_image(image, prompt, steps=20, seed=seed, model=model, result_dir=result_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3734113f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video alt=\"video\" controls>\n",
       "        <source src=\"results/man_fishing_in_a_boat_at_sunset.mp4\" type=\"video/mp4\">\n",
       "    </video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\n",
    "    f\"\"\"\n",
    "    <video alt=\"video\" controls>\n",
    "        <source src=\"{video_path}\" type=\"video/mp4\">\n",
    "    </video>\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e8548d7",
   "metadata": {},
   "source": [
    "## Quantization\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[NNCF](https://github.com/openvinotoolkit/nncf/) enables post-training quantization by adding quantization layers into model graph and then using a subset of the training dataset to initialize the parameters of these additional quantization layers. Quantized operations are executed in `INT8` instead of `FP32`/`FP16` making model inference faster.\n",
    "\n",
    "According to `DynamiCrafter` structure, denoising UNet model is used in the cycle repeating inference on each diffusion step, while other parts of pipeline take part only once. Now we will show you how to optimize pipeline using [NNCF](https://github.com/openvinotoolkit/nncf/) to reduce memory and computation cost.\n",
    "\n",
    "Please select below whether you would like to run quantization to improve model inference speed.\n",
    "\n",
    "> **NOTE**: Quantization is time and memory consuming operation. Running quantization code below may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f991e3c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ac4286d32934cbb861c354857d48c0d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Checkbox(value=True, description='Quantization')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to_quantize = widgets.Checkbox(\n",
    "    value=True,\n",
    "    description=\"Quantization\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "to_quantize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03c3dfb5",
   "metadata": {},
   "source": [
    "Let's load `skip magic` extension to skip quantization if `to_quantize` is not selected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "61d199d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch `skip_kernel_extension` module\n",
    "import requests\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py\",\n",
    ")\n",
    "open(\"skip_kernel_extension.py\", \"w\").write(r.text)\n",
    "\n",
    "int8_model = None\n",
    "\n",
    "%load_ext skip_kernel_extension"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5ff698",
   "metadata": {},
   "source": [
    "### Prepare calibration dataset\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We use a portion of [`jovianzm/Pexels-400k`](https://huggingface.co/datasets/jovianzm/Pexels-400k) dataset from Hugging Face as calibration data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c1b7bb27",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "\n",
    "def download_image(url):\n",
    "    try:\n",
    "        response = requests.get(url)\n",
    "        response.raise_for_status()\n",
    "        img = Image.open(BytesIO(response.content))\n",
    "        # Convert the image to a NumPy array\n",
    "        img_array = np.array(img)\n",
    "        return img_array\n",
    "    except Exception as err:\n",
    "        print(f\"Error occurred: {err}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb54119d",
   "metadata": {},
   "source": [
    "To collect intermediate model inputs for calibration we should customize `CompiledModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7c141d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import datasets\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "class CompiledModelDecorator(ov.CompiledModel):\n",
    "    def __init__(self, compiled_model, keep_prob, data_cache = None):\n",
    "        super().__init__(compiled_model)\n",
    "        self.data_cache = data_cache if data_cache else []\n",
    "        self.keep_prob = np.clip(keep_prob, 0, 1)\n",
    "\n",
    "    def __call__(self, *args, **kwargs):\n",
    "        if np.random.rand() <= self.keep_prob:\n",
    "            self.data_cache.append(*args)\n",
    "        return super().__call__(*args, **kwargs)\n",
    "\n",
    "def collect_calibration_data(model, subset_size):\n",
    "    original_diffusion_model = model.model.diffusion_model.diffusion_model\n",
    "    modified_model = CompiledModelDecorator(original_diffusion_model, keep_prob=1)\n",
    "    model.model.diffusion_model = CModelWrapper(modified_model, model.model.diffusion_model.out_channels)\n",
    "\n",
    "    dataset = datasets.load_dataset(\"jovianzm/Pexels-400k\", split=\"train\", streaming=True).shuffle(seed=42).take(subset_size)\n",
    "\n",
    "    pbar = tqdm(total=subset_size)\n",
    "    channels = model.model.diffusion_model.out_channels\n",
    "    frames = model.temporal_length\n",
    "    h, w = 256 // 8, 256 // 8\n",
    "    noise_shape = [1, channels, frames, h, w]\n",
    "    for batch in dataset:\n",
    "        prompt = batch[\"title\"]\n",
    "        image_path = batch[\"thumbnail\"]\n",
    "        image = download_image(image_path)\n",
    "        if image is None:\n",
    "            continue\n",
    "\n",
    "        cond = process_input(model, prompt, image)\n",
    "        batch_samples = batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=20, ddim_eta=1.0, cfg_scale=7.5)\n",
    "\n",
    "        collected_subset_size = len(model.model.diffusion_model.diffusion_model.data_cache)\n",
    "        if collected_subset_size >= subset_size:\n",
    "            pbar.update(subset_size - pbar.n)\n",
    "            break\n",
    "        pbar.update(collected_subset_size - pbar.n)\n",
    "\n",
    "    calibration_dataset = model.model.diffusion_model.diffusion_model.data_cache[:subset_size]\n",
    "    model.model.diffusion_model.diffusion_model = original_diffusion_model\n",
    "    return calibration_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1213b8cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0354e90b50bd41e4b08e3af5740c7e95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/376 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3417657a28ab4649bed9f6d10c7f214a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/300 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "MODEL_INT8_OV_PATH = Path(\"models/model_ir_int8.xml\")\n",
    "if not MODEL_INT8_OV_PATH.exists():\n",
    "    subset_size = 300\n",
    "    calibration_data = collect_calibration_data(model, subset_size=subset_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c23abd",
   "metadata": {},
   "source": [
    "### Run Quantization\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Quantization of the first and last `Convolution` layers impacts the generation results. We recommend using `IgnoredScope` to keep accuracy sensitive layers in FP16 precision. `FastBiasCorrection` algorithm is disabled due to minimal accuracy improvement in SD models and increased quantization time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "37af0dd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-07-25 11:58:58.202753: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2024-07-25 11:58:58.205117: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-07-25 11:58:58.246671: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-07-25 11:58:58.247399: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-07-25 11:58:59.006398: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49fefe46d45d4569a5c878b4d80789ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "102df3b931314387a921bf6440a7419a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:2 ignored nodes were found by names in the NNCFGraph\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 65 __module.diffusion_model.input_blocks.0.0/aten::_convolution/Convolution\n",
      "158 __module.diffusion_model.input_blocks.0.0/aten::_convolution/Add\n",
      "\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 3789 __module.diffusion_model.out.2/aten::_convolution/Convolution\n",
      "4086 __module.diffusion_model.out.2/aten::_convolution/Add\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8140b8ffd17f4877bedf566c48c09e53",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import nncf\n",
    "\n",
    "\n",
    "if MODEL_INT8_OV_PATH.exists():\n",
    "    print(\"Loading quantized model\")\n",
    "    quantized_model = core.read_model(MODEL_INT8_OV_PATH)\n",
    "else:\n",
    "    ov_model_ir = core.read_model(MODEL_OV_PATH)\n",
    "    quantized_model = nncf.quantize(\n",
    "        model=ov_model_ir,\n",
    "        subset_size=subset_size,\n",
    "        calibration_dataset=nncf.Dataset(calibration_data),\n",
    "        model_type=nncf.ModelType.TRANSFORMER,\n",
    "        ignored_scope=nncf.IgnoredScope(names=[\n",
    "            \"__module.diffusion_model.input_blocks.0.0/aten::_convolution/Convolution\",\n",
    "            \"__module.diffusion_model.out.2/aten::_convolution/Convolution\",\n",
    "        ]),\n",
    "        advanced_parameters=nncf.AdvancedQuantizationParameters(disable_bias_correction=True)\n",
    "    )\n",
    "    ov.save_model(quantized_model, MODEL_INT8_OV_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011835c3",
   "metadata": {},
   "source": [
    "### Run Weights Compression\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Quantizing of the remaining components of the pipeline does not significantly improve inference performance but can lead to a substantial degradation of accuracy. The weight compression will be applied to footprint reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fe98145f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (97 / 97)              │ 100% (97 / 97)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8843fe7ba47d40d09fa86aadd366727a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (39 / 39)              │ 100% (39 / 39)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adca15bca575477bb181fc920e14aac3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (31 / 31)              │ 100% (31 / 31)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1b747e9b37e4a9a83581154d70a57cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (129 / 129)            │ 100% (129 / 129)                       │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bce5b2de47134bd682eae49cbbe262da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "COND_STAGE_MODEL_INT8_OV_PATH = Path(\"models/cond_stage_model_int8.xml\")\n",
    "DECODER_FIRST_STAGE_INT8_OV_PATH = Path(\"models/decoder_first_stage_ir_int8.xml\")\n",
    "ENCODER_FIRST_STAGE_INT8_OV_PATH = Path(\"models/encoder_first_stage_ir_int8.xml\")\n",
    "EMBEDDER_INT8_OV_PATH = Path(\"models/embedder_ir_int8.xml\")\n",
    "\n",
    "def compress_model_weights(fp_model_path, int8_model_path):\n",
    "    if not int8_model_path.exists():\n",
    "        model = core.read_model(fp_model_path)\n",
    "        compressed_model = nncf.compress_weights(model)\n",
    "        ov.save_model(compressed_model, int8_model_path)\n",
    "\n",
    "\n",
    "compress_model_weights(COND_STAGE_MODEL_OV_PATH, COND_STAGE_MODEL_INT8_OV_PATH)\n",
    "compress_model_weights(DECODER_FIRST_STAGE_OV_PATH, DECODER_FIRST_STAGE_INT8_OV_PATH)\n",
    "compress_model_weights(ENCODER_FIRST_STAGE_OV_PATH, ENCODER_FIRST_STAGE_INT8_OV_PATH)\n",
    "compress_model_weights(EMBEDDER_OV_PATH, EMBEDDER_INT8_OV_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8d7527",
   "metadata": {},
   "source": [
    "Let's run the optimized pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d311de3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "compiled_cond_stage_model = core.compile_model(COND_STAGE_MODEL_INT8_OV_PATH, device.value)\n",
    "compiled_encode_first_stage = core.compile_model(ENCODER_FIRST_STAGE_INT8_OV_PATH, device.value)\n",
    "compiled_embedder = core.compile_model(EMBEDDER_INT8_OV_PATH, device.value)\n",
    "compiled_model = core.compile_model(MODEL_INT8_OV_PATH, device.value)\n",
    "compiled_decoder_first_stage = core.compile_model(DECODER_FIRST_STAGE_INT8_OV_PATH, device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "a0ce59ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.\n",
      ">>> model checkpoint loaded.\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "int8_model = download_model()\n",
    "int8_model.first_stage_model.decode = types.MethodType(decode, int8_model.first_stage_model)\n",
    "int8_model.embedder.model.visual.input_patchnorm = None  # fix error: visual model has not  attribute 'input_patchnorm'\n",
    "\n",
    "int8_model.cond_stage_model = CondStageModelWrapper(compiled_cond_stage_model)\n",
    "int8_model.first_stage_model.encoder = EncoderFirstStageModelWrapper(compiled_encode_first_stage)\n",
    "int8_model.embedder = EmbedderWrapper(compiled_embedder)\n",
    "int8_model.model.diffusion_model = CModelWrapper(compiled_model, int8_model.model.diffusion_model.out_channels)\n",
    "int8_model.first_stage_model.decoder = DecoderFirstStageModelWrapper(compiled_decoder_first_stage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1e77da42",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Seed set to 234\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start: man fishing in a boat at sunset 2024-07-25 12:56:18\n",
      "Saved in man_fishing_in_a_boat_at_sunset.mp4. Time used: 84.56 seconds\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "image_path = \"dynamicrafter/prompts/256/art.png\"\n",
    "prompt = \"man fishing in a boat at sunset\"\n",
    "seed = 234\n",
    "image = Image.open(image_path)\n",
    "image = np.asarray(image)\n",
    "\n",
    "result_dir = \"results_int8\"\n",
    "video_path = get_image(image, prompt, steps=20, seed=seed, model=int8_model, result_dir=result_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d8a817f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video alt=\"video\" controls>\n",
       "        <source src=results_int8/man_fishing_in_a_boat_at_sunset.mp4 type=\"video/mp4\">\n",
       "    </video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "display(HTML(f\"\"\"\n",
    "    <video alt=\"video\" controls>\n",
    "        <source src={video_path} type=\"video/mp4\">\n",
    "    </video>\n",
    "\"\"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81410f5a",
   "metadata": {},
   "source": [
    "### Compare model file sizes\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f2134675",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cond_stage_model compression rate: 3.977\n",
      "decoder_first_stage_ir compression rate: 3.987\n",
      "encoder_first_stage_ir compression rate: 3.986\n",
      "embedder_ir compression rate: 3.977\n",
      "model_ir compression rate: 3.981\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "fp32_model_paths = [COND_STAGE_MODEL_OV_PATH, DECODER_FIRST_STAGE_OV_PATH, ENCODER_FIRST_STAGE_OV_PATH, EMBEDDER_OV_PATH, MODEL_OV_PATH]\n",
    "int8_model_paths = [COND_STAGE_MODEL_INT8_OV_PATH, DECODER_FIRST_STAGE_INT8_OV_PATH, ENCODER_FIRST_STAGE_INT8_OV_PATH, EMBEDDER_INT8_OV_PATH, MODEL_INT8_OV_PATH]\n",
    "\n",
    "for fp16_path, int8_path in zip(fp32_model_paths, int8_model_paths):\n",
    "    fp32_ir_model_size = fp16_path.with_suffix(\".bin\").stat().st_size\n",
    "    int8_model_size = int8_path.with_suffix(\".bin\").stat().st_size\n",
    "    print(f\"{fp16_path.stem} compression rate: {fp32_ir_model_size / int8_model_size:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa624ee9",
   "metadata": {},
   "source": [
    "### Compare inference time of the FP32 and INT8 models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "To measure the inference performance of the `FP32` and `INT8` models, we use median inference time on calibration subset.\n",
    "\n",
    "> **NOTE**: For the most accurate performance estimation, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7755a0cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import time\n",
    "\n",
    "\n",
    "def calculate_inference_time(model, validation_size=3):\n",
    "    calibration_dataset = datasets.load_dataset(\"jovianzm/Pexels-400k\", split=\"train\", streaming=True).take(validation_size)\n",
    "    inference_time = []\n",
    "    channels = model.model.diffusion_model.out_channels\n",
    "    frames = model.temporal_length\n",
    "    h, w = 256 // 8, 256 // 8\n",
    "    noise_shape = [1, channels, frames, h, w]\n",
    "    for batch in calibration_dataset:\n",
    "        prompt = batch[\"title\"]\n",
    "        image_path = batch[\"thumbnail\"]\n",
    "        image = download_image(image_path)\n",
    "        cond = process_input(model, prompt, image, transform, fs=3)\n",
    "\n",
    "        start = time.perf_counter()\n",
    "        _ = batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=20, ddim_eta=1.0, cfg_scale=7.5)\n",
    "        end = time.perf_counter()\n",
    "        delta = end - start\n",
    "        inference_time.append(delta)\n",
    "    return np.median(inference_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e61b1152",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "fp_latency = calculate_inference_time(model)\n",
    "print(f\"FP32 latency: {fp_latency:.3f}\")\n",
    "int8_latency = calculate_inference_time(int8_model)\n",
    "print(f\"INT8 latency: {int8_latency:.3f}\")\n",
    "print(f\"Performance speed up: {fp_latency / int8_latency:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4417db2b-2f65-407c-a384-ec466f18bca0",
   "metadata": {},
   "source": [
    "## Interactive inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Please select below whether you would like to use the quantized models to launch the interactive demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cdf535e",
   "metadata": {},
   "outputs": [],
   "source": [
    "quantized_models_present = int8_model is not None\n",
    "\n",
    "use_quantized_models = widgets.Checkbox(\n",
    "    value=quantized_models_present,\n",
    "    description=\"Use quantized models\",\n",
    "    disabled=not quantized_models_present,\n",
    ")\n",
    "\n",
    "use_quantized_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf57f8a8-8cf6-45c5-ae78-02a3ed04fcc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import gradio as gr\n",
    "from functools import partial\n",
    "\n",
    "\n",
    "i2v_examples_256 = [\n",
    "    [\"dynamicrafter/prompts/256/art.png\", \"man fishing in a boat at sunset\", 50, 7.5, 1.0, 3, 234],\n",
    "    [\"dynamicrafter/prompts/256/boy.png\", \"boy walking on the street\", 50, 7.5, 1.0, 3, 125],\n",
    "    [\"dynamicrafter/prompts/256/dance1.jpeg\", \"two people dancing\", 50, 7.5, 1.0, 3, 116],\n",
    "    [\"dynamicrafter/prompts/256/fire_and_beach.jpg\", \"a campfire on the beach and the ocean waves in the background\", 50, 7.5, 1.0, 3, 111],\n",
    "    [\"dynamicrafter/prompts/256/guitar0.jpeg\", \"bear playing guitar happily, snowing\", 50, 7.5, 1.0, 3, 122],\n",
    "]\n",
    "\n",
    "demo_model = int8_model if use_quantized_models.value else model\n",
    "get_image_fn = partial(get_image, model=demo_model)\n",
    "\n",
    "\n",
    "def dynamicrafter_demo():\n",
    "    css = \"\"\"#input_img {max-width: 256px !important} #output_vid {max-width: 256px; max-height: 256px}\"\"\"\n",
    "\n",
    "    with gr.Blocks(analytics_enabled=False, css=css) as dynamicrafter_iface:\n",
    "        with gr.Tab(label=\"Image2Video_256x256\"):\n",
    "            with gr.Column():\n",
    "                with gr.Row():\n",
    "                    with gr.Column():\n",
    "                        with gr.Row():\n",
    "                            i2v_input_image = gr.Image(label=\"Input Image\", elem_id=\"input_img\")\n",
    "                        with gr.Row():\n",
    "                            i2v_input_text = gr.Text(label=\"Prompts\")\n",
    "                        with gr.Row():\n",
    "                            i2v_seed = gr.Slider(label=\"Random Seed\", minimum=0, maximum=10000, step=1, value=123)\n",
    "                            i2v_eta = gr.Slider(minimum=0.0, maximum=1.0, step=0.1, label=\"ETA\", value=1.0, elem_id=\"i2v_eta\")\n",
    "                            i2v_cfg_scale = gr.Slider(minimum=1.0, maximum=15.0, step=0.5, label=\"CFG Scale\", value=7.5, elem_id=\"i2v_cfg_scale\")\n",
    "                        with gr.Row():\n",
    "                            i2v_steps = gr.Slider(minimum=1, maximum=60, step=1, elem_id=\"i2v_steps\", label=\"Sampling steps\", value=50)\n",
    "                            i2v_motion = gr.Slider(minimum=1, maximum=4, step=1, elem_id=\"i2v_motion\", label=\"Motion magnitude\", value=3)\n",
    "                        i2v_end_btn = gr.Button(\"Generate\")\n",
    "                    with gr.Row():\n",
    "                        i2v_output_video = gr.Video(label=\"Generated Video\", elem_id=\"output_vid\", autoplay=True, show_share_button=True)\n",
    "\n",
    "                gr.Examples(\n",
    "                    examples=i2v_examples_256,\n",
    "                    inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed],\n",
    "                    outputs=[i2v_output_video],\n",
    "                    fn=get_image_fn,\n",
    "                    cache_examples=False,\n",
    "                )\n",
    "            i2v_end_btn.click(\n",
    "                inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed],\n",
    "                outputs=[i2v_output_video],\n",
    "                fn=get_image_fn,\n",
    "            )\n",
    "\n",
    "    return dynamicrafter_iface\n",
    "\n",
    "\n",
    "demo = dynamicrafter_demo()\n",
    "\n",
    "\n",
    "try:\n",
    "    demo.queue().launch(debug=True)\n",
    "except Exception:\n",
    "    demo.queue().launch(debug=True, share=True)\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "# demo.launch(server_name='your server name', server_port='server port in int')\n",
    "# Read more in the docs: https://gradio.app/docs/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.gif?raw=true",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Image-to-Video"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "0354e90b50bd41e4b08e3af5740c7e95": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_3ce2399bcf714c3681f07328c58c850b",
        "IPY_MODEL_cc0fbe6edb7d4488983633030d6572c8",
        "IPY_MODEL_c4067386014b4331abdc4e5860dc6002"
       ],
       "layout": "IPY_MODEL_8a7f01541bf544cdbad6a9654dd03341"
      }
     },
     "050ba2deb97c44fea4932980babcf422": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "09ae14d72d854422943609e79dde62d5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "09e2c5cca48145249e558d5b2a8d9f07": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "102df3b931314387a921bf6440a7419a": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_bf8ce7c7b5f54a739927c480ea73f0ae",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying Smooth Quant <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">269/269</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:33</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Applying Smooth Quant \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m269/269\u001b[0m • \u001b[38;2;0;104;181m0:00:33\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "10bc1275f679401b83cf92d460409c8d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "15711837c7084083bc248db140a32da7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "1f8bda015fad48f18ab8bef827fdb4f6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "306b95f37bee4e1192f1e6b28979259d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "3417657a28ab4649bed9f6d10c7f214a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_a9805e5163d9440092570f5c5c122d30",
        "IPY_MODEL_7d47c3b2963c44a4b05866b64fe62838",
        "IPY_MODEL_883be8e255984be5b767531b3238c1f5"
       ],
       "layout": "IPY_MODEL_b0800f0e5b7345a0bd0482c984412b2a"
      }
     },
     "35243ae51b58460e900598c06ae92547": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DropdownModel",
      "state": {
       "_options_labels": [
        "CPU",
        "AUTO"
       ],
       "description": "Device:",
       "index": 1,
       "layout": "IPY_MODEL_9de71f74c3224737a74e3343afefdd44",
       "style": "IPY_MODEL_c83b8b4248bb4a799ddb9a145794bd99"
      }
     },
     "39c4a84a10e94541b7aa5280e5f95481": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "3ac4286d32934cbb861c354857d48c0d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxModel",
      "state": {
       "description": "Quantization",
       "disabled": false,
       "layout": "IPY_MODEL_e5d12e2ab1d74ac9af1f368db0ac7cdc",
       "style": "IPY_MODEL_c37a185add934dfb97c07dae77458c36",
       "value": true
      }
     },
     "3ce2399bcf714c3681f07328c58c850b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_10bc1275f679401b83cf92d460409c8d",
       "style": "IPY_MODEL_8e8e73da453645f486f482724d65d970",
       "value": "Downloading readme: 100%"
      }
     },
     "459ddb60988b4bc2b8503d16fa8190bb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "49fefe46d45d4569a5c878b4d80789ac": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_84b83c54834e41c583def37c2f8b5da8",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Statistics collection <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">300/300</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:23:53</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Statistics collection \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m300/300\u001b[0m • \u001b[38;2;0;104;181m0:23:53\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "4b374a11575045d492b37ccb3d4069dc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "61df30552c794fa8b6144efc104b3d35": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "7d47c3b2963c44a4b05866b64fe62838": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "layout": "IPY_MODEL_86188feeaf82478696c31c68779e29f3",
       "max": 300,
       "style": "IPY_MODEL_eff5c1ea328a4c31b54b03b018088f97",
       "value": 300
      }
     },
     "8140b8ffd17f4877bedf566c48c09e53": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_306b95f37bee4e1192f1e6b28979259d",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Statistics collection <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">300/300</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:28:57</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Statistics collection \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m300/300\u001b[0m • \u001b[38;2;0;104;181m0:28:57\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "833bf18bbcda47118cbf7782d281d753": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "84b83c54834e41c583def37c2f8b5da8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "84f1e7e84d484043901f8b8ad0a043a7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "86188feeaf82478696c31c68779e29f3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "883be8e255984be5b767531b3238c1f5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_459ddb60988b4bc2b8503d16fa8190bb",
       "style": "IPY_MODEL_050ba2deb97c44fea4932980babcf422",
       "value": " 300/300 [24:31&lt;00:00,  5.44s/it]"
      }
     },
     "8843fe7ba47d40d09fa86aadd366727a": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_1f8bda015fad48f18ab8bef827fdb4f6",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying Weight Compression <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">97/97</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:01</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Applying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m97/97\u001b[0m • \u001b[38;2;0;104;181m0:00:01\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "8a7f01541bf544cdbad6a9654dd03341": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8e8e73da453645f486f482724d65d970": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "9bb461a1a3124cf6abed1bb725adee32": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9de71f74c3224737a74e3343afefdd44": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a9805e5163d9440092570f5c5c122d30": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_9bb461a1a3124cf6abed1bb725adee32",
       "style": "IPY_MODEL_09ae14d72d854422943609e79dde62d5",
       "value": "100%"
      }
     },
     "adca15bca575477bb181fc920e14aac3": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_84f1e7e84d484043901f8b8ad0a043a7",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying Weight Compression <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">39/39</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Applying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m39/39\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "b0800f0e5b7345a0bd0482c984412b2a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "bce5b2de47134bd682eae49cbbe262da": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_833bf18bbcda47118cbf7782d281d753",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying Weight Compression <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">129/129</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:03</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Applying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m129/129\u001b[0m • \u001b[38;2;0;104;181m0:00:03\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "bf8ce7c7b5f54a739927c480ea73f0ae": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c1b747e9b37e4a9a83581154d70a57cf": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_09e2c5cca48145249e558d5b2a8d9f07",
       "outputs": [
        {
         "data": {
          "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying Weight Compression <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #800080; text-decoration-color: #800080\">100%</span> <span style=\"color: #0068b5; text-decoration-color: #0068b5\">31/31</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span> • <span style=\"color: #0068b5; text-decoration-color: #0068b5\">0:00:00</span>\n</pre>\n",
          "text/plain": "Applying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[38;2;0;104;181m31/31\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "c37a185add934dfb97c07dae77458c36": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "c4067386014b4331abdc4e5860dc6002": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_4b374a11575045d492b37ccb3d4069dc",
       "style": "IPY_MODEL_15711837c7084083bc248db140a32da7",
       "value": " 376/376 [00:00&lt;00:00, 31.0kB/s]"
      }
     },
     "c83b8b4248bb4a799ddb9a145794bd99": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "cc0fbe6edb7d4488983633030d6572c8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "layout": "IPY_MODEL_39c4a84a10e94541b7aa5280e5f95481",
       "max": 376,
       "style": "IPY_MODEL_61df30552c794fa8b6144efc104b3d35",
       "value": 376
      }
     },
     "e5d12e2ab1d74ac9af1f368db0ac7cdc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "eff5c1ea328a4c31b54b03b018088f97": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}