Seminars/Seminar 03/Основы Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Практикум по программированию на языке Python`\n",
    "<br>\n",
    "\n",
    "## `Занятие 03: Основы Pytorch`\n",
    "<br><br>\n",
    "\n",
    "### `Находнов Максим (nakhodnov17@gmail.com)`\n",
    "\n",
    "#### `Москва, 2024`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "О чём можно узнать из этого ноутбука:\n",
    "\n",
    "* Базовый синтаксис и понятия библиотеки Pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:21:09.482708Z",
     "start_time": "2023-02-28T12:21:08.354796Z"
    }
   },
   "outputs": [],
   "source": [
    "%config Completer.use_jedi = False\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Почему torch?`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ключевые отличия от `numpy/sklearn`:\n",
    "1. Поддерживает вычисления на **CPU/GPU/TPU**\n",
    "2. **Автоматическое дифференцирование**\n",
    "\n",
    "Основные методы обучения нейронных сетей являются методами оптимизации первого порядка. Они присутствуют, например, в пакете `scipy.optimize`, но либо считают градиенты численно, либо требуют внешних функций, предоставляющих им градиенты в требуемой точке. Написание таких функций вручную для каждой нейронной сети — задача решаемая, но достаточно бессмысленная, т.к. производные, даже аналитически, достаточно легко может считать и машина.\n",
    "\n",
    "Концепция библиотеки `pytorch` — расширить функционал `numpy`, добавив туда возможность автоматического расчёта градиентов произвольных функций и их композиций, стараясь максимально сохранить привычную семантику.\n",
    "\n",
    "Особенности:\n",
    "1. Синтаксиччески повторяет `numpy`/`scipy`\n",
    "2. Большой набор готовых компонент для реализации и обучения DL моделей\n",
    "3. Для решения домен-спецефичных задач есть свои библиотеки\n",
    "    * **`torchvision`**\n",
    "    * `torchaudio`\n",
    "    * `torchtext`\n",
    "    * `TorchData`\n",
    "    * `TorchRec`\n",
    "    * `TorchServe`\n",
    "4. В конкурентной борьбе победил конкурентов в исследованиях и образовании\n",
    "<img src=\"data:image/webp;base64,UklGRhIxAABXRUJQVlA4IAYxAACQ8gCdASrrAs0BPlEkkEajoaGhIlBZiHAKCWdu/Bjc7UOwESniohpKQlaxf/afjB5PnkfCf3j9kPyo66vl7wX8mGNx4xlV9YP6P7nffz/0P8P/fPgJ+Tv8j/Wf3s+gT9Cf7j/Q/x8+dv98/1X+V9xf8p/1X/A/tf+S+AH8Z/mf+//tf7x/Pd6Ev/f6gH/D/vXo8exL6DX6oel1/7f9t8Kf7Uf9z/PfBJ/PP8b/7ezt6Zfpz/TPxX8FP7d+PH7zetf4t8y/Xvyh/vf7N/Dh+yfld5jek/8X6Efx/7YfWf8D+2P9l/d34c/5Xgz7+P6P8s/gC/GP5F/VPzB/sPmI/lJ4mea/23/e/mF8AvqD8v/xP9x/bb/H/tp7AP87/e/1e92/5T+Z/178yv7Z9gP8o/lf9w/q37J/4L/++7B/qP6H/LfOS+y/2j/gf4D4Av5F/M/8N/df9L/mv8h///tg/fP9R/hf8f/zP7/////z8ofy/+8f7P/Kf6T/yf4n///gR/JP5x/iv7h/oP+d/ff///+fvG9lX69+xr+oX+u/P8XRUHVr5fvnY/fOx++dj987H752P3zsfvnY/fOx++dj987H752P3zsfvnY/fOx++dFpajcYRed2XAR5C1UMStYZzjh0HLrOkVojQ3CFkPN2Uoz1Agkh1Z8/DGQUj5saWWQTE2tAfeq79tVSfTe9RlQPQFRExMrXgS+9hCS/qakIgLQMWm1StsdmEbXu7+UlHFDsfRkMomg3ZwO2fT1ecX45JdIT5YUZDPvfkCGWCxDg9juBvCwB/NW4vD7+R9vlRXzqo8hJZcranZHfy1o7GAW0NdwXVwBcCP5owVdLF7NyP29C7nMZrAv2PY8RQAeZUhNyf6fmiRwnU5hTtvW7EIl5vRFMm8hffT2qxflFP60xpUc5Je5iLlMgBvPiQNRfrbc0wLM2ANUW/yioKCz6BETxMnblfnSYcbRBTMkRM9Cq2yg9qmjEp2IXIz/UYeuSFqjlujBMIVo4yytvHLuMQTfDj/UYeuSFqmjC+a0e7kZ/qMPPZAxxN7VaygmCKwyItj/Zy/fOyJOxfvnpSJ2JoMmdFx7/UDSJ2P4Nx987H+zl++doiLY9vPZ0XzZZiAJi+1W28RFsfwbj752RJ2L9dVXzki1ELJaS43i3mPvnaIi2P3z0pE7H8G4++YlwY4AhpKllbji7AFgOdZWDMT+iO5I9EdyR6I5BrinlQmOl1oTNralmCChUt1jlN1snc9FNSr2OlxgOh0u0Fd+9/hXEpqq7bgUsJ69soD+khvux3gFhk8xp9q/U33zI+1WtFAw9KROjrmZZcQOBTSLMGuFA1ip4ZtRzPhiPoAVlN/7bhK7i7sm1kcI84XLGI4s/BUAet3FrvjOmnbj2TFOUglbv4734UmqisSgYekcxo0cIuHfxF4/e6WMsg8lffDylxzdMIp5nFRqYvs4dCDs0IokDaSi5rjHX6i/pdIjM0JfA7QRLL85ET/y3Uos8zQiwvtVr6ERbH76wyItNo6HECo/WBVNZ6m1+HLfWKfF4phR3zZx047kJUR3JHojuSPRHckeiO5I9EdyLgBBytqxgf7oRTZ42NeRxZ+AJcJlJMYpjvrHvxA2m/Zf5juSPQhSPRHckd9IIqBL2gjEL7MpAMZ/PEWL7rC+1WvoRFsfvrMmvlNBgyic9FQW7QTP6nRwyGTA8N5LYjqK+bH1VYGrMuGzlmBwJnRfdYX2q2A+75e3ns6L5PZWS8EAW+b+BmIn20FAg9NQIMwyOJtaVlqzeQHMaKXakfrvsnpZ1CExHrreIjwyItj/deIiiXBjf4SKZE1c7DPGVKcEsNU+2X+Y7kj0RsQ/E3kyTPJR+pVt7buXksDYFi9kR5SiQgWmJ/CHVPlFd3l6tLh4XiTJSjVrYv1WHCvAaROx/BuPvnY+/Sq5bsZNXFDhwaaIbeJFxljgMfXkyBItJy/erAux+9WBdoiLY/fWGRFsfwb4uWeCvRfBTgdFhYqb2rFkYmT59kM4FTgDPW53mkRabR3+zl++doiLY/fPSkTsfwbkUA7SJNg6uFyvlzUZxSRacwdRFYZEWx/s5fvnZEnYv3z093zHIDGcUh7OBdb7OC/d1hZyHaCvlGiuvx6jnQ9ywTWT6r17qxsmSM/1GHrkP4+Rp9uX4xvzZIOBFWemVwmLsmdxhgkEJisPj7uY7Gu3Tn16Xr35s0yxGaE8KEK0cZY0/byGF+28nysUgtavqm0K0ranefRzLAKOyIEFaRFsexHqNkaaJ3eoKvhGlo8JSu8Hz37slf1a9e+64OMS/9PQ0vb3qoN9hKrS1j5EY+q448g9v/Z7VIdNXaUv+J2gjCpivefEWxCp7aulv8riZzuu+cMHTsUoXxBQ3zsfvnY/fOx++dj987H752P3zsfvnY/fOx++dj987H752PVkAL0et72UArMogkNu9Kbah94ZC1xuwwomCRQnxHYObcH3o9IZiLCX7p3R6LWiItj987H752P3zsfvnY/fLmTl+oF0lDDSaBnc2qXAEcco7YvgaUH6TYfMESX54Ydns6tfL987H752P3zsfvnY/fO3XDwvrL/H/arXy/fOw4AD+/6wCAAAAAAMNSAqrkg3LnZgTVFP7d7lcE4iBiXlEWsTYkyf5osPZqOkX9eVC02a/4lm6dCLru21Blqpl2VMSMP7VRfPEZOhAoDAXidHWZYfZKbcjS8Nay+HpJ7rqtpdp7iM7/geWQJlLabMIab050wyCiaxCDQYWtH0n9cEk7Aexi+Ex+IFo+BpWgmLPd58IPXGcIDy46geOd1LvYc+iNY0WmCJmr5rN57ixrKe8oWfMyJvD6Zu0mLFk3yks4VSJdWeh3IVdZYQEvmJfRjuibNp1ccISpUtrDEgdJSoMajnVHQZMMqyPMGldMlo158da0vQAC3IG8et1F1wTAIbCfAygZSovIejAg26i12BGlJRHasFstap5mRkCHydOc/XoFo//4CcgtxO7wY34BadbpA7w1Lxx4oEzyco++pVG5bAJAxDe4SIlyq7sLGHdk2Ay7xQN0WgviYt6AsbCevau6lcX/xwCrh4mhoLnSfs+At4KASDKVEJly3+wg1m41c/8cbf/CkdEDGCv90G6molouxRH8O+ZL0L22IEkPPzYTnpB1CU7bGJ+hM5kaOq9r9KySFizMsUaLBCJVZOZdI2c6N2TqC3XpNg4kWfaF90cLp04kGWr8D550lGoBWzmyccm8FOTxJW1CffCadrYLZM+yiewhsBVNHoycwJu0I2juICnPCLwxcebOr1rdZiZ4Yc7wCx/TYAKJ5cZL3Qe74iM3kmPk7DZjuXYMBP8G0xv7BzRyjOEySEgb0KfWlTYjRPKUrAJeD7n4t02VUG0S+deKm787Tk4pxEsrySfgnoCGr9z7XQd/TWjGRZ24w5Ze3h/DU9zKWpeXywoWy1eiA5BOcwAzIf/kMlCtIJeoKhO0pYx6vKEMoVTGKB6wtHBM5rpAtRQB11s0l9ygvUVroyatujsM9QD7+0tQoa/RqG3xPN9wI+mvZeQsxcfZ5DpUSsQYpHEFuJG5pJ1yza7SFRra5oSLYEo/8P/JsZvmgQ96t33yaOb+GYBmHqmdRiUMgZw7eCjhcbx8mJdw7YIeF6loNVbfVMU54nb2MUu+Gjmt1x8rK2JO6xGhZ5SIff6UuLm9zSvZ73t1iuEaHe013mwVUGPbgwbuhAMOZ4CLPPUbO01Ybo8kqJDsJPh09MekBLW3Om0QDYSqv4kVTtyO+dI5Kyzchge69qS+qJRa2kU5Y6xODvcdRcl4SYjcoL5nQLDtuSGC5GFjigxVA/HU+Ji77MhtNqSkJNbWI1vDHuUOV8NeRYrxG2VWwWQXV4b7reqOUNFByRKm002kH/9aSuQ7evKnx5el3bAGemDtoosiNDjRlbEfIerQnO11b1H32mIVLlMoMtunKhshToWYpjFIsg5JW0S16EMQLlGKxRnSF0mU63FzjuZug3NdHUOUSJxFL6SgeSDa4POE4T7fbJVY2cahR4XPPCCapw/eV5eCHht9OTPTfQCf3TvllLyjRPslAmfTQy0YrDLzFG0CwvaK8vMM760AQ0MmyrG/avS6a2lTvY/93KdFZD9qekQ4wgoKIAAjGhIjgmbxLY3/75qTVKniEwA7hw7IfJbrourCvUypW5tzCcAwND35+U8E2nEcverVNRJqRR19/G5wUqCU93lIBgTVK2VLdCLFqTBrQ4Mrdt/OP1UUxrp1B4eYFup72Xx+QRJd9o//sBKBHvUcyLDU7MUF4RrYyHZx8A6hFrTZuASwL6gMDSMfzzRNzMt+XNNf7M7A8W/tXYHXcRflI9mq3IBbCc5KAFS7II6EUTeSTPYJ9ArJAvyPeFDk5GA2m9D05uwViozOTRQvwB2jhQYyjoOoIjNBDyeTDlMSI5fOWXyUGbnjaygnc5K0bmtEgFEmkVSW8KHFEN3iVVBUFydUKGILwT9nwFvBIdRSR1/eCQsIJAMOzIQwDxhlWwck/A7y3kPNzcxfV/gATocRnZIkAzph+PnCSB3+BRndGItFMlYbeeRbsTlXgdMxswWapq7IwgiUrwF1eSYGgN8fVfmlgBVH+/m+jfM2B02jInvxVdVJX09qu6BuV5uH8XaqUnLqY/414Uu4xy3/i/OQYWx65ZTc7Eb0fIlMJz4DqdEUBSTHycv++ak1Sp4hTdozggo6gS4WOj1Wdirn2nitlasPYGUltGvzfvBQRWySIiNuoYUS7NH28UBApey2K/HDWvpagUBbVSnlcJTqaICOO/RNdIQ8pgigtjQog9YprJKrLM27YbxN4nrAA9jSBIZ2ePLlo4Zwv9Wj7wGOUlkUcT39OA6G6ETIuM0gwq9F+ZuKviN8R3GdHp4OVZbgdtAoUgqbj1jFcPgUTxB4quZovChTwpevNBredjw1R1EOvWjDKm3w7kDcxhsrbcrALktRRAT/rWjCFZPmGuNWKqXChxb/rPNB9GQFM9evD1kbZyyuxQm8EtmMqTTjb1dfLXAFePuC+GPP2nmeEE8/UC3Lwo1kDsnfUYNVRJaPluKyagkL+lpxYpCeGRy9dxlIIxcbUnwexnyOrZrpACzmed9yR2r6gYd8b6uxIhBSq+HlEDpzNe75Nx52jzaekswK/voUwBIii1rvQd95DWGQ9eJIQe7BiZhRajY0MIeGfGY2QR6gyox5hR1mcbgngOyJ6Qe2kjtM9PCIVIC9ThdsJ/+n72Krsea/J9zpX4MlT5BG/AZQUPhJOY3+dGodCayRp8HnPE5Or/rzDy7/pTENd2uTTNnY03OnKntYe5b3WGyXjCYNkUw4XsJYU2m3cNkW6MkaFX3MQSTqDttvZRpMS6J1J/0y3gHBtBZQlzzsyP5/fakgcAZvtW97eRI84Cfg/cxQbYderIbqH0oZJNK4i9li3LsfCsnb9RTr1WJtK+vvTC8q9CYlRe9K8ZmJ3Qr/NvmwO51Xi6vfZ6JsLNGEDWSbcgcSVk/Pm+MUa3e0iG85v9qOcFTZVnaUPrLTUfWTeNwcedfCJ8Zcjn0jFipVFHNDD0D5yBTGVsfqCts3gGxEDsAHsne7wLEgBHS0QbHy70haWw97gQd2I8jSPmhYPLEkiFolX20Av6uvaX4824YVqMCDmdWKfLhrkluSOFyh83LAw11ILmDEgOoZ6jy8ZdDBbsArdJh0OXMxU1ZJoW2jao22OKGNXVxfPhKsr1QgvSsTlBYPbiVuh9abVTksCnLofbHJ5gBxncdaTkvyVePPGANK2m1vDxfmYigmVdWvAk6ARqHj+L/7L2QLmctn6WuiUw7jim2MEMGOLepk/lsQaOrr4Ks6QvpVfq3fPQaVz3eCCaFFZhdWTTN5/mAp7efr0sypAisEbZUku1OZbPjAQuex64qfhRTAf0myet8r8WGP99+ynakDDBf+/a559YLFU9YFZQNIKl0lHSPH21aqxeSaFvNog/iIFHhKnkHvYIqj8HLlanszUQXbJ8Yxet/et62gzhY/sV5oy2d4JcVTX6fNw/Iy4Q83qMyigqLXb/Kupjc2KunA8TFh1efoKj96oOSNnn0J9iOIBgDdUQvJAw5eX9uyPIpujW5oMBEzBKPgDCHFvorkFxxG2cerAdBOaCbVNsKyPluIBX0etey/ua2Yyd6vP7pNuQB805PlKAvkhG6JTrnkZzIU89dkdKZq4aZqPH9P7W9ULPZZGRGd42s+fDazyUlNRlCSHvIAaDjUkdnzWIvWd8JLjyM0gOMg2NZz2RnybYys1HpHrkGxA0bmI/rInoG5Ys+yB0NeBgGXcmtRHID7BpYXuPT9ZH3kmrwF/8TOKP2vVFC9ONJIaA0n6Rsqt75EaXSDwreF9l2Gsj6TuY99jvE7/fLgN5MQemgnkEjLprSnds8qWBWBNmVOGoGUISjtGfUOdwA/Qcs4Pvc3TNq1dnqqhsgmiY6FKj0kO3tMgAGY2LMhWdHfOJmVecTSARIGe+Sr0xkQAAAEBYM/SltyAZ7rz9GOsVFYY3qrxVJzCZDkf6x5UXTaszkWTZfZ7V3CgbwEpZDYIoJyydaAooD9k/Mjyoh+Ntc7LE87BUChSe4JBYZ4+HDa/xOokcrgHX+iiZMSBBCFELWtrgcUUYziS0XNgEWCScsdSiiGuzByQSr83GfK6B3Q1KXUvlIeYLliRKxF9Ryc4I/kyfVI8RKwbkcDkXz8ckeuDqFGzgSf0nhH+guRotihSUhP7ot8SxruYmwfmh/BGbd4qTkOXk0pR0Yz+zO2pdpvfW05iflP0ZOJ/nOPuhQAsBNIlEXqv32sYvUGedIYKq75DgZudIHFvMUxGk2h6FAraq38nvrhetqHLsDhakts2lW0rrZSKa/JTvxYFh0zCCkRAdBch/aaBZBWrybTQQoK1eTaaBZBWrybTQQoK1ZQmLd3+CtXk2mghQVq8m00EKCtXk2mghQVq8m00EKCtXk2mgKPv1YLqJev3sn2mghQVq8m00EKCtXk2mghQVq8m00EKCtWNMyzcDBWr9po2mgWQVq8IS4DhrZTmLBDoVXLB0nl82KWC3TEMTlj9ccoVdBK/oEn+ZgAAAAAAAAAAkZoAAAAAAMZBnQl69LplR2n061l+b9WCvRvO3htb6NmWG16QPAOZq8ktxxeDtNCO+o9+SLYAaeRfAb9ipKPhaBJYGzMBWeAoVFGeDqPKHUYPZAAAPW+UuTuOZVr/nx89OqNUrnzYEjJ83lD28fV9t3Nv5ozrFzEQQqV0N/ba8+SudihW7tV2sDDAj5f1c/gBEAkZb/vRL5XZKqW2hARgRyZoZDmzooT7di3JWryxW35TmR0geIDOZ18JC24NXxamHKi29C4tfCvaiQnpQBmFEyATSHGxNmZKH6cBDfNvTSPR0prDcLOhHJW8klwM2MeSLf7z2z5s7F33QS6uYH7Y0CwWvd1PxYSySjFderKjHjVuExYvB64+uxi9Ybl0XJGk+ThTFINFYLKPd/7hCwkEukIzoaZfjxCfOPZ1Y2lOLr/HNmcXMDkHqFLYP5B2p6xPWMGLwkDrYTUo6FyvmJtr6dKZ3CuIG7I3z5IdFvVZCPF6pW+nL7pJGOOolhAwCQ7RSF3Ta+0kd9SKbaLU1Yqb/qgB4tgrK3GgO0/8FwOxlLO2SpLT5xjiPAGuBaAzweH9sntQdIjI5xQP8W5YKlyhD9MzjrdONJ4ylYeR15YI9hOPqwFON0Sg+RbE+khFimfpMmOgXyxsfgTQUvAiVFR/q1IsOGLiGKO2yAkSTGIntB7938m5URE6wHmVmy/irwgDaLJBkFejAWw/CsZCWA8dxA+SDVxcWY6iT2BedKuGghIPhkYIt2xoIlDdSdS/l39rJvuejRxxQuj8AGBbBCdYHkTe70n6XbVXUYjaH5UruK5y0OXg+YAo0yn20QVoU6tK0v5aPGpUmm38K9+tJnnZ1x18MizdoFTO5TxwUSHVOPR2/PFYDrDLmt1j3H6KC2jriuShYUbi7NZKXMsS1fB70l2R+5QWmq7kkxyg5170yzgp1mA62mx6xuE+W5Tcrvu2vMcCgYrSNM6T4sCro204AK7Dt9ePI4xJZWUTKP+RY1NLLfOAV2qLyj+xz011CcYWcag1TvjrB63xLkXn+kazjv7HBxGdNo57JUsHhqiyW+5wOMbLIYd58FkE0keNYOKGnfnrTQWKtzPpIE2G0pGRZUhH3fZTIPxj04LGzIdwHXSzSE8+YcDCZAlNVfcdL8Dw/Fud9i7sMGaItLeXKDxCBGBGCBLky9giIHOvGXFRSqjmq42auwdoHcqQ4GZivr8cflyvzgfxenYUNZ+zQ2EcUhSTyI4q988h8KNQPnKzVhJ+60ucZrkr78IMVo3VfTRvgQkDdlugXzNCWOKyDIn3/cpUVMs3lG9xE3YKih7K5wp0Ie2iq3EFX9Q0qxLv6OhbG321h9SeACxBordlaspxEeQJPC9I/b3WyIz/fULe08V7a0GUYlDrGIBsRnHAPonZgn5RLIv6PvA25vURGyz0wZi95FGrOa/7AAlT14vENlwra3N6OLkouYk7AX6RTyNzuH6PoPPIIIaKaSM4kW1O09DP8621mPoQ4eHbHfCnLzSLP4HkWOT55GHUh/jB31Wsydd/UHZt8lQFpRHNqsT7A/GEUWPYlRbx0rMZ2fBiRzk7UI1fLqIlMnYiF9QEai/CdZ5JJ3JKHm7/cp0p5Yvn2OEPpShUpAFOZ1qfnTXKweVzZ3XwvRk2a2IqAd+WdJh0kRJ/4E3N3jr3y6v0nE8jgKTiUgD1Njm9Pwzv+4PPPhgpnMWe/894H5ApfPJLYu1gYA5IWtmTODH73VUIp3xXwoYiV0W8awICJSoLSL76UzQDQR2krHyTQJsuDkGimx3MQKuFStGVIKLpy+lzr/Qqj0ZHIrXRC3HA2vGGXdTjM6DPJyAQi2W3hlCN+MKNxGfcaE30NRSKAz8i2wlBS3BoWgwBtRLjjXS6vg9pAGwzemAEbRFGSKqjHDGOIpDT11+GJOo+B0jQcRU/uBDUm36byhzno5GEBi51jWSUj2sf5/qtbrN5TRorunf2zAtR3Ik2p05zbBlTt6fw4ekLgnyY/w6gOHg2SKOPCzsmTySpWZtFx2OGjOVfDSedW9Id94laN+/42A8MCyTmQe4uN6zrQ6g4yIPb+/E5rsabnr5zldwwBA0VPcO8Fa9+tpig94AzhBmnSMg3ef/hk6KaDzOBrULHZiQS49oACSh83aREqikJcCfnSPoAgeyPEt9u6S+RLBaObpTzUQJmp8JjYF1vvi/bwKRvu7jCwF6oBvtetdumRzSZNNdkQSgn9iVJiQvH+9LwSsISz68rGsBZHJXS74WSwYalYTSIgrtw/CtJJxwAAugOWeB+veG0IPRdPg3kX+pm9IXKZThu8Qq/OoEkeIT+2x2jZBkeJJobNEqKGtWQYe9YENODU2c8kzJkgN1mAqkbNvTSdE9grnTKEMnYgR2Nc73H+nfeqKcnygAVoU42esjPrGIWE5fFIxU78Tnv7m+fAdrOC6NYgS7jqFqS57Ij8dXFYK9EiYZmNR33HcDueLDIESKVFy/BQb8aICLGVg8SVTppYKqw8J6e3c0jjx53CXBa9Xov1BiMx6mOrMLH0qRs3K+9vZKpsejhAkOapiAzrxgmgG4ejO6Lh/OoiHm7WxtIjfHwGZHxP0WEHvvYz2pYDJC6oUlHRbE+Okci3Q0vbLeHpkot8U6cfHii4TouCLJy8Qi3OLEfvDFmBGlQweojlSSh3NqVQI/ZTozIlDhbQnmhuCkrzvrE14DHXgSlNs3OMQTdHTn8R+gvLIQOwXL2EyXiseqGKizRpnbKdC17Dscl1jgAYZ0mlQ/SGEmgZAAAAIj+/+Y7+7YAQfGEinxKThxQE44YLZ3Dxyuj5IJiPaVvSnCQwcmE5Wx4x/BsqwHxSojE0zcyRtGOp/155KoqG0MM5Nev3Gx6XMs7S3Gw/6ZO7ZdbinScUfVXKD2v+k7xALMcd5AdCO+EF6QbS8UcAUTDVW2vSYzVyrns9d2/RFlnBLMQICljR6rFrwFi/JKzL/ZP9aZ0ypFZXwenglNhtn+t07a0kwIg6gXiYTC2Bwfc8RTqTCj24wZhtX17wQe6XrB6MyBwqg08e60+KMokUD83Qbg79Kf+vG7RCAcJQIPk+pXyDzyD2Ub64MHJSZN5sxS2hpLPSlMFvazIZCTp3vK49MMmC5D+d+Q0B4I2TtxeN+ys9zOOZYJju2lzzHMEGtF716eC2azX6S8GB4edgh/XW/9POnG3oPCWNPHNq96gsCQmU0nOD7x1dz8+Y8KIDoqY+nscm7hC+cfustXbz+I7pb1aTOPD+d1UZsCHJBNcMvPC4u0BhlLFMEQIjcDUAEFoezSvK7wtrY1VnGgyviRV9RHE8YD/qwTi8UV0mLDGj+r/OhHJMoyn5vAJD7YJGtIZUpvxuEV4+uViM56dk9Y7UcvDCY/Gpc42yFKv+x5kuNSnQlcYmg+rXyXb6jk5MLjORBlMrHTDx8SjhNHO6he7L5n5s0yG1Ae0AYPmmQWulQIxS5QI7NnHnnVwj9OTbQ6Cl4AWu5pCP/0o6JpikHNIJJubMRUMwT2txEcf9iS8n3WFFKP1zi95m2zC8skGV/2v2JWbZ6LH7rExES8t6Ks3dCl/atPhyjul2VUKwH5NdbuFzLLO+6Fmt/V1NAW0pjWN4/6yLYKkT0ZFK2jNG9eJb4fReCOMDOTq8F1f/oxLB0iLcCWPn+9CDToAr4d9SZihZ6XLMYz+vh7bDU0LgYvQR6qNWMRFq7mfI9OV/DnMT9gs8U9Z39ETD1yT9uNQ1e2LDtzXKdcqY1n5SN93FwtznKTEQSj0tI2321RiqEl5A55C67p4Q2l79eiH0NmRbbBwA5ZesecDcRIjXZyYF9Kd8OwR5Z7uL39egZUOF0EMqUYsHmg8kAATCo47w//ILRnMBIbBYPzrMSqfeblT+MGnatOJ82pwOQMMvlLc3H0FIXtSN8f4oub8l/2avvCdXLFLLMAWDO6Y5q/iBp1wdBoBqGRLT89XU8IbS9+ukYtsxFJkiMMkmDQrtttJidPdFXbHwUq+ccayYAA/KABpgQ4Vhyf3mHweJ3bId8wGIVblaV1u1AGDIS0oJuRrXfZY2nIRywKX2qLvGKvmv7RUaJZ43Dy5wTaUKfUjVHcqU82oQ5Z1bJRYUoemvjZKxX22khIruT4AR2LbNYkYrjf/PgtpoAnb7h5X3HArFzCcAAPw7xIiOJX5W/0w8E8hbBjCvBjAnSaZps+oN9vakOjI5xUe65yG+il1BLyML/zM7Wnse4HnMX27EMY9zGSgG62//d99Kolc5wpmpGNMfWdIXM4tcWBoGT7fkjwLwIM9mG9qYI20sgqpNka8o1vAzCXwJl16bVceH2cCcoruN0TqrZatqhqRgiKjXIohIWRDHrNf/GMPbEG8dmUvFcsrWC5viom6wIRGKpOMDBaLjtMl0u7IL7w7ESS1hMyKIXhgFegQ1YCN3QQ2QLB89zi+YJ30mvgzQb0jXlaxwAZw2VwWnSvGN0W6Cz8dK/Cu90BckhdJR0yntDdjwM6Xhv9+wtBCEct+S+LlqNkIBMVIOKFcWx1CymW/ueqs5LbShiKGBewsNpv2PJO3b/NG5n1UTmcjfWbf8lQxxPMoVmH526q8mk4IMFqEEOmyGE7PBR/Kr92njCREyD00jjlbm3ixeSs1ABJ+71/NVXe2MZLVxYyuj4crKIjY0LfjxWy/+POrLk2hF8/EjpZlbj5rCUiDpJnql7xXZpkrCH3KONrw4GUbKxhXbIYpM9KsGIqFKwfJ87y1XcVCu019OzPJWVflFzomRdPUyuy9pfFx9GSa9t2m4m3N7d45s0YmH3KlrJna5HPP7FgZNzoR9JJcPJbwuLRvqhTFqz0PvHVJZJ60cW8GRjwNjyAQDPBRERNGALWp/kjkZ6zFlTkiHaBpZtIYSeUGKuVpuQ8OOQjBIX99Gv54b1/JbC3cnpniRZBYjSuxrkGKfolwb8OzUXBBAI9+t7WMfPFAKv1eFqfnDwXhd8pqnDymwG7JG3Henp4Md6QLYgyKL+3OrOr9s6o/4P8bRNkGMphSzjwJqIzvQO/1S62eDsjBUwqL09ts9May5miZXcn2By3WIrPqP69XJUGMlABJIZS74gbAOL8PDRc/ugNxYedXsD5qV5sUh4JUtcOwhSj/dnaSLwSi0g84W3ZuWWI2Ddw0PTjkJkIN8tipNxY2ctChSpfsx6RanzVoL5kABv86NEHri8njdZ1yxay1VR+TqM/p0WuT/EyIHbHLdS/+u9Tz5D0TQDzIaoJ/y4+rQDp92sjQdEFdanpHzJghTptzhdqkgAdxFpugAAFxgAZ86tbMez4xjMKm6AUOe1c77LJKqJjabhz/MLQ5T0VhVOaI66HTL5f8hwbSJ8CDvuV3IQF+cmlHnKwswsTXUU8ukQFknaNC7Ff54ONCdeOg8GuZOjRtEQ0SiNGiURo0SiNE2gAcSvSr/5lN2fKtCWej0Tme3b5DJyUn0V2DUQ0+qAQ98tvpDTUm1xkticgP39e3X9xLyEEXg4n1CYg+jS0CpwPsYmas2LpMqKQERFvY+LSOPLNLawmaXlpyBSMOlc5DqsyrUeQgXQKusBHiYM891c0Nh+6QVfv5Q2XetaxwAZxZKsJS+PfLj5OAmVOIo4PjZ5tir8FluTy2+6ndHmKk2LjOlxmqIK4a/SqR3ehmJTtYSxEiajsGDurtk0TIR80mnqHVTRjC04oc1h3vPwd58y34piUWyWPoZFOEM+PoMdOkMnY/aiV2kPnJBd7+M1c3QoCDBM2Kbcc+cD82b9htlYHbclUdHOdE5rhhcjPTa83TUCIjjBG/VsohTJHEqlsWVEATlQu0rSzkl4WSxnTo5mq9UbVGkk5thbP7J0AACixMAN+RlmkFYgMF6Arj6K+hme5vZlskwk/DL3SmenPQfMniDewheHZX4VJNGUkv568WeSbEf0O+0ujGEQZ//655oS+Q1v5YpxZ/Kpfy4acTL5CERqxnPDxBt5l/Cbyhj9gABGAYuXamovr0aY9H7vyAs8TKBSo97GRhoinhVm2EuuQExKOeUqU3r+a+9P8Riswoi7/M1Y04VTFVoVFghApo1zFwggaO9MsNUjwJaSNhu2BKOOyf9PUZqOyMoBq7sy2APbZO+1/hzvQoR3N8yqITMHAobUugCTZScXDAM0W3BhgH8+G47mqzph3143eVojkcGWakHCIWKNYPpWReYh7F50Bsjd9OsgazjGi2UnFubtDlNqMssDbaZBbtNy45V0zXy55eXm7fxXoV4fBNExKTTrpsrPVlWorzBqsrk2ki02PE39zNc52FhKR+PCF+C1T6gu+cHkcljrAdpFIpFIpFIeAEeadrqaLGPKgkjLTrMc4tXRjYI9Qn2F8bB/wmcb/wGelgmUx5zrPV/+mDrC3Z+bBkwJFR6eHRQM08BzjWzDtkQkx6ZSfacoWnZXe1yxa7g+ovGJo4aOqtm9XtyDzPSsMVM10TfXC0Bsjd9OsgazjGqqTPFhQZPMfuBPEOd6PnW8B8pEjdH1xwyW6KZMZ3Ekb33iahJIwA5KF/Gh/ie/7fhr2TxRs8QlvtI2nqFiSXulFxbGFCb0Am9ksbSSPUCs09WJbxBNsoMhlikVy6keuXTwAR5p2uposY8qCSOVCDHjtWBgu0VMtGzBUyfTNKYsQZa/iHvHxEG8I5yDZTnXGsD+vYKWMIIysEP9rAayTTDBHqE+wvjYQGvCix8YKnRx/CCK0FtavuW0MiCSNArpvrVlhmuxLFo8zKyOdPGX/k1Fg1/x2DrnPFK+21Ku1g3ctzxRKG4f8hDtNSjnAF8X9W3YliKxpx+WdzmlyVn1/RauFQ6uZxtol4bAWF7OxBhW50tlzNLU8UbwiWYrbEjDdfhVMYWQBtKBCnvJTZImfwsopBf4JSfdW86/oEoroyMImO2obNZebwnNCBb4C0IRBeU6sqqTpFpk6wMVcFD1mrTBdYjKWlHpkjg6rDvEDO8BPYT16exUi+4n1MvSebPklTAKMpqu+bNwShjfSrVLBe5oLxhtjyIJCB7s4cHxzfBSSwc68UX2wENrp4qK5ECBmlmK4Ghgi/kqoioSW5cbcdkLelee/wXm/znZmZuvgYcRcx31rzXfA9hrGS2c1USVvXu/69OnoIxf1SmhmrTLA5zfc2Y5XpeCaEm2Lrm8R5m+H3L8umUbM5NgFzBYyR4whyRVvzZTwL1f99EiGgQj013iyumim1yp/IdQuqfl+8H1JtUUlWyuc5XhD52+wS8M1OgqXpoMnTIsnWpn8J9JCv9022zN7f6pFhgiqZfFV0cq1UpXfbqlMzJWhY8vV3NwrIQLesB9ubva7yLSscYrG1HZHr7OCXGRePMoN0LNt+9TtHAJ/6FyObB5lqRtBN/X0O30qKlq/+bFVdwNUzdP/sLCJvxB82hI9RMzTfAySvr6adEYAwUy1xJUxfvQasH1nZyUttW+PN21u2b6uk1+ogrEDsIEjkjD3pIKVCAruHVzADJ5DUZmCmd0Y2DyN+rXuf7h8KoOTgi6qae+5UzXMSH7rfGm2gDNr7qyWtg4cx7E4soiAFYIM/PBYss4c10ezZbS6rzxQvXzM9JVFs/uXysPi4x82bglDG+lWqWCa5Y1DHt6KyqMdblhdK7Kptx4195jQ/J9YQKjP4pUqtFtusQIWqOTjSwdDlChXP2XOpGZnpM3YuVpfG8fT7aYBVUfyvlZxABAJ3I+WTI0o1/fzlPQoFseEeXLW+BSzm95sNmRA1lGvUCry4X4nv291cM0qewJKvPhAw2b15wxojiG/CnSn+w3jlthXuxc0+Y76KUkDzvBGY+8u7vW5sWJyVYzB+3a6zUnQIUnO24/nrD34rw24UbMZqiKogdB65FnYaSwehps2ls1eYCNY0TBEnlG387CO0tZGG7i0KaioVOyfbiV15TMejStzOtCmNAaFDl+B3L7mtUuqL4O6Yaw6SZ3LZgtfy35IYYVB0aDBEy+KMCPsekpLD2mA8wXdE3g4/1qAAAAAAZ51s8FlZcL5/3mdi8XaR5pxDAmLE72EH6oYEJKWNTGKxflGc3f2sH2MkHIbt9EC4e+WgcPpv+03jij+qxMEP68QPOWGQuV7+thl52ET6A0/QO/+qopqqLrTdjDlcw7WgYaOrcXLyuMol2Vpx/O99i9fj+mDe+s/FNJ2f6tykhtqNLzsjQimBLOBELqhkB+MrHLoSUy1HN9YML2WwPilvRZsbpGZxfrCvkYwnXWcUbv8PeUsxOUAAKlEsfbOLsAX2W6aDHbf/XUyJldNT0xDh5XDiOGHqQQqPAQyGsWOHDbEndDOREVaG8sfUlI5bqItKwlLlaLuIdmFd3sGa5iWC0S9VRPuZC6q8pFCRLnyHxyITc+5QF8ZP77PvXWwHnKlNBVFp4K9lTk0erv7/ddPIm1T+u+nzy0cSDe/8vuiKMQd8h4xYQp+SOHChKrv5cFDkIfwsqdP/CaeeuA1Y8ubYN7aaa6IyP/gqSmI9X7vbkmzED/5rGdDgG6YqC3oraDZv7newCliaqBb9MSCLAhL0R214vV8SS7P0YQkppg0KbK/jvccd7nxa6RWaOJmSBmAEuMjIlWmkaSmhep435BH1OiWpqFnLk0tngyL8bccN7KdnN6GjLHDvKNITEIgtCYmf8O2DzDuU8AmJjpf45zH8ZeXqki8mxrHcwdc4/pHABgqj5Z0DkLXrHrix6Mdc0n6SD8z8fi3kGOeYvny824CeE7sGWxFJMHLMn6LnVSZ7YrO/Gx5+TuZy0ecX0Spau4HCK6fUzpC6wrtcqUKIy+rLLr6d9MeeymV69roWFvXmfGQjbXa5zN6ydvQ2EaicNfJ97pLUOanJ36I5uSPyRAs7GP8JYK0/4eYxLCMAVJNvq5HmscPosdv/bKkQw2+6CH84ys3kUvanbgb+U0xjAJdnVQrUTA4YlFgrSudVlPz/nPqnioPueIoM7Tawt73UsTNnX4KMva48vOmdA1u+vaI8xCnKtOaI11aCEBIMdSMJa8K+L7+dZJQ3MhNYgoLBt8EsX5XA2E9dcT8CCA5bnIiGSq0cKrLLMFMKov4PZC3tth4S6BHmgz+4AFNJxIWYJ0sxdjD4xbTxXCVJgSccJFtGZCSGEd1SwGcRwdmpRd3pII7oq1dw4duE7QPuravQh9DRVUyyN2LJFQvUW6T5r3XLxkUN+SaA/8uhSyAFFtvKEZAQu/q5udBFVbREq/jSnhJKS/jYZ4038k5nEhFbs/RMX3oxpdpgYTYc61NBhYtDCGLUzH8IDFpczd5CD8WLbt8/NEXRex0Li9SKvRy7iODyOi1TJGhI7z5HBVCim9L2Y96jBuE7Ye9wiYvpyCmtJRD4Pm1cWt2shZTzAgDmh+jsA5JI04TymbhXTD/3jJwuaA9Ty9Csm9wrCvXyt+mmZdeq8Leza+gdudCca2P9jp6ObmHivP6OF72Y1JDmKxQ+OL5wlA3V1rs4bBS5DCftJNU6FHKQiboxfNgrx45rDi7/Y3ip3f5zMUQgk8wGlTtDTN0cghnFu4TgOwb92uf6m/O3AKuNu3VHD6mAxt/SO/CIzEb4NrPapJsznZnk+lAhk8XpVEvMpHJreF41+sutOIqAhRmsi9L45NoYE9LzGYYAR7wI4OWV4mjWwaAAAAAAAA\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Общий обзор`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:06:16.174682Z",
     "start_time": "2023-02-28T09:06:16.155921Z"
    }
   },
   "source": [
    "База:\n",
    "* [torch.Tensor](https://pytorch.org/docs/stable/tensors.html)\n",
    "* [Tensor Views](https://pytorch.org/docs/stable/tensor_view.html)                  \n",
    "* [Tensor Attributes](https://pytorch.org/docs/stable/tensor_attributes.html)            \n",
    "* [torch](https://pytorch.org/docs/stable/torch.html) — основные операции над тензорами\n",
    "    * Tensors\n",
    "        * Creation Ops\n",
    "        * Indexing, Slicing, Joining, Mutating Ops\n",
    "    * Generators\n",
    "    * Random sampling\n",
    "        * In-place random sampling\n",
    "        * Quasi-random sampling\n",
    "    * Serialization\n",
    "    * Parallelism\n",
    "    * Locally disabling gradient computation\n",
    "    * Math operations\n",
    "        * Pointwise Ops\n",
    "        * Reduction Ops\n",
    "        * Comparison Ops\n",
    "        * Spectral Ops\n",
    "        * Other Operations\n",
    "        * BLAS and LAPACK Operations\n",
    "    * Utilities\n",
    "    * Operator Tags\n",
    "\n",
    "\n",
    "* [torch.nn](https://pytorch.org/docs/stable/nn.html) — блоки-классы для DL моделей\n",
    "    * Containers\n",
    "    * Convolution Layers\n",
    "    * Pooling layers\n",
    "    * Padding Layers\n",
    "    * Non-linear Activations (weighted sum, nonlinearity)\n",
    "    * Non-linear Activations (other)\n",
    "    * Normalization Layers\n",
    "    * Recurrent Layers\n",
    "    * Transformer Layers\n",
    "    * Linear Layers\n",
    "    * Dropout Layers\n",
    "    * Sparse Layers\n",
    "    * Distance Functions\n",
    "    * Loss Functions\n",
    "    * Vision Layers\n",
    "    * Shuffle Layers\n",
    "    * DataParallel Layers (multi-GPU, distributed)\n",
    "    * Utilities\n",
    "    * Quantized Functions\n",
    "    * Lazy Modules Initialization\n",
    "* [torch.nn.functional](https://pytorch.org/docs/stable/nn.functional.html) — блоки-функции для DL моделей в \n",
    "* [torch.linalg](https://pytorch.org/docs/stable/linalg.html) — аналог `numpy.linalg`\n",
    "* [torch.special](https://pytorch.org/docs/stable/special.html) — аналог `scipy.special` \n",
    "* [torch.nn.init](https://pytorch.org/docs/stable/nn.init.html) — методы для инициализации весов      \n",
    "* [torch.optim](https://pytorch.org/docs/stable/optim.html) — классы-оптимизаторы\n",
    "* [torch.random](https://pytorch.org/docs/stable/random.html) — случайное семплирование     \n",
    "* [torch.autograd](https://pytorch.org/docs/stable/autograd.html) — продвинутое дифференцирование\n",
    "* [torch.hub](https://pytorch.org/docs/stable/hub.html) — предобученные модели\n",
    "* [torch.fft](https://pytorch.org/docs/stable/fft.html) — аналог `scipy.fft` \n",
    "* [torch.utils.tensorboard](https://pytorch.org/docs/stable/tensorboard.html) — логгирование в tensorboard            \n",
    "\n",
    "\n",
    "Популярные модули:\n",
    "* [torch.amp](https://pytorch.org/docs/stable/amp.html) — mixed precision training                \n",
    "* [Complex Numbers](https://pytorch.org/docs/stable/complex_numbers.html) — комплексные числа\n",
    "* [Quantization](https://pytorch.org/docs/stable/quantization.html) — квантизация\n",
    "* [torch.distributions](https://pytorch.org/docs/stable/distributions.html) — классы для распределений с поддержкой дифференцирования (Reparameterization Trick/REINFORCE)\n",
    "\n",
    "\n",
    "Вспомодательные модули:\n",
    "* [torch.cuda](https://pytorch.org/docs/stable/cuda.html) — параметры и свойства вычисления на GPU\n",
    "* [torch.backends](https://pytorch.org/docs/stable/backends.html) — параметры устройства для вычисления \n",
    "* [torch.__config__](https://pytorch.org/docs/stable/config_mod.html) — текущая конфигурация\n",
    "\n",
    "\n",
    "Распределённые вычисления:\n",
    "* [torch.distributed](https://pytorch.org/docs/stable/distributed.html)                  \n",
    "* [torch.distributed.algorithms.join](https://pytorch.org/docs/stable/distributed.algorithms.join.html)  \n",
    "* [torch.distributed.elastic](https://pytorch.org/docs/stable/distributed.elastic.html)        \n",
    "* [torch.distributed.fsdp](https://pytorch.org/docs/stable/fsdp.html)                         \n",
    "* [torch.distributed.optim](https://pytorch.org/docs/stable/distributed.optim.html )           \n",
    "* [DDP Communication Hooks](https://pytorch.org/docs/stable/ddp_comm_hooks.html)               \n",
    "* [Pipeline Parallelism](https://pytorch.org/docs/stable/pipeline.html)  \n",
    "* [Distributed RPC Framework](https://pytorch.org/docs/stable/rpc.html)                          \n",
    "\n",
    "\n",
    "Для разработки расширений:\n",
    "* [torch.library](https://pytorch.org/docs/stable/library.html)                      \n",
    "* [torch.utils.cpp_extension](https://pytorch.org/docs/stable/cpp_extension.html)                \n",
    "\n",
    "\n",
    "Разное:\n",
    "* [torch.jit](https://pytorch.org/docs/stable/jit.html)                          \n",
    "* [torch.futures](https://pytorch.org/docs/stable/futures.html)                      \n",
    "* [torch.fx](https://pytorch.org/docs/stable/fx.html)                           \n",
    "* [torch.monitor](https://pytorch.org/docs/stable/monitor.html)                      \n",
    "* [torch.overrides](https://pytorch.org/docs/stable/torch.overrides.html)          \n",
    "* [torch.package](https://pytorch.org/docs/stable/package.html)                      \n",
    "* [torch.profiler](https://pytorch.org/docs/stable/profiler.html)                     \n",
    "* [torch.onnx](https://pytorch.org/docs/stable/onnx.html)                                 \n",
    "* [torch.masked](https://pytorch.org/docs/stable/masked.html)                       \n",
    "* [torch.nested](https://pytorch.org/docs/stable/nested.html)                       \n",
    "* [torch.sparse](https://pytorch.org/docs/stable/sparse.html)                       \n",
    "* [torch.Storage](https://pytorch.org/docs/stable/storage.html)                      \n",
    "* [torch.testing](https://pytorch.org/docs/stable/testing.html)                      \n",
    "* [torch.utils.benchmark](https://pytorch.org/docs/stable/benchmark_utils.html)              \n",
    "* [torch.utils.bottleneck](https://pytorch.org/docs/stable/bottleneck.html)                   \n",
    "* [torch.utils.checkpoint](https://pytorch.org/docs/stable/checkpoint.html)                   \n",
    "* [torch.utils.jit](https://pytorch.org/docs/stable/jit_utils.html)                    \n",
    "* [torch.utils.dlpack](https://pytorch.org/docs/stable/dlpack.html)                       \n",
    "* [torch.utils.mobile_optimizer](https://pytorch.org/docs/stable/mobile_optimizer.html)             \n",
    "* [torch.utils.model_zoo](https://pytorch.org/docs/stable/model_zoo.html)                    \n",
    "* [Type Info](https://pytorch.org/docs/stable/type_info.html)                    \n",
    "* [Named Tensors](https://pytorch.org/docs/stable/named_tensor.html)                 \n",
    "* [Named Tensors operator coverage](https://pytorch.org/docs/stable/name_inference.html)               "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Базовые операции над тензорами`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Основной объект в pytorch — тензор (`torch.Tensor`), который является близким аналогом массива из numpy (`np.ndarray`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:21:13.862256Z",
     "start_time": "2023-02-28T12:21:13.826787Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0.]]),\n",
       " tensor([[1, 1, 1, 1, 1, 1, 1],\n",
       "         [1, 1, 1, 1, 1, 1, 1],\n",
       "         [1, 1, 1, 1, 1, 1, 1],\n",
       "         [1, 1, 1, 1, 1, 1, 1],\n",
       "         [1, 1, 1, 1, 1, 1, 1]], dtype=torch.int32),\n",
       " tensor([[ 0.6421, -0.7510,  0.2902, -1.1373, -0.3381, -0.2419,  2.1136],\n",
       "         [-0.1471, -0.9218, -0.7532, -1.4782,  0.2996,  0.1003,  1.6461],\n",
       "         [ 1.2146, -0.2992,  0.3497,  0.5643,  0.9128, -0.3024,  0.5329],\n",
       "         [ 0.4227, -0.4736,  2.5723, -0.5186,  1.4417, -1.1723, -0.9542],\n",
       "         [ 0.5630, -1.0024, -0.9261, -0.8134,  0.2057,  0.1802, -2.2309]]))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    torch.zeros([5, 7]), \n",
    "    torch.ones([5, 7], dtype=torch.int32), \n",
    "    torch.randn([5, 7])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Набор операций тоже очень схож:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:21:20.161760Z",
     "start_time": "2023-02-28T12:21:20.132694Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[-0.9861, -1.1153, -0.1128, -0.9361, -1.9882],\n",
       "         [-2.2350, -0.1438, -0.0523, -1.2782,  1.2346],\n",
       "         [ 0.3968, -0.9047,  1.2549,  0.5906,  0.7667]]),\n",
       " tensor([[-0.4045, -1.6527,  0.4999, -1.5573, -3.4186],\n",
       "         [-4.0367, -0.7244, -0.1112, -1.1940,  0.1017],\n",
       "         [ 0.7854,  0.2155, -0.7974,  1.2074,  0.1830]]),\n",
       " tensor([[ 3.3820,  4.6045, -0.8179],\n",
       "         [-2.2267,  2.6070, -2.4310],\n",
       "         [ 0.0223, -1.0824, -3.5179]]),\n",
       " tensor(-5.5089),\n",
       " tensor(-0.0003))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A, B = torch.randn(3, 5), torch.randn(5, 3)\n",
    "A, A + B.T, A @ B, torch.sum(A), B.prod()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Преобразование из torch в numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(numpy.ndarray,\n",
       " array([1., 2., 3.], dtype=float32),\n",
       " torch.Tensor,\n",
       " tensor([1., 2., 3.]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w = torch.tensor([1, 2, 3.0])\n",
    "w_np = w.numpy()\n",
    "type(w_np), w_np, type(w), w"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Преобразование из numpy в torch через копирование данных:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(numpy.ndarray,\n",
       " array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10]),\n",
       " torch.Tensor,\n",
       " tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v_np = np.arange(10)\n",
    "v = torch.tensor(v_np)\n",
    "\n",
    "v_np += 1\n",
    "\n",
    "type(v_np), v_np, type(v), v"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Преобразование из numpy в torch с общим буфером:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(numpy.ndarray,\n",
       " array([[9., 9., 9.],\n",
       "        [9., 9., 9.]]),\n",
       " torch.Tensor,\n",
       " tensor([[9., 9., 9.],\n",
       "         [9., 9., 9.]], dtype=torch.float64))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u_np = np.full([2, 3], -1.0)\n",
    "u = torch.from_numpy(u_np)\n",
    "\n",
    "u_np += 10.0\n",
    "\n",
    "type(u_np), u_np, type(u), u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Однако есть небольшие различия:\n",
    "* `axis` -> `dim`: `np.sum(A, axis=0) -> torch.sum(A, dim=0)`\n",
    "* `.reshape` -> `.view/.reshape`\n",
    "* `.astype` -> `.to/.type`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Так же, есть возможность использовать inplace операции:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:23:19.024456Z",
     "start_time": "2023-02-28T12:23:18.997234Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.9861, -1.1153, -0.1128, -0.9361, -1.9882],\n",
       "        [-1.8382, -1.0485,  1.2025, -0.6876,  2.0014],\n",
       "        [ 0.3968, -0.9047,  1.2549,  0.5906,  0.7667]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[1] += A[2]\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:23:29.919725Z",
     "start_time": "2023-02-28T12:23:29.890635Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.zero_()\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:24:02.853009Z",
     "start_time": "2023-02-28T12:24:02.826216Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.exp(A)\n",
    "A.exp_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Полезные функции\n",
    "\n",
    "* Сконкатенировать набор тензоров вдоль заданной размерности `torch.cat`\n",
    "* Соединить тензоры одинаковой формы вдоль новой размерности `torch.stack`\n",
    "* Добавить/убрать новую \"единичную\" размерность в тензор `torch.unsqueeze/torch.squeeze`\n",
    "* Разбить тензор на заданное число блоков `torch.chunk`\n",
    "* Переставить между собой две размерности `torch.transpose`\n",
    "* Переставить местами все размерности `torch.permute`\n",
    "* Повторить тензор `torch.tile/torch.repeat`\n",
    "* Найти максимальный элемент (возвращает И положение, И значение) `torch.max`\n",
    "* Поэлементный максимум между двумя `тензорамиtorch.maximum`\n",
    "* \"Выпрямить\" тензор, объединив все размерности в одну `torch.ravel`\n",
    "* Объединить только заданные размерности `torch.flatten`\n",
    "* Вернуть k наибольших элементов `torch.topk`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Первое ключевое отличие — возможность перемещения на GPU`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:24:42.667044Z",
     "start_time": "2023-02-28T12:24:42.640050Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.float32, device(type='cpu'))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(3, 5)\n",
    "a.dtype, a.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим, что GPU доступна:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:24:47.248017Z",
     "start_time": "2023-02-28T12:24:47.221484Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:25:16.973109Z",
     "start_time": "2023-02-28T12:25:16.946647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = (\n",
    "    torch.device('cuda', 0) # эквивалентно: torch.device('cuda:0') / 'cuda' / 'cuda:0'\n",
    "    if torch.cuda.is_available() \n",
    "    else torch.device('cpu') \n",
    ")\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:25:49.614124Z",
     "start_time": "2023-02-28T12:25:49.587632Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1152,  0.1546, -0.9449, -1.6251, -0.8447],\n",
       "        [-0.5225, -1.1298, -1.2684,  1.1318,  0.0315],\n",
       "        [ 0.2922, -1.4419,  0.7172,  1.2302, -0.3422]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Операции между тензорами на разных устройствах не возможны:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:44:15.351137Z",
     "start_time": "2023-02-28T09:44:15.268716Z"
    }
   },
   "outputs": [],
   "source": [
    "a.cuda() + a"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обратите внимание, что при изменении device (c GPU на CPU и обратно происходит копирование):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_cpu = torch.rand(10)\n",
    "p_gpu = p_cpu.to(device)\n",
    "\n",
    "p_cpu -= 1\n",
    "p_cpu, p_gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Второе ключевое отличие — Autograd`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Все тензоры содержат атрибут `.grad`, который может хранить градиент по этому тензору."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:32:18.734804Z",
     "start_time": "2023-02-28T12:32:18.709812Z"
    }
   },
   "outputs": [],
   "source": [
    "a.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По умолчанию, тензоры, создаваемые в pytorch, не будут требовать, чтобы для них посчитали градиент. Для этого надо добавить дополнительный аргумент `requires_grad=True`, либо вызвав метод `.requires_grad_()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:32:37.982633Z",
     "start_time": "2023-02-28T12:32:37.955461Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 2., 3.], requires_grad=True)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([1., 2., 3.], requires_grad=True)\n",
    "y = torch.tensor([1., 2., 3.])\n",
    "y.requires_grad_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:50:22.820474Z",
     "start_time": "2023-02-28T09:50:22.793992Z"
    }
   },
   "source": [
    "Производя операции с переменными, по которым нужно считать градиенты, мы конструируем граф вычислений:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:33:38.274017Z",
     "start_time": "2023-02-28T12:33:38.247800Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = 3 * x**3 - y**2\n",
    "z.requires_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:50:37.275708Z",
     "start_time": "2023-02-28T09:50:37.248285Z"
    }
   },
   "source": [
    "В каждой переменной есть информация о том, как именно она была получена при проходе вперёд. Исходя из этой информации у тензоров в графе вычислений хранятся функции, которые должны быть вызваны на обратном проходе для расчёта градиента."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:34:15.874921Z",
     "start_time": "2023-02-28T12:34:15.848470Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<SubBackward0 at 0x106a83730>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Считать для каждого тензора якобиан целиком — сильно неоптимально. Вместо этого на всех промежуточных этапах autograd считает только произведения якобиан-вектор. В частности из-за этого, конечный тензор в графе всегда должен быть скаляром, что выполняется для всех функций потерь по определению."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:50:49.361799Z",
     "start_time": "2023-02-28T09:50:49.334010Z"
    }
   },
   "source": [
    "Для примера сделаем из тензора `z` скаляр, сложив все его элементы, и посчитаем градиенты с помощью функции .backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:36:35.186266Z",
     "start_time": "2023-02-28T12:36:35.134969Z"
    }
   },
   "outputs": [],
   "source": [
    "z.sum().backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:36:36.720780Z",
     "start_time": "2023-02-28T12:36:36.698582Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 9., 36., 81.]), tensor([-2., -4., -6.]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad, y.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:50:49.361799Z",
     "start_time": "2023-02-28T09:50:49.334010Z"
    }
   },
   "source": [
    "Сравним с посчитанным вручную градиентом:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:36:55.067720Z",
     "start_time": "2023-02-28T12:36:55.038990Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.allclose(x.grad, 9 * x**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Однако, для оптимизации вычислений, градиенты не вычисляются в явном виде для промежуточных вершин графа"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:37:59.693186Z",
     "start_time": "2023-02-28T12:37:59.666173Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/3c/vr2463p11lz5fr_80mrg72gh0000gq/T/ipykernel_47876/4110045842.py:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/core/TensorBody.h:485.)\n",
      "  z.grad\n"
     ]
    }
   ],
   "source": [
    "z.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Если такое всё же нужно, требуется указать это явно с помощью вызова `.retain_grad()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:38:19.420533Z",
     "start_time": "2023-02-28T12:38:19.390708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1.])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([1., 2., 3.], requires_grad=True)\n",
    "y = torch.tensor([1., 2., 3.])\n",
    "\n",
    "z = 3 * x**3 - y**2\n",
    "z.retain_grad()\n",
    "\n",
    "z.sum().backward()\n",
    "z.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Визуализация графа вычислений`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для примера визуализируем, как бы выглядел граф вычислений для линейной регресси.\n",
    "\n",
    "Заметим, что пакет визуализации `pytorchviz` предназначен в первую очередь для визуализации нейронных сетей, поэтому нам необходимо будет использовать класс `torch.nn.Parameter`, который является обёрткой над тензором и несколько расширяет возможности аргумента `requires_grad=True`\n",
    "\n",
    "Графы визуализируются библиотекой `graphviz`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T09:53:47.806077Z",
     "start_time": "2023-02-28T09:53:20.107297Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torchviz in /usr/local/lib/python3.10/site-packages (0.0.2)\n",
      "Requirement already satisfied: graphviz in /usr/local/lib/python3.10/site-packages (from torchviz) (0.20.1)\n",
      "Requirement already satisfied: torch in /usr/local/lib/python3.10/site-packages (from torchviz) (1.13.0)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/site-packages (from torch->torchviz) (4.4.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.0.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.10 -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "! pip3 install torchviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:39:02.053088Z",
     "start_time": "2023-02-28T12:39:02.027188Z"
    }
   },
   "outputs": [],
   "source": [
    "from torchviz import make_dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:42:47.149224Z",
     "start_time": "2023-02-28T12:42:46.981157Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 7.1.0 (20230121.1956)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"219pt\" height=\"445pt\"\n",
       " viewBox=\"0.00 0.00 219.00 445.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 441)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-441 215,-441 215,4 -4,4\"/>\n",
       "<!-- 4889039616 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>4889039616</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"black\" points=\"132.5,-30 78.5,-30 78.5,0 132.5,0 132.5,-30\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-18\" font-family=\"monospace\" font-size=\"10.00\">Loss</text>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-7\" font-family=\"monospace\" font-size=\"10.00\"> ()</text>\n",
       "</g>\n",
       "<!-- 4888947888 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>4888947888</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"153,-85 58,-85 58,-66 153,-66 153,-85\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-73\" font-family=\"monospace\" font-size=\"10.00\">MeanBackward0</text>\n",
       "</g>\n",
       "<!-- 4888947888&#45;&gt;4889039616 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>4888947888&#45;&gt;4889039616</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M105.5,-65.87C105.5,-59.4 105.5,-50.2 105.5,-41.45\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"109,-41.63 105.5,-31.63 102,-41.63 109,-41.63\"/>\n",
       "</g>\n",
       "<!-- 4888948464 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>4888948464</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-140 61,-140 61,-121 150,-121 150,-140\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-128\" font-family=\"monospace\" font-size=\"10.00\">PowBackward0</text>\n",
       "</g>\n",
       "<!-- 4888948464&#45;&gt;4888947888 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>4888948464&#45;&gt;4888947888</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M105.5,-120.75C105.5,-114.27 105.5,-105.16 105.5,-96.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"109,-96.96 105.5,-86.96 102,-96.96 109,-96.96\"/>\n",
       "</g>\n",
       "<!-- 4888948320 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>4888948320</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-195 61,-195 61,-176 150,-176 150,-195\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-183\" font-family=\"monospace\" font-size=\"10.00\">SubBackward0</text>\n",
       "</g>\n",
       "<!-- 4888948320&#45;&gt;4888948464 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>4888948320&#45;&gt;4888948464</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M105.5,-175.75C105.5,-169.27 105.5,-160.16 105.5,-151.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"109,-151.96 105.5,-141.96 102,-151.96 109,-151.96\"/>\n",
       "</g>\n",
       "<!-- 4888948128 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4888948128</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-250 61,-250 61,-231 150,-231 150,-250\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.5\" y=\"-238\" font-family=\"monospace\" font-size=\"10.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 4888948128&#45;&gt;4888948320 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4888948128&#45;&gt;4888948320</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M105.5,-230.75C105.5,-224.27 105.5,-215.16 105.5,-206.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"109,-206.96 105.5,-196.96 102,-206.96 109,-206.96\"/>\n",
       "</g>\n",
       "<!-- 4888948608 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>4888948608</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"92,-305 9,-305 9,-286 92,-286 92,-305\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-293\" font-family=\"monospace\" font-size=\"10.00\">MvBackward0</text>\n",
       "</g>\n",
       "<!-- 4888948608&#45;&gt;4888948128 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>4888948608&#45;&gt;4888948128</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M59.58,-285.75C67.34,-278.27 78.71,-267.32 88.2,-258.17\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"90.36,-260.95 95.13,-251.49 85.5,-255.91 90.36,-260.95\"/>\n",
       "</g>\n",
       "<!-- 4888948032 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>4888948032</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"101,-365.5 0,-365.5 0,-346.5 101,-346.5 101,-365.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-353.5\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n",
       "</g>\n",
       "<!-- 4888948032&#45;&gt;4888948608 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4888948032&#45;&gt;4888948608</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-346.37C50.5,-338.5 50.5,-326.6 50.5,-316.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-316.68 50.5,-306.68 47,-316.68 54,-316.68\"/>\n",
       "</g>\n",
       "<!-- 4889039136 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>4889039136</title>\n",
       "<polygon fill=\"lightblue\" stroke=\"black\" points=\"77.5,-437 23.5,-437 23.5,-407 77.5,-407 77.5,-437\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-425\" font-family=\"monospace\" font-size=\"10.00\">Weight</text>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-414\" font-family=\"monospace\" font-size=\"10.00\"> (10)</text>\n",
       "</g>\n",
       "<!-- 4889039136&#45;&gt;4888948032 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4889039136&#45;&gt;4888948032</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-406.8C50.5,-398.09 50.5,-386.81 50.5,-377.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-377.36 50.5,-367.36 47,-377.36 54,-377.36\"/>\n",
       "</g>\n",
       "<!-- 4888948560 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>4888948560</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"211,-305 110,-305 110,-286 211,-286 211,-305\"/>\n",
       "<text text-anchor=\"middle\" x=\"160.5\" y=\"-293\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n",
       "</g>\n",
       "<!-- 4888948560&#45;&gt;4888948128 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>4888948560&#45;&gt;4888948128</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M151.42,-285.75C143.66,-278.27 132.29,-267.32 122.8,-258.17\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"125.5,-255.91 115.87,-251.49 120.64,-260.95 125.5,-255.91\"/>\n",
       "</g>\n",
       "<!-- 4889039296 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>4889039296</title>\n",
       "<polygon fill=\"lightblue\" stroke=\"black\" points=\"187.5,-371 133.5,-371 133.5,-341 187.5,-341 187.5,-371\"/>\n",
       "<text text-anchor=\"middle\" x=\"160.5\" y=\"-359\" font-family=\"monospace\" font-size=\"10.00\">Bias</text>\n",
       "<text text-anchor=\"middle\" x=\"160.5\" y=\"-348\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n",
       "</g>\n",
       "<!-- 4889039296&#45;&gt;4888948560 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>4889039296&#45;&gt;4888948560</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M160.5,-340.54C160.5,-333.34 160.5,-324.53 160.5,-316.68\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"164,-316.69 160.5,-306.69 157,-316.69 164,-316.69\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x123676ce0>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.rand(5, 10)\n",
    "y = torch.rand(5)\n",
    "\n",
    "w = torch.nn.Parameter(torch.rand(10))\n",
    "b = torch.nn.Parameter(torch.rand(1))\n",
    "\n",
    "y_hat = X @ w + b\n",
    "\n",
    "loss = torch.mean((y - y_hat)**2)\n",
    "make_dot(loss, params={'Weight': w, 'Bias': b, 'Loss': loss})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Что можно сказать об этом графе вычислений?\n",
    "\n",
    "* В листьях графа мы не видим тензора `X`, так кка нам требуется расчёт градиента только по параметрам модели.\n",
    "* `MvBackward0` соответствует матрично-векторному умножению (отсюда и первые буквы `Mv`) c матрицей `X`.\n",
    "* `AddBackward0` соответствует добавлению смещения `b`, остальная часть графа — вычисление MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Аккумулирование градиентов`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Если не предпринимать никаких дополнительных действий, то множественный вызов `.backward()` будет не перезаписывать градиенты тензоров, а складывать их с уже существующим значением (сам граф вычислений каждый раз разрушается). Такое поведение может быть, например, нужно, чтобы посчитать градиент по батчу данных, который не влезает в память компьютера целиком, так как градиент модели аддитивен по входным данным."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:45:48.289960Z",
     "start_time": "2023-02-28T12:45:48.259785Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.6422, -1.6038,  1.4735],\n",
       "        [ 0.8832, -1.2830, -0.0582],\n",
       "        [-2.3583, -0.6569,  1.2666]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.randn(3, 3, requires_grad=True)\n",
    "y = torch.sum(x * x)\n",
    "y.backward()\n",
    "\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:45:57.689497Z",
     "start_time": "2023-02-28T12:45:57.660403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 3.6422,  0.3962,  3.4735],\n",
       "        [ 2.8832,  0.7170,  1.9418],\n",
       "        [-0.3583,  1.3431,  3.2666]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.sum(2 * x)\n",
    "z.backward()\n",
    "\n",
    "# Заметьте, что ко всем значениям прибавилось 2\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Также подобный подход используется в рекуррентных нейронных сетях, где к одному и тому же тензору весов нейросети происходит несколько обращений во время прямого прохода.\n",
    "\n",
    "Чтобы считать градиент с нуля, достаточно удалить тензор."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:46:58.330786Z",
     "start_time": "2023-02-28T12:46:58.304503Z"
    }
   },
   "outputs": [],
   "source": [
    "x.grad = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T10:04:16.122722Z",
     "start_time": "2023-02-28T10:04:16.088550Z"
    }
   },
   "source": [
    "#### `Inplace операции`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По умолчанию, вызов `y = 2 * x` создаст новый тензор, в который скопирует значения `x`, умноженные на `2`. И есть большое желание провести данную операцию на месте, то есть без аллокации памяти. В nupmy это не имело бы никаких дополнительных последствий, но в pytorch нам надо помнить о графе вычислений, который должен быть без петель, а также может использовать тензоры, рассчитанные при прямом вычислении. В некоторых случаях библиотека может выполнить код и не ругнуться, но описание ситуаций, когда такое сработает, а когда нет, очень сложно, и потому сами разработчики не рекомендуют использовать in-place операции там, где необходим расчёт градиента.\n",
    "\n",
    "In-place операции всегда имеют символ `_` на конце."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:49:24.782235Z",
     "start_time": "2023-02-28T12:49:24.642776Z"
    }
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 3]], which is output 0 of ExpBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar 03/Основы Pytorch.ipynb Cell 68\u001b[0m in \u001b[0;36m6\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar%2003/%D0%9E%D1%81%D0%BD%D0%BE%D0%B2%D1%8B%20Pytorch.ipynb#Y114sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m# inplace operation!\u001b[39;00m\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar%2003/%D0%9E%D1%81%D0%BD%D0%BE%D0%B2%D1%8B%20Pytorch.ipynb#Y114sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m y\u001b[39m.\u001b[39mexp_()  \n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar%2003/%D0%9E%D1%81%D0%BD%D0%BE%D0%B2%D1%8B%20Pytorch.ipynb#Y114sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m z\u001b[39m.\u001b[39;49msum()\u001b[39m.\u001b[39;49mbackward()\n",
      "File \u001b[0;32m/usr/local/lib/python3.10/site-packages/torch/_tensor.py:487\u001b[0m, in \u001b[0;36mTensor.backward\u001b[0;34m(self, gradient, retain_graph, create_graph, inputs)\u001b[0m\n\u001b[1;32m    477\u001b[0m \u001b[39mif\u001b[39;00m has_torch_function_unary(\u001b[39mself\u001b[39m):\n\u001b[1;32m    478\u001b[0m     \u001b[39mreturn\u001b[39;00m handle_torch_function(\n\u001b[1;32m    479\u001b[0m         Tensor\u001b[39m.\u001b[39mbackward,\n\u001b[1;32m    480\u001b[0m         (\u001b[39mself\u001b[39m,),\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    485\u001b[0m         inputs\u001b[39m=\u001b[39minputs,\n\u001b[1;32m    486\u001b[0m     )\n\u001b[0;32m--> 487\u001b[0m torch\u001b[39m.\u001b[39;49mautograd\u001b[39m.\u001b[39;49mbackward(\n\u001b[1;32m    488\u001b[0m     \u001b[39mself\u001b[39;49m, gradient, retain_graph, create_graph, inputs\u001b[39m=\u001b[39;49minputs\n\u001b[1;32m    489\u001b[0m )\n",
      "File \u001b[0;32m/usr/local/lib/python3.10/site-packages/torch/autograd/__init__.py:197\u001b[0m, in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001b[0m\n\u001b[1;32m    192\u001b[0m     retain_graph \u001b[39m=\u001b[39m create_graph\n\u001b[1;32m    194\u001b[0m \u001b[39m# The reason we repeat same the comment below is that\u001b[39;00m\n\u001b[1;32m    195\u001b[0m \u001b[39m# some Python versions print out the first line of a multi-line function\u001b[39;00m\n\u001b[1;32m    196\u001b[0m \u001b[39m# calls in the traceback and some print out the last line\u001b[39;00m\n\u001b[0;32m--> 197\u001b[0m Variable\u001b[39m.\u001b[39;49m_execution_engine\u001b[39m.\u001b[39;49mrun_backward(  \u001b[39m# Calls into the C++ engine to run the backward pass\u001b[39;49;00m\n\u001b[1;32m    198\u001b[0m     tensors, grad_tensors_, retain_graph, create_graph, inputs,\n\u001b[1;32m    199\u001b[0m     allow_unreachable\u001b[39m=\u001b[39;49m\u001b[39mTrue\u001b[39;49;00m, accumulate_grad\u001b[39m=\u001b[39;49m\u001b[39mTrue\u001b[39;49;00m)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 3]], which is output 0 of ExpBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)."
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, 3, requires_grad=True)\n",
    "y = 2 * x\n",
    "z = y ** 2\n",
    "# inplace operation!\n",
    "y.exp_()  \n",
    "z.sum().backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Другой распространённый в numpy сценарий - маскированное изменение значений тензора. Это тоже является in-place операцией. В pytorch для этого лучше использовать функцию `torch.where`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:49:57.458715Z",
     "start_time": "2023-02-28T12:49:57.429632Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4336, 0.4062, 0.5464],\n",
       "        [0.6937, 0.9116, 0.0409],\n",
       "        [0.7574, 0.9459, 0.3376]], requires_grad=True)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(3,3, requires_grad=True)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:49:58.744154Z",
     "start_time": "2023-02-28T12:49:58.685246Z"
    }
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "a leaf Variable that requires grad is being used in an in-place operation.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar 03/Основы Pytorch.ipynb Cell 71\u001b[0m in \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/nakhodnov/mmp_practicum_spring_2023/Seminars/Seminar%2003/%D0%9E%D1%81%D0%BD%D0%BE%D0%B2%D1%8B%20Pytorch.ipynb#Y120sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m x[x \u001b[39m>\u001b[39;49m \u001b[39m0.5\u001b[39;49m] \u001b[39m=\u001b[39m \u001b[39m0\u001b[39m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: a leaf Variable that requires grad is being used in an in-place operation."
     ]
    }
   ],
   "source": [
    "x[x > 0.5] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:50:26.664407Z",
     "start_time": "2023-02-28T12:50:26.634154Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4336, 0.4062, 0.0000],\n",
       "        [0.0000, 0.0000, 0.0409],\n",
       "        [0.0000, 0.0000, 0.3376]], grad_fn=<WhereBackward0>)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.where(x > 0.5, 0, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Копирование тензоров`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В numpy существует интуитивно понятная функция `.copy()`, но в pytorch функции с таким названием нет! Это связано с тем, что тензоры в pytorch привязаны к графу вычислений, который надо также учитывать при копировании."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.clone()` копирует тензор и сохраняет его привязку к текущему дереву вычислений:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:55:41.493960Z",
     "start_time": "2023-02-28T12:55:41.465480Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(3, requires_grad=True)\n",
    "y = x.clone()\n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.detach()` исходя из своего названия, копирует лишь значения элементов тензора, отвязывая его от текущего графа вычислений:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T12:55:55.365469Z",
     "start_time": "2023-02-28T12:55:55.336445Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.detach()\n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Продемонстрируем на графе вычислений эти две операции:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T13:03:36.118640Z",
     "start_time": "2023-02-28T13:03:35.939848Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 7.1.0 (20230121.1956)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"149pt\" height=\"380pt\"\n",
       " viewBox=\"0.00 0.00 149.00 380.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 376)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-376 145,-376 145,4 -4,4\"/>\n",
       "<!-- 4894714480 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>4894714480</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"black\" points=\"77.5,-31 23.5,-31 23.5,0 77.5,0 77.5,-31\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-7\" font-family=\"monospace\" font-size=\"10.00\"> ()</text>\n",
       "</g>\n",
       "<!-- 4892350480 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>4892350480</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"95,-86 6,-86 6,-67 95,-67 95,-86\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-74\" font-family=\"monospace\" font-size=\"10.00\">SumBackward0</text>\n",
       "</g>\n",
       "<!-- 4892350480&#45;&gt;4894714480 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>4892350480&#45;&gt;4894714480</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-66.54C50.5,-60.07 50.5,-50.98 50.5,-42.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-42.58 50.5,-32.58 47,-42.58 54,-42.58\"/>\n",
       "</g>\n",
       "<!-- 4892350384 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>4892350384</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"95,-141 6,-141 6,-122 95,-122 95,-141\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-129\" font-family=\"monospace\" font-size=\"10.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 4892350384&#45;&gt;4892350480 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>4892350384&#45;&gt;4892350480</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-121.75C50.5,-115.27 50.5,-106.16 50.5,-97.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-97.96 50.5,-87.96 47,-97.96 54,-97.96\"/>\n",
       "</g>\n",
       "<!-- 4892351008 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>4892351008</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"95,-196 6,-196 6,-177 95,-177 95,-196\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-184\" font-family=\"monospace\" font-size=\"10.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 4892351008&#45;&gt;4892350384 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>4892351008&#45;&gt;4892350384</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-176.75C50.5,-170.27 50.5,-161.16 50.5,-152.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-152.96 50.5,-142.96 47,-152.96 54,-152.96\"/>\n",
       "</g>\n",
       "<!-- 4892351488 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4892351488</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"101,-306 0,-306 0,-287 101,-287 101,-306\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-294\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n",
       "</g>\n",
       "<!-- 4892351488&#45;&gt;4892351008 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4892351488&#45;&gt;4892351008</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M44.61,-286.6C37.29,-274.45 26.24,-251.89 30.5,-232 32.38,-223.21 36.15,-214.06 39.9,-206.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.92,-208.14 44.48,-197.66 36.72,-204.88 42.92,-208.14\"/>\n",
       "</g>\n",
       "<!-- 4892349040 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>4892349040</title>\n",
       "<polygon fill=\"lightgrey\" stroke=\"black\" points=\"141,-251 40,-251 40,-232 141,-232 141,-251\"/>\n",
       "<text text-anchor=\"middle\" x=\"90.5\" y=\"-239\" font-family=\"monospace\" font-size=\"10.00\">CloneBackward0</text>\n",
       "</g>\n",
       "<!-- 4892351488&#45;&gt;4892349040 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4892351488&#45;&gt;4892349040</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M57.11,-286.75C62.46,-279.66 70.18,-269.42 76.85,-260.59\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"79.61,-262.74 82.84,-252.65 74.02,-258.52 79.61,-262.74\"/>\n",
       "</g>\n",
       "<!-- 4894376320 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>4894376320</title>\n",
       "<polygon fill=\"lightblue\" stroke=\"black\" points=\"77.5,-372 23.5,-372 23.5,-342 77.5,-342 77.5,-372\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-360\" font-family=\"monospace\" font-size=\"10.00\">X</text>\n",
       "<text text-anchor=\"middle\" x=\"50.5\" y=\"-349\" font-family=\"monospace\" font-size=\"10.00\"> (3)</text>\n",
       "</g>\n",
       "<!-- 4894376320&#45;&gt;4892351488 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>4894376320&#45;&gt;4892351488</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M50.5,-341.54C50.5,-334.34 50.5,-325.53 50.5,-317.68\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"54,-317.69 50.5,-307.69 47,-317.69 54,-317.69\"/>\n",
       "</g>\n",
       "<!-- 4892349040&#45;&gt;4892351008 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4892349040&#45;&gt;4892351008</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M83.89,-231.75C78.54,-224.66 70.82,-214.42 64.15,-205.59\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"66.98,-203.52 58.16,-197.65 61.39,-207.74 66.98,-203.52\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1239b50f0>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.nn.Parameter(torch.rand(3))\n",
    "y = torch.nn.Parameter(torch.rand(3))\n",
    "\n",
    "z = torch.sum(x + x.clone() + y.detach())\n",
    "\n",
    "make_dot(z, params={'X': x, 'Y': y})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Что можно понять из данного графа?\n",
    "\n",
    "* Видна операция `CloneBackward0`, которая клонирует тензор `x`. Благодаря ей в сумме участвует как исходный тензор, так и его экспонированная версия.\n",
    "* Во второй сумме мы не видим параметра `y`, потому что он входит в граф вычислений только через `.detach()`, что убирает проход градиентов."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Продвинутое дифференцирование`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В модуле `autorgrad` есть набор функций для более сложных операций по подсчёту градиентов: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T13:08:50.014567Z",
     "start_time": "2023-02-28T13:08:49.984942Z"
    }
   },
   "outputs": [],
   "source": [
    "X = torch.randn(5, 10)\n",
    "w = torch.randn(10, requires_grad=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Например, подсчёт градиентов скалярной функции:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((tensor([-0.3403,  0.0802,  0.4556,  0.1444, -0.0829, -0.4680, -0.4309, -0.6910,\n",
       "          -0.0474, -0.1296]),),\n",
       " tensor([-0.3403,  0.0802,  0.4556,  0.1444, -0.0829, -0.4680, -0.4309, -0.6910,\n",
       "         -0.0474, -0.1296]))"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.autograd.grad(torch.mean(X @ w), [w]), X.mean(dim=0)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вычисление произведения якобиана на вектор $vJ$, где $J = \\nabla_{w} L(w) \\in \\mathbb{R}^{n \\times d}, w \\in \\mathbb{R}^{d}, L(w)\\in \\mathbb{R}^{n}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((tensor([ 2.8464,  0.2514, -1.2352,  3.5278, -0.8807, -1.1547,  1.6524,  2.5139,\n",
       "          -0.7124, -0.1250]),),\n",
       " tensor([ 2.8464,  0.2514, -1.2352,  3.5278, -0.8807, -1.1547,  1.6524,  2.5139,\n",
       "         -0.7124, -0.1250]))"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v = torch.randn(5)\n",
    "torch.autograd.grad(X @ w, [w], grad_outputs=v), v @ X"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подсчёт градиентов по элементам в батче:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((tensor([[-1.2696,  0.8204,  0.4455, -1.5470, -1.1027, -1.9419, -0.4124, -1.1925,\n",
       "            0.9744,  0.0025],\n",
       "          [-0.1860,  0.2552,  0.0605,  1.3691, -0.4573, -0.7374, -0.0127, -0.0958,\n",
       "            0.0967, -0.6392],\n",
       "          [-0.1859,  0.5224, -0.4162,  1.1868,  0.2611,  0.5703, -0.3630,  0.1817,\n",
       "           -1.1567,  0.1271],\n",
       "          [-2.1699, -0.3734,  0.5346, -1.7271,  1.0141,  1.8544, -1.0369, -1.3105,\n",
       "            0.3716, -0.3080],\n",
       "          [ 2.1097, -0.8237,  1.6536,  1.4400, -0.1296, -2.0855, -0.3292, -1.0377,\n",
       "           -0.5233,  0.1694]]),),\n",
       " tensor([[-1.2696,  0.8204,  0.4455, -1.5470, -1.1027, -1.9419, -0.4124, -1.1925,\n",
       "           0.9744,  0.0025],\n",
       "         [-0.1860,  0.2552,  0.0605,  1.3691, -0.4573, -0.7374, -0.0127, -0.0958,\n",
       "           0.0967, -0.6392],\n",
       "         [-0.1859,  0.5224, -0.4162,  1.1868,  0.2611,  0.5703, -0.3630,  0.1817,\n",
       "          -1.1567,  0.1271],\n",
       "         [-2.1699, -0.3734,  0.5346, -1.7271,  1.0141,  1.8544, -1.0369, -1.3105,\n",
       "           0.3716, -0.3080],\n",
       "         [ 2.1097, -0.8237,  1.6536,  1.4400, -0.1296, -2.0855, -0.3292, -1.0377,\n",
       "          -0.5233,  0.1694]]))"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Экспериментальная фича\n",
    "torch.autograd.grad(X @ w, [w], grad_outputs=(torch.eye(5), ), is_grads_batched=True), X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[-1.2696,  0.8204,  0.4455, -1.5470, -1.1027, -1.9419, -0.4124, -1.1925,\n",
       "           0.9744,  0.0025],\n",
       "         [-0.1860,  0.2552,  0.0605,  1.3691, -0.4573, -0.7374, -0.0127, -0.0958,\n",
       "           0.0967, -0.6392],\n",
       "         [-0.1859,  0.5224, -0.4162,  1.1868,  0.2611,  0.5703, -0.3630,  0.1817,\n",
       "          -1.1567,  0.1271],\n",
       "         [-2.1699, -0.3734,  0.5346, -1.7271,  1.0141,  1.8544, -1.0369, -1.3105,\n",
       "           0.3716, -0.3080],\n",
       "         [ 2.1097, -0.8237,  1.6536,  1.4400, -0.1296, -2.0855, -0.3292, -1.0377,\n",
       "          -0.5233,  0.1694]]),\n",
       " tensor([[-1.2696,  0.8204,  0.4455, -1.5470, -1.1027, -1.9419, -0.4124, -1.1925,\n",
       "           0.9744,  0.0025],\n",
       "         [-0.1860,  0.2552,  0.0605,  1.3691, -0.4573, -0.7374, -0.0127, -0.0958,\n",
       "           0.0967, -0.6392],\n",
       "         [-0.1859,  0.5224, -0.4162,  1.1868,  0.2611,  0.5703, -0.3630,  0.1817,\n",
       "          -1.1567,  0.1271],\n",
       "         [-2.1699, -0.3734,  0.5346, -1.7271,  1.0141,  1.8544, -1.0369, -1.3105,\n",
       "           0.3716, -0.3080],\n",
       "         [ 2.1097, -0.8237,  1.6536,  1.4400, -0.1296, -2.0855, -0.3292, -1.0377,\n",
       "          -0.5233,  0.1694]]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Медленная альтернатива:\n",
    "\n",
    "y = X @ w\n",
    "torch.stack(([torch.autograd.grad(y[idx], w, retain_graph=True)[0] for idx in range(y.shape[0])])), X"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обратите внимание, что в предыдущем примере проход через один и тот же граф вычислений делается несколько раз. По умолчанию, после первого прохода граф уничтожается. Чтобы этого избежать используйте `retain_graph=True`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подсчёт вторых производных:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(6.5043e-06, grad_fn=<LinalgVectorNormBackward0>)\n",
      "tensor(0.)\n",
      "tensor(0.)\n"
     ]
    }
   ],
   "source": [
    "S = w @ X.T @ X @ w\n",
    "\n",
    "# Подсчёт в два шага. Для создания графа вычислений вторых производных используется параметр create_graph=True:\n",
    "\n",
    "[grad] = torch.autograd.grad(S, w, create_graph=True)\n",
    "print(torch.linalg.norm(grad - 2 * X.T @ X @ w))\n",
    "\n",
    "[hessian] = torch.autograd.grad(grad, w, grad_outputs=(torch.eye(w.shape[0]), ), is_grads_batched=True)\n",
    "print(torch.linalg.norm(hessian - 2 * X.T @ X))\n",
    "\n",
    "# Подсчёт в функциональном виде:\n",
    "\n",
    "hessian_func = torch.autograd.functional.hessian(\n",
    "    lambda w: w @ X.T @ X @ w, w\n",
    ")\n",
    "print(torch.linalg.norm(hessian - hessian_func))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Менеджеры контекста`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Поведением градиента сразу группы тензоров можно управлять с помощью специальных функций, которые вызываются через стандартную семантику питона `with foo():`, где `foo` — одна из трёх функций ниже:\n",
    "\n",
    "**Default Mode**\n",
    "\n",
    "Стандартный режим работы pytorch, в котором управление градиентами происходит через requires_grad. Явно его нужно вызывать только внутри других контекстных менеджеров, чтобы временно снова активировать расчёт градиентов (что случается крайне редко) вызовом `torch.enable_grad()`.\n",
    "\n",
    "**No grad mode**\n",
    "\n",
    "Данный режим используется когда блока кода нет необходимости вычислять градиенты, что занимает как вычислительные ресурсы, так и дополнительную память. Реализуется вызовом `torch.no_grad()`.\n",
    "\n",
    "**Inference mode**\n",
    "\n",
    "Аналогично No grad mode отключает расчёт градиентов, но кроме того проводит дополнительные оптимизации, что делает вычисления внутри блока кода ещё быстрее. Однако, тензоры, созданные в таком блоке будет невозможно использовать совместно с тензорами, для которых расчёт градиента необходим. Реализуется вызовом `torch.inference_mode()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T13:12:02.050827Z",
     "start_time": "2023-02-28T13:12:02.022463Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(3, requires_grad=True)\n",
    "\n",
    "with torch.no_grad():\n",
    "    y = x + x\n",
    "    \n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T13:12:09.802900Z",
     "start_time": "2023-02-28T13:12:09.774068Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def foo(x):\n",
    "    return x + 2\n",
    "\n",
    "y = foo(x)\n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### `.item`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Часто при подсчёте метрик возникают тензоры из одного элемента/скаляры, которые могут находится на GPU или быть частью графа вычислений.\n",
    "\n",
    "Сохранение таких тензоров в массив приведёт к утечке памяти. Чтобы этого избежать существует возможность трансформирования таких тензоров в Python скаляр:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-28T10:59:22.222287Z",
     "start_time": "2023-02-28T10:59:22.193552Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([-0.2333], grad_fn=<MeanBackward1>), -0.23329778015613556)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.randn(10, requires_grad=True)\n",
    "loss = torch.mean(x, dim=0, keepdim=True)\n",
    "loss, loss.item()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}