CS224W_Colab_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XuXWJLEm2UWS"
   },
   "source": [
    "# **CS224W - Colab 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gzsP50bF6Gb"
   },
   "source": [
    "In this Colab, we will construct our own graph neural network by using PyTorch Geometric (PyG) and apply the model on two of Open Graph Benchmark (OGB) datasets. Those two datasets are used to benchmark the model performance on two different graph-related tasks. One is node property prediction, predicting properties of single nodes. Another one is graph property prediction, predicting the entire graphs or subgraphs.\n",
    "\n",
    "At first, we will learn how PyTorch Geometric stores the graphs in PyTorch tensor.\n",
    "\n",
    "We will then load and take a quick look on one of the Open Graph Benchmark (OGB) datasets by using the `ogb` package. OGB is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. The `ogb` package not only provides the data loader of the dataset but also the evaluator.\n",
    "\n",
    "At last, we will build our own graph neural networks by using PyTorch Geometric. And then apply and evaluate the models on node property prediction and grpah property prediction tasks.\n",
    "\n",
    "**Note**: Make sure to **sequentially run all the cells in each section**, so that the intermediate variables / packages will carry over to the next cell\n",
    "\n",
    "Have fun on Colab 2 :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nwwq0nSdmsOL"
   },
   "source": [
    "# 1 PyTorch Geometric (Datasets and Data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sf7vUmdNKCjA"
   },
   "source": [
    "PyTorch Geometric generally has two classes for storing or transforming the graphs into tensor format. One is the `torch_geometric.datasets`, which contains a variety of common graph datasets. Another one is `torch_geometric.data` that provides the data handling of graphs in PyTorch tensors.\n",
    "\n",
    "In this section, we will learn how to use the `torch_geometric.datasets` and `torch_geometric.data`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ic-o1P3r6hr2"
   },
   "source": [
    "## PyG Datasets\n",
    "\n",
    "The `torch_geometric.datasets` has many common graph datasets. Here we will explore the usage by using one example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "zT5qca3x6XpG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES(600)\n"
     ]
    }
   ],
   "source": [
    "from torch_geometric.datasets import TUDataset\n",
    "\n",
    "root = './enzymes'\n",
    "name = 'ENZYMES'\n",
    "\n",
    "# The ENZYMES dataset\n",
    "pyg_dataset= TUDataset('./enzymes', 'ENZYMES')\n",
    "\n",
    "# You can find that there are 600 graphs in this dataset\n",
    "print(pyg_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch_geometric.datasets.tu_dataset.TUDataset"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#WPRW\n",
    "type(pyg_dataset)\n",
    "#Initializing a dataset is straightforward.\n",
    "#An initialization of a dataset will automatically download its raw files and process them to the previously described Data format.\n",
    "#（来源：https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data(edge_index=[2, 168], x=[37, 3], y=[1])\n",
      "37\n",
      "tensor([[ 0,  0,  0,  1,  1,  1,  1,  1,  2,  2,  2,  2,  2,  3,  3,  3,  3,  3,\n",
      "          3,  4,  4,  4,  4,  5,  5,  5,  5,  5,  6,  6,  6,  6,  7,  7,  7,  7,\n",
      "          7,  8,  8,  8,  9,  9,  9,  9,  9, 10, 10, 10, 10, 11, 11, 11, 11, 12,\n",
      "         12, 12, 12, 12, 13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 15, 16, 16,\n",
      "         16, 16, 17, 17, 17, 17, 18, 18, 18, 19, 19, 19, 20, 20, 20, 20, 20, 20,\n",
      "         21, 21, 21, 21, 21, 22, 22, 22, 22, 23, 23, 23, 23, 24, 24, 24, 24, 25,\n",
      "         25, 25, 25, 25, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 28, 28, 28, 28,\n",
      "         28, 28, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 31, 31, 31, 32,\n",
      "         32, 32, 32, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 35, 35, 35,\n",
      "         35, 35, 36, 36, 36, 36],\n",
      "        [ 1,  2,  3,  0,  2,  3, 24, 27,  0,  1,  3, 27, 28,  0,  1,  2,  4,  5,\n",
      "         28,  3,  5,  6, 29,  3,  4,  6,  7, 29,  4,  5,  7,  8,  5,  6,  8,  9,\n",
      "         10,  6,  7,  9,  7,  8, 10, 11, 12,  7,  9, 11, 12,  9, 10, 12, 26,  9,\n",
      "         10, 11, 25, 26, 14, 15, 16, 25, 13, 15, 16, 25, 13, 14, 16, 17, 13, 14,\n",
      "         15, 17, 15, 16, 18, 19, 17, 19, 20, 17, 18, 20, 18, 19, 21, 22, 23, 30,\n",
      "         20, 22, 23, 30, 35, 20, 21, 23, 35, 20, 21, 22, 33,  1, 27, 28, 29, 12,\n",
      "         13, 14, 26, 29, 11, 12, 25, 28, 29,  1,  2, 24, 28, 29,  2,  3, 24, 26,\n",
      "         27, 29,  4,  5, 24, 25, 26, 27, 28, 20, 21, 33, 34, 35, 32, 34, 36, 31,\n",
      "         33, 34, 36, 23, 30, 32, 34, 35, 36, 30, 31, 32, 33, 35, 36, 21, 22, 30,\n",
      "         33, 34, 31, 32, 33, 34]])\n",
      "tensor([[1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [1., 0., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 1., 0.]])\n",
      "tensor([5])\n"
     ]
    }
   ],
   "source": [
    "#PRW\n",
    "print(pyg_dataset[0])\n",
    "print(pyg_dataset[0].num_nodes)\n",
    "print(pyg_dataset[0].edge_index)\n",
    "print(pyg_dataset[0].x)\n",
    "print(pyg_dataset[0].y)\n",
    "#特征在点上嘛"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NLm5vVYMAP2x"
   },
   "source": [
    "## Question 1: What is the number of classes and number of features in the ENZYMES dataset? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "8iF_Kyqr_JbY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES dataset has 6 classes\n",
      "ENZYMES dataset has 3 features\n"
     ]
    }
   ],
   "source": [
    "#代码参考：https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html#common-benchmark-datasets\n",
    "def get_num_classes(pyg_dataset):\n",
    "  # TODO: Implement this function that takes a PyG dataset object\n",
    "  # and return the number of classes for that dataset.\n",
    "\n",
    "  num_classes = 0\n",
    "\n",
    "  ############# Your code here ############\n",
    "  ## (~1 line of code)\n",
    "  ## Note\n",
    "  ## 1. Colab autocomplete functionality might be useful.\n",
    "  num_classes=pyg_dataset.num_classes\n",
    "  #########################################\n",
    "\n",
    "  return num_classes\n",
    "\n",
    "def get_num_features(pyg_dataset):\n",
    "  # TODO: Implement this function that takes a PyG dataset object\n",
    "  # and return the number of features for that dataset.\n",
    "\n",
    "  num_features = 0\n",
    "\n",
    "  ############# Your code here ############\n",
    "  ## (~1 line of code)\n",
    "  ## Note\n",
    "  ## 1. Colab autocomplete functionality might be useful.\n",
    "  num_features=pyg_dataset.num_node_features\n",
    "  #https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Dataset.num_node_features\n",
    "  #Returns the number of features per node in the dataset.\n",
    "  #########################################\n",
    "\n",
    "  return num_features\n",
    "\n",
    "# You may find that some information need to be stored in the dataset level,\n",
    "# specifically if there are multiple graphs in the dataset\n",
    "\n",
    "num_classes = get_num_classes(pyg_dataset)\n",
    "num_features = get_num_features(pyg_dataset)\n",
    "print(\"{} dataset has {} classes\".format(name, num_classes))\n",
    "print(\"{} dataset has {} features\".format(name, num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rwKbzhHUAckZ"
   },
   "source": [
    "## PyG Data\n",
    "\n",
    "Each PyG dataset usually stores a list of `torch_geometric.data.Data` objects. Each `torch_geometric.data.Data` object usually represents a graph. You can easily get the `Data` object by indexing on the dataset.\n",
    "\n",
    "For more information such as what will be stored in `Data` object, please refer to the [documentation](https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7sCV3xJWCddX"
   },
   "source": [
    "## Question 2: What is the label of the graph (index 100 in the ENZYMES dataset)? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "LIis9oTZAfs3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data(edge_index=[2, 168], x=[37, 3], y=[1])\n",
      "Graph with index 100 has label 4\n"
     ]
    }
   ],
   "source": [
    "def get_graph_class(pyg_dataset, idx):\n",
    "  # TODO: Implement this function that takes a PyG dataset object,\n",
    "  # the index of the graph in dataset, and returns the class/label \n",
    "  # of the graph (in integer).\n",
    "\n",
    "  label = -1\n",
    "\n",
    "  ############# Your code here ############\n",
    "  ## (~1 line of code)\n",
    "  the_graph=pyg_dataset[idx]\n",
    "  label=the_graph.y.item()\n",
    "  #########################################\n",
    "\n",
    "  return label\n",
    "\n",
    "# Here pyg_dataset is a dataset for graph classification\n",
    "graph_0 = pyg_dataset[0]\n",
    "print(graph_0)\n",
    "idx = 100\n",
    "label = get_graph_class(pyg_dataset, idx)\n",
    "print('Graph with index {} has label {}'.format(idx, label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fKhcVeAhCwoY"
   },
   "source": [
    "## Question 3: What is the number of edges for the graph (index 200 in the ENZYMES dataset)? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "f5m2DOfhBtWv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph with index 200 has 53 edges\n"
     ]
    }
   ],
   "source": [
    "def get_graph_num_edges(pyg_dataset, idx):\n",
    "  # TODO: Implement this function that takes a PyG dataset object,\n",
    "  # the index of the graph in dataset, and returns the number of \n",
    "  # edges in the graph (in integer). You should not count an edge \n",
    "  # twice if the graph is undirected. For example, in an undirected \n",
    "  # graph G, if two nodes v and u are connected by an edge, this edge\n",
    "  # should only be counted once.\n",
    "\n",
    "  num_edges = 0\n",
    "\n",
    "  ############# Your code here ############\n",
    "  ## Note:\n",
    "  ## 1. You can't return the data.num_edges directly\n",
    "  ## 2. We assume the graph is undirected\n",
    "  ## (~4 lines of code)\n",
    "  the_graph=pyg_dataset[idx]\n",
    "  num_edges=the_graph.edge_index.size()[1]/2  #edge_index长度除以2（无向）\n",
    "  num_edges=int(num_edges)\n",
    "  #########################################\n",
    "\n",
    "  return num_edges\n",
    "\n",
    "idx = 200\n",
    "num_edges = get_graph_num_edges(pyg_dataset, idx)\n",
    "print('Graph with index {} has {} edges'.format(idx, num_edges))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AXa7yIG4E0Fp"
   },
   "source": [
    "# 2 Open Graph Benchmark (OGB)\n",
    "\n",
    "The Open Graph Benchmark (OGB) is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. Its datasets are automatically downloaded, processed, and split using the OGB Data Loader. The model performance can also be evaluated by using the OGB Evaluator in a unified manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HnazPGGAJAZN"
   },
   "source": [
    "## Dataset and Data\n",
    "\n",
    "OGB also supports the PyG dataset and data. Here we take a look on the `ogbn-arxiv` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.3.0\n"
     ]
    }
   ],
   "source": [
    "#PRW\n",
    "import ogb\n",
    "print(ogb.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Gpc6bTm3GF02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ogbn-arxiv dataset has 1 graph\n",
      "Data(adj_t=[169343, 169343, nnz=1166243], node_year=[169343, 1], x=[169343, 128], y=[169343, 1])\n"
     ]
    }
   ],
   "source": [
    "#https://github.com/snap-stanford/ogb/tree/master/examples/nodeproppred/arxiv\n",
    "#https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/arxiv/gnn.py（如下一系列代码应该都是参照这个文件写的）\n",
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset\n",
    "#注意这里它会提醒我的ogb版本过时，但是虽然过时却也能用。暂时还没发现什么问题，那就先当它没有问题好了\n",
    "\n",
    "dataset_name = 'ogbn-arxiv'\n",
    "# Load the dataset and transform it to sparse tensor\n",
    "dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                 transform=T.ToSparseTensor())\n",
    "print('The {} dataset has {} graph'.format(dataset_name, len(dataset)))\n",
    "\n",
    "# Extract the graph\n",
    "data = dataset[0]\n",
    "print(data)\n",
    "\n",
    "#问：为什么要转换成稀疏矩阵形式？\n",
    "#答：在图比较大或比较稠密的时候，使用边表edge_index这种图结构数据，需要在Aggregate阶段显式的矩阵化src（x_j），\n",
    "#这会导致很高的内存占用（可能导致GPU内存溢出），反而不如使用稀疏矩阵（SparseTensor）进行存储和运算了。\n",
    "#具体可参考https://pytorch-geometric.readthedocs.io/en/latest/notes/sparse_tensor.html\n",
    "#来自：https://blog.csdn.net/weixin_41650348/article/details/113857387"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cw0xZJKZI-n3"
   },
   "source": [
    "## Question 4: What is the number of features in the ogbn-arxiv graph? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "ZP844_nT2ZJl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The graph has 128 features\n"
     ]
    }
   ],
   "source": [
    "def graph_num_features(data):\n",
    "  # TODO: Implement this function that takes a PyG data object,\n",
    "  # and returns the number of features in the graph (in integer).\n",
    "\n",
    "  num_features = 0\n",
    "\n",
    "  ############# Your code here ############\n",
    "  ## (~1 line of code)\n",
    "  num_features=data.num_features\n",
    "  #https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Data.num_features\n",
    "  #########################################\n",
    "\n",
    "  return num_features\n",
    "\n",
    "num_features = graph_num_features(data)\n",
    "print('The graph has {} features'.format(num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9DP_yEQZ0NVW"
   },
   "source": [
    "# 3 GNN: Node Property Prediction\n",
    "\n",
    "In this section we will build our first graph neural network by using PyTorch Geometric and apply it on node property prediction (node classification).\n",
    "\n",
    "We will build the graph neural network by using GCN operator ([Kipf et al. (2017)](https://arxiv.org/pdf/1609.02907.pdf)).\n",
    "\n",
    "You should use the PyG built-in `GCNConv` layer directly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O4CcOUEoInjD"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "-DCtgcHpGIpd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.7.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "print(torch.__version__)\n",
    "\n",
    "# The PyG built-in GCNConv\n",
    "from torch_geometric.nn import GCNConv\n",
    "\n",
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset, Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0IK9z0wQIwzQ"
   },
   "source": [
    "## Load and Preprocess the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "0ibJ0ieoIwQM"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:The OGB package is out of date. Your version is 1.3.0, while the latest version is 1.3.1.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n"
     ]
    }
   ],
   "source": [
    "dataset_name = 'ogbn-arxiv'\n",
    "dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                 transform=T.ToSparseTensor())\n",
    "data = dataset[0]\n",
    "\n",
    "# Make the adjacency matrix to symmetric\n",
    "data.adj_t = data.adj_t.to_symmetric()\n",
    "#……我根本就搜不到这个函数的文档，只能找到源代码的位置：\n",
    "#https://github.com/rusty1s/pytorch_sparse/blob/master/torch_sparse/tensor.py\n",
    "#啊……算了，看不懂，我放弃了。直接用吧\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# If you use GPU, the device should be cuda\n",
    "print('Device: {}'.format(device))\n",
    "\n",
    "data = data.to(device)\n",
    "split_idx = dataset.get_idx_split()  #从结果直观地来看就是将数据集分成了train，valid，test三部分\n",
    "train_idx = split_idx['train'].to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch_sparse.tensor.SparseTensor"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PRW\n",
    "type(data.adj_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': tensor([     0,      1,      2,  ..., 169145, 169148, 169251]),\n",
       " 'valid': tensor([   349,    357,    366,  ..., 169185, 169261, 169296]),\n",
       " 'test': tensor([   346,    398,    451,  ..., 169340, 169341, 169342])}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PRW\n",
    "split_idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OgUA815bNJ8w"
   },
   "source": [
    "## GCN Model\n",
    "\n",
    "Now we will implement our GCN model!\n",
    "\n",
    "Please follow the figure below to implement your `forward` function.\n",
    "\n",
    "![test](https://drive.google.com/uc?id=128AuYAXNXGg7PIhJJ7e420DoPWKb-RtL)\n",
    "\n",
    "（为了方便无法看Google的读者浏览图片，下面是不用Google的地址，放到本地）\n",
    "![picture](cs224w-colab2-3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#PRW\n",
    "#ModuleList文档示例\n",
    "class MyModule(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyModule, self).__init__()\n",
    "        self.linears = torch.nn.ModuleList([torch.nn.Linear(10, 10) for i in range(10)])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # ModuleList can act as an iterable, or be indexed using ints\n",
    "        for i, l in enumerate(self.linears):\n",
    "            x = self.linears[i // 2](x) + l(x)  #//是整数除法\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MyModule(\n",
       "  (linears): ModuleList(\n",
       "    (0): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (1): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (2): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (3): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (4): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (5): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (6): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (7): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (8): Linear(in_features=10, out_features=10, bias=True)\n",
       "    (9): Linear(in_features=10, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PRW续上一cell\n",
    "mymodule=MyModule()\n",
    "mymodule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "IgspXTYpNJLA"
   },
   "outputs": [],
   "source": [
    "class GCN(torch.nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim, num_layers,\n",
    "                 dropout, return_embeds=False):\n",
    "        # TODO: Implement this function that initializes self.convs, \n",
    "        # self.bns, and self.softmax.\n",
    "\n",
    "        super(GCN, self).__init__()\n",
    "\n",
    "        # A list of GCNConv layers\n",
    "        self.convs = None\n",
    "\n",
    "        # A list of 1D batch normalization layers\n",
    "        self.bns = None\n",
    "\n",
    "        # The log softmax layer\n",
    "        self.softmax = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. You should use torch.nn.ModuleList for self.convs and self.bns\n",
    "        ## 2. self.convs has num_layers GCNConv layers\n",
    "        ## 3. self.bns has num_layers - 1 BatchNorm1d layers\n",
    "        ## 4. You should use torch.nn.LogSoftmax for self.softmax\n",
    "        ## 5. The parameters you can set for GCNConv include 'in_channels' and \n",
    "        ## 'out_channels'. More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv\n",
    "        ## 6. The only parameter you need to set for BatchNorm1d is 'num_features'\n",
    "        ## More information please refer to the documentation: \n",
    "        ## https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html\n",
    "        ## (~10 lines of code)\n",
    "        \n",
    "        #nn.ModuleList文档：https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html\n",
    "        self.convs = torch.nn.ModuleList()\n",
    "        for i in range(num_layers - 1):\n",
    "            self.convs.append(GCNConv(input_dim, hidden_dim))\n",
    "            input_dim = hidden_dim\n",
    "        self.convs.append(GCNConv(hidden_dim, output_dim))\n",
    "        \n",
    "        #Batch Normalization讲解视频：https://v.youku.com/v_show/id_XMTg1MTYwNDg2OA==\n",
    "        #这个东西要放在神经元和激励函数之间\n",
    "        self.bns=torch.nn.ModuleList([torch.nn.BatchNorm1d(hidden_dim) for i in range(num_layers-1)])\n",
    "        \n",
    "        self.softmax=torch.nn.LogSoftmax()\n",
    "        #########################################\n",
    "\n",
    "        # Probability of an element to be zeroed\n",
    "        self.dropout = dropout\n",
    "\n",
    "        # Skip classification layer and return node embeddings\n",
    "        self.return_embeds = return_embeds\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        for conv in self.convs:\n",
    "            conv.reset_parameters()\n",
    "        for bn in self.bns:\n",
    "            bn.reset_parameters()\n",
    "    #没看文档，应该是初始化参数的意思？\n",
    "\n",
    "    def forward(self, x, adj_t):\n",
    "        # TODO: Implement this function that takes the feature tensor x,\n",
    "        # edge_index tensor adj_t and returns the output tensor as\n",
    "        # shown in the figure.\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct the network as showing in the figure\n",
    "        ## 2. torch.nn.functional.relu and torch.nn.functional.dropout are useful\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch.org/docs/stable/nn.functional.html\n",
    "        ## 3. Don't forget to set F.dropout training to self.training\n",
    "        ## 4. If return_embeds is True, then skip the last softmax layer\n",
    "        ## (~7 lines of code)\n",
    "        for layer in range(len(self.convs)-1):  #layer：层数\n",
    "            x=self.convs[layer](x,adj_t)  #叠GCNConv\n",
    "            #forward(x: torch.Tensor, edge_index: Union[torch.Tensor, torch_sparse.tensor.SparseTensor], \n",
    "            #edge_weight: Optional[torch.Tensor] = None)\n",
    "            x=self.bns[layer](x)  #叠BN\n",
    "            x=F.relu(x)  #叠relu\n",
    "            x=F.dropout(x,self.dropout,self.training)  #叠dropout。这个self.dropout看下文是概率。\n",
    "            #这个self.training反正是要传的，还没仔细看为什么要传，但是有看到说必须要传\n",
    "        #最后一层\n",
    "        out=self.convs[-1](x,adj_t)  #GCNVonv\n",
    "        if not self.return_embeds:\n",
    "            out=self.softmax(out)\n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 4],\n",
      "        [ 5],\n",
      "        [28],\n",
      "        ...,\n",
      "        [28],\n",
      "        [23],\n",
      "        [22]], device='cuda:0')\n",
      "tensor([ 4,  5, 28,  ..., 28, 23, 22], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "#PRW\n",
    "#注意下面这个提取y的时候\n",
    "print(data.y[train_idx])\n",
    "print(data.y[train_idx,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "FF1hnHUhO81e"
   },
   "outputs": [],
   "source": [
    "def train(model, data, train_idx, optimizer, loss_fn):\n",
    "    # TODO: Implement this function that trains the model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## Note:\n",
    "    ## 1. Zero grad the optimizer\n",
    "    ## 2. Feed the data into the model\n",
    "    ## 3. Slicing the model output and label by train_idx\n",
    "    ## 4. Feed the sliced output and label to loss_fn\n",
    "    ## (~4 lines of code)\n",
    "    optimizer.zero_grad()\n",
    "    out=model(data.x,data.adj_t)\n",
    "    train_output=out[train_idx]\n",
    "    train_label=data.y[train_idx,0]\n",
    "    loss=loss_fn(train_output,train_label)\n",
    "    #########################################\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "aJdlrJQhPBsK"
   },
   "outputs": [],
   "source": [
    "# Test function here\n",
    "@torch.no_grad()\n",
    "def test(model, data, split_idx, evaluator):\n",
    "    # TODO: Implement this function that tests the model by \n",
    "    # using the given split_idx and evaluator.\n",
    "    model.eval()\n",
    "    #Sets the module in evaluation mode.\n",
    "\n",
    "    # The output of model on all data\n",
    "    out = None\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note:\n",
    "    ## 1. No index slicing here\n",
    "    out=model(data.x,data.adj_t)\n",
    "    #########################################\n",
    "\n",
    "    y_pred = out.argmax(dim=-1, keepdim=True)\n",
    "\n",
    "    train_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['train']],\n",
    "        'y_pred': y_pred[split_idx['train']],\n",
    "    })['acc']\n",
    "    valid_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['valid']],\n",
    "        'y_pred': y_pred[split_idx['valid']],\n",
    "    })['acc']\n",
    "    test_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['test']],\n",
    "        'y_pred': y_pred[split_idx['test']],\n",
    "    })['acc']\n",
    "\n",
    "    return train_acc, valid_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#PRW理解evaluation\n",
    "#from ogb.nodeproppred import PygNodePropPredDataset, Evaluator\n",
    "#evaluator = Evaluator(name='ogbn-arxiv')\n",
    "#print(evaluator.expected_output_format) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "o7F46xkuLiOL"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'device': 'cuda',\n",
       " 'num_layers': 3,\n",
       " 'hidden_dim': 256,\n",
       " 'dropout': 0.5,\n",
       " 'lr': 0.01,\n",
       " 'epochs': 100}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Please do not change the args\n",
    "args = {\n",
    "    'device': device,\n",
    "    'num_layers': 3,\n",
    "    'hidden_dim': 256,\n",
    "    'dropout': 0.5,\n",
    "    'lr': 0.01,\n",
    "    'epochs': 100,\n",
    "}\n",
    "args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "dT8RyM2cPGxM"
   },
   "outputs": [],
   "source": [
    "model = GCN(data.num_features, args['hidden_dim'],\n",
    "            dataset.num_classes, args['num_layers'],\n",
    "            args['dropout']).to(device)\n",
    "evaluator = Evaluator(name='ogbn-arxiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "qd5O5cnPPdVF",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/start/anaconda3/envs/cs224w/lib/python3.7/site-packages/ipykernel_launcher.py:86: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 4.0492, Train: 25.35%, Valid: 28.71% Test: 25.71%\n",
      "Epoch: 02, Loss: 2.3172, Train: 26.54%, Valid: 25.99% Test: 31.52%\n",
      "Epoch: 03, Loss: 1.9148, Train: 28.85%, Valid: 23.71% Test: 27.98%\n",
      "Epoch: 04, Loss: 1.7875, Train: 38.76%, Valid: 34.36% Test: 38.54%\n",
      "Epoch: 05, Loss: 1.6523, Train: 44.26%, Valid: 41.42% Test: 39.92%\n",
      "Epoch: 06, Loss: 1.5905, Train: 42.94%, Valid: 37.80% Test: 36.83%\n",
      "Epoch: 07, Loss: 1.5237, Train: 38.57%, Valid: 30.34% Test: 32.19%\n",
      "Epoch: 08, Loss: 1.4630, Train: 35.33%, Valid: 26.59% Test: 29.29%\n",
      "Epoch: 09, Loss: 1.4224, Train: 33.98%, Valid: 26.00% Test: 29.01%\n",
      "Epoch: 10, Loss: 1.3806, Train: 32.60%, Valid: 23.77% Test: 26.16%\n",
      "Epoch: 11, Loss: 1.3508, Train: 33.04%, Valid: 24.98% Test: 26.93%\n",
      "Epoch: 12, Loss: 1.3235, Train: 35.74%, Valid: 30.87% Test: 32.98%\n",
      "Epoch: 13, Loss: 1.2994, Train: 39.26%, Valid: 35.40% Test: 38.59%\n",
      "Epoch: 14, Loss: 1.2766, Train: 42.55%, Valid: 38.34% Test: 42.86%\n",
      "Epoch: 15, Loss: 1.2579, Train: 46.46%, Valid: 42.82% Test: 47.82%\n",
      "Epoch: 16, Loss: 1.2423, Train: 50.38%, Valid: 47.91% Test: 52.64%\n",
      "Epoch: 17, Loss: 1.2272, Train: 54.46%, Valid: 53.63% Test: 57.32%\n",
      "Epoch: 18, Loss: 1.2086, Train: 57.20%, Valid: 57.74% Test: 60.28%\n",
      "Epoch: 19, Loss: 1.1971, Train: 58.74%, Valid: 59.79% Test: 61.64%\n",
      "Epoch: 20, Loss: 1.1854, Train: 59.59%, Valid: 60.64% Test: 62.07%\n",
      "Epoch: 21, Loss: 1.1732, Train: 60.31%, Valid: 61.18% Test: 62.34%\n",
      "Epoch: 22, Loss: 1.1665, Train: 60.55%, Valid: 61.09% Test: 62.17%\n",
      "Epoch: 23, Loss: 1.1569, Train: 60.76%, Valid: 61.12% Test: 62.10%\n",
      "Epoch: 24, Loss: 1.1494, Train: 61.55%, Valid: 61.65% Test: 62.50%\n",
      "Epoch: 25, Loss: 1.1322, Train: 62.90%, Valid: 63.09% Test: 63.70%\n",
      "Epoch: 26, Loss: 1.1262, Train: 64.00%, Valid: 64.11% Test: 65.01%\n",
      "Epoch: 27, Loss: 1.1184, Train: 64.68%, Valid: 64.75% Test: 65.66%\n",
      "Epoch: 28, Loss: 1.1145, Train: 65.20%, Valid: 64.86% Test: 65.83%\n",
      "Epoch: 29, Loss: 1.1042, Train: 65.63%, Valid: 65.07% Test: 65.96%\n",
      "Epoch: 30, Loss: 1.0977, Train: 65.80%, Valid: 65.28% Test: 66.03%\n",
      "Epoch: 31, Loss: 1.0915, Train: 66.15%, Valid: 65.58% Test: 66.41%\n",
      "Epoch: 32, Loss: 1.0855, Train: 66.42%, Valid: 66.01% Test: 66.88%\n",
      "Epoch: 33, Loss: 1.0797, Train: 66.93%, Valid: 66.58% Test: 67.31%\n",
      "Epoch: 34, Loss: 1.0748, Train: 67.44%, Valid: 67.21% Test: 67.74%\n",
      "Epoch: 35, Loss: 1.0710, Train: 67.99%, Valid: 67.92% Test: 68.17%\n",
      "Epoch: 36, Loss: 1.0658, Train: 68.62%, Valid: 68.59% Test: 68.32%\n",
      "Epoch: 37, Loss: 1.0624, Train: 69.06%, Valid: 68.80% Test: 68.39%\n",
      "Epoch: 38, Loss: 1.0585, Train: 69.19%, Valid: 69.07% Test: 68.44%\n",
      "Epoch: 39, Loss: 1.0525, Train: 69.41%, Valid: 69.42% Test: 68.87%\n",
      "Epoch: 40, Loss: 1.0507, Train: 69.70%, Valid: 69.65% Test: 69.32%\n",
      "Epoch: 41, Loss: 1.0435, Train: 69.93%, Valid: 69.78% Test: 69.64%\n",
      "Epoch: 42, Loss: 1.0412, Train: 70.27%, Valid: 70.00% Test: 69.61%\n",
      "Epoch: 43, Loss: 1.0392, Train: 70.46%, Valid: 70.18% Test: 69.69%\n",
      "Epoch: 44, Loss: 1.0333, Train: 70.59%, Valid: 70.13% Test: 69.82%\n",
      "Epoch: 45, Loss: 1.0307, Train: 70.59%, Valid: 70.07% Test: 69.97%\n",
      "Epoch: 46, Loss: 1.0276, Train: 70.72%, Valid: 70.00% Test: 69.94%\n",
      "Epoch: 47, Loss: 1.0243, Train: 70.80%, Valid: 70.11% Test: 69.78%\n",
      "Epoch: 48, Loss: 1.0219, Train: 70.85%, Valid: 70.06% Test: 69.66%\n",
      "Epoch: 49, Loss: 1.0178, Train: 71.00%, Valid: 70.14% Test: 69.66%\n",
      "Epoch: 50, Loss: 1.0142, Train: 71.01%, Valid: 70.26% Test: 69.60%\n",
      "Epoch: 51, Loss: 1.0094, Train: 71.10%, Valid: 70.46% Test: 69.63%\n",
      "Epoch: 52, Loss: 1.0076, Train: 71.29%, Valid: 70.58% Test: 69.63%\n",
      "Epoch: 53, Loss: 1.0048, Train: 71.49%, Valid: 70.71% Test: 69.50%\n",
      "Epoch: 54, Loss: 1.0041, Train: 71.55%, Valid: 70.54% Test: 69.41%\n",
      "Epoch: 55, Loss: 0.9992, Train: 71.59%, Valid: 70.51% Test: 69.44%\n",
      "Epoch: 56, Loss: 0.9976, Train: 71.64%, Valid: 70.63% Test: 69.61%\n",
      "Epoch: 57, Loss: 0.9940, Train: 71.68%, Valid: 70.61% Test: 69.62%\n",
      "Epoch: 58, Loss: 0.9913, Train: 71.76%, Valid: 70.56% Test: 69.31%\n",
      "Epoch: 59, Loss: 0.9947, Train: 71.87%, Valid: 70.68% Test: 69.53%\n",
      "Epoch: 60, Loss: 0.9858, Train: 71.95%, Valid: 70.87% Test: 69.82%\n",
      "Epoch: 61, Loss: 0.9847, Train: 71.90%, Valid: 70.95% Test: 70.15%\n",
      "Epoch: 62, Loss: 0.9830, Train: 71.92%, Valid: 70.92% Test: 70.16%\n",
      "Epoch: 63, Loss: 0.9800, Train: 71.99%, Valid: 71.01% Test: 70.24%\n",
      "Epoch: 64, Loss: 0.9785, Train: 72.09%, Valid: 71.10% Test: 70.17%\n",
      "Epoch: 65, Loss: 0.9766, Train: 72.17%, Valid: 71.17% Test: 70.13%\n",
      "Epoch: 66, Loss: 0.9722, Train: 72.20%, Valid: 71.18% Test: 70.19%\n",
      "Epoch: 67, Loss: 0.9742, Train: 72.16%, Valid: 71.15% Test: 70.63%\n",
      "Epoch: 68, Loss: 0.9688, Train: 72.10%, Valid: 71.04% Test: 70.69%\n",
      "Epoch: 69, Loss: 0.9681, Train: 72.27%, Valid: 71.23% Test: 70.66%\n",
      "Epoch: 70, Loss: 0.9608, Train: 72.45%, Valid: 71.23% Test: 70.30%\n",
      "Epoch: 71, Loss: 0.9639, Train: 72.57%, Valid: 71.17% Test: 70.01%\n",
      "Epoch: 72, Loss: 0.9603, Train: 72.57%, Valid: 71.31% Test: 70.30%\n",
      "Epoch: 73, Loss: 0.9572, Train: 72.52%, Valid: 71.45% Test: 70.81%\n",
      "Epoch: 74, Loss: 0.9572, Train: 72.51%, Valid: 71.52% Test: 70.92%\n",
      "Epoch: 75, Loss: 0.9554, Train: 72.62%, Valid: 71.51% Test: 70.69%\n",
      "Epoch: 76, Loss: 0.9542, Train: 72.76%, Valid: 71.41% Test: 70.15%\n",
      "Epoch: 77, Loss: 0.9537, Train: 72.87%, Valid: 71.55% Test: 70.44%\n",
      "Epoch: 78, Loss: 0.9498, Train: 72.85%, Valid: 71.53% Test: 70.85%\n",
      "Epoch: 79, Loss: 0.9486, Train: 72.73%, Valid: 71.41% Test: 71.09%\n",
      "Epoch: 80, Loss: 0.9472, Train: 72.76%, Valid: 71.46% Test: 71.09%\n",
      "Epoch: 81, Loss: 0.9465, Train: 73.03%, Valid: 71.60% Test: 70.77%\n",
      "Epoch: 82, Loss: 0.9410, Train: 73.12%, Valid: 71.51% Test: 70.52%\n",
      "Epoch: 83, Loss: 0.9386, Train: 73.05%, Valid: 71.57% Test: 70.52%\n",
      "Epoch: 84, Loss: 0.9383, Train: 73.01%, Valid: 71.67% Test: 70.83%\n",
      "Epoch: 85, Loss: 0.9423, Train: 73.11%, Valid: 71.69% Test: 70.79%\n",
      "Epoch: 86, Loss: 0.9351, Train: 73.11%, Valid: 71.53% Test: 70.54%\n",
      "Epoch: 87, Loss: 0.9336, Train: 73.20%, Valid: 71.34% Test: 70.33%\n",
      "Epoch: 88, Loss: 0.9324, Train: 73.30%, Valid: 71.51% Test: 70.51%\n",
      "Epoch: 89, Loss: 0.9315, Train: 73.39%, Valid: 71.86% Test: 70.98%\n",
      "Epoch: 90, Loss: 0.9284, Train: 73.41%, Valid: 71.92% Test: 70.98%\n",
      "Epoch: 91, Loss: 0.9278, Train: 73.38%, Valid: 71.85% Test: 70.85%\n",
      "Epoch: 92, Loss: 0.9248, Train: 73.28%, Valid: 71.77% Test: 71.09%\n",
      "Epoch: 93, Loss: 0.9242, Train: 73.42%, Valid: 71.64% Test: 70.69%\n",
      "Epoch: 94, Loss: 0.9197, Train: 73.46%, Valid: 71.46% Test: 70.07%\n",
      "Epoch: 95, Loss: 0.9235, Train: 73.65%, Valid: 71.56% Test: 70.26%\n",
      "Epoch: 96, Loss: 0.9211, Train: 73.65%, Valid: 71.84% Test: 70.87%\n",
      "Epoch: 97, Loss: 0.9153, Train: 73.45%, Valid: 71.69% Test: 71.03%\n",
      "Epoch: 98, Loss: 0.9190, Train: 73.57%, Valid: 71.85% Test: 71.05%\n",
      "Epoch: 99, Loss: 0.9167, Train: 73.65%, Valid: 71.84% Test: 71.05%\n",
      "Epoch: 100, Loss: 0.9143, Train: 73.72%, Valid: 71.84% Test: 71.04%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "# reset the parameters to initial random value\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = F.nll_loss\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "  loss = train(model, data, train_idx, optimizer, loss_fn)\n",
    "  result = test(model, data, split_idx, evaluator)\n",
    "  train_acc, valid_acc, test_acc = result\n",
    "  if valid_acc > best_valid_acc:\n",
    "      best_valid_acc = valid_acc\n",
    "      best_model = copy.deepcopy(model)\n",
    "  print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "EqcextqOL2FX"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 73.41%, Valid: 71.92% Test: 70.98%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/start/anaconda3/envs/cs224w/lib/python3.7/site-packages/ipykernel_launcher.py:86: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    }
   ],
   "source": [
    "best_result = test(best_model, data, split_idx, evaluator)\n",
    "train_acc, valid_acc, test_acc = best_result\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "duMEg-olLjbJ"
   },
   "source": [
    "## Question 5: What are your `best_model` validation and test accuracy? Please report them on Gradescope. For example, for an accuracy such as 50.01%, just report 50.01 and please don't include the percent sign. (20 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R8pOD6y80TyI"
   },
   "source": [
    "# 4 GNN: Graph Property Prediction\n",
    "\n",
    "In this section we will create a graph neural network for graph property prediction (graph classification)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#PRW\n",
    "#以下内容应该是参照这个了：https://github.com/snap-stanford/ogb/tree/master/examples/graphproppred/mol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vRg5VOEdQTa4"
   },
   "source": [
    "## Load and preprocess the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "LXb-O5QUIgTH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n",
      "Task type: binary classification\n"
     ]
    }
   ],
   "source": [
    "from ogb.graphproppred import PygGraphPropPredDataset, Evaluator\n",
    "from torch_geometric.data import DataLoader\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# Load the dataset \n",
    "dataset = PygGraphPropPredDataset(name='ogbg-molhiv')\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Device: {}'.format(device))\n",
    "\n",
    "split_idx = dataset.get_idx_split()\n",
    "\n",
    "# Check task type\n",
    "print('Task type: {}'.format(dataset.task_type))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "7cHHbgW1c5hi"
   },
   "outputs": [],
   "source": [
    "# Load the data sets into dataloader\n",
    "# We will train the graph classification task on a batch of 32 graphs\n",
    "# Shuffle the order of graphs for training set\n",
    "train_loader = DataLoader(dataset[split_idx[\"train\"]], batch_size=32, shuffle=True, num_workers=0)\n",
    "valid_loader = DataLoader(dataset[split_idx[\"valid\"]], batch_size=32, shuffle=False, num_workers=0)\n",
    "test_loader = DataLoader(dataset[split_idx[\"test\"]], batch_size=32, shuffle=False, num_workers=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "AYrSnOj0Y4DK"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'device': 'cuda',\n",
       " 'num_layers': 5,\n",
       " 'hidden_dim': 256,\n",
       " 'dropout': 0.5,\n",
       " 'lr': 0.001,\n",
       " 'epochs': 30}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Please do not change the args\n",
    "args = {\n",
    "    'device': device,\n",
    "    'num_layers': 5,\n",
    "    'hidden_dim': 256,\n",
    "    'dropout': 0.5,\n",
    "    'lr': 0.001,\n",
    "    'epochs': 30,\n",
    "}\n",
    "args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7WLhguSTeazy"
   },
   "source": [
    "## Graph Prediction Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u05Z14TRYPGn"
   },
   "source": [
    "Now we will implement our GCN Graph Prediction model!\n",
    "\n",
    "We will reuse the existing GCN model to generate `node_embeddings` and use  Global Pooling on the nodes to predict properties for the whole graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "3_Kq3zyjeZ22"
   },
   "outputs": [],
   "source": [
    "from ogb.graphproppred.mol_encoder import AtomEncoder\n",
    "from torch_geometric.nn import global_add_pool, global_mean_pool\n",
    "\n",
    "### GCN to predict graph property\n",
    "class GCN_Graph(torch.nn.Module):\n",
    "    def __init__(self, hidden_dim, output_dim, num_layers, dropout):\n",
    "        super(GCN_Graph, self).__init__()\n",
    "\n",
    "        # Load encoders for Atoms in molecule graphs\n",
    "        self.node_encoder = AtomEncoder(hidden_dim)\n",
    "\n",
    "        # Node embedding model\n",
    "        # Note that the input_dim and output_dim are set to hidden_dim\n",
    "        self.gnn_node = GCN(hidden_dim, hidden_dim,\n",
    "            hidden_dim, num_layers, dropout, return_embeds=True)\n",
    "\n",
    "        self.pool = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Initialize the self.pool to global mean pooling layer\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        ## (~1 line of code)\n",
    "        self.pool=global_mean_pool\n",
    "        #########################################\n",
    "\n",
    "        # Output layer\n",
    "        self.linear = torch.nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "\n",
    "    def reset_parameters(self):\n",
    "      self.gnn_node.reset_parameters()\n",
    "      self.linear.reset_parameters()\n",
    "\n",
    "    def forward(self, batched_data):\n",
    "        # TODO: Implement this function that takes the input tensor batched_data,\n",
    "        # returns a batched output tensor for each graph.\n",
    "        x, edge_index, batch = batched_data.x, batched_data.edge_index, batched_data.batch\n",
    "        embed = self.node_encoder(x)\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct node embeddings using existing GCN model\n",
    "        ## 2. Use global pooling layer to construct features for the whole graph\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        ## 3. Use a linear layer to predict the graph property \n",
    "        ## (~3 lines of code)\n",
    "        out=self.gnn_node(embed,edge_index)\n",
    "        out=self.pool(out,batch)\n",
    "        out=self.linear(out)\n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "FJjnGuMSbjX0"
   },
   "outputs": [],
   "source": [
    "def train(model, device, data_loader, optimizer, loss_fn):\n",
    "    # TODO: Implement this function that trains the model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    for step, batch in enumerate(tqdm(data_loader, desc=\"Iteration\")):\n",
    "      batch = batch.to(device)\n",
    "\n",
    "      if batch.x.shape[0] == 1 or batch.batch[-1] == 0:\n",
    "          pass\n",
    "      else:\n",
    "        ## ignore nan targets (unlabeled) when computing training loss.\n",
    "        is_labeled = batch.y == batch.y\n",
    "        \n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Zero grad the optimizer\n",
    "        ## 2. Feed the data into the model\n",
    "        ## 3. Use `is_labeled` mask to filter output and labels\n",
    "        ## 4. You might change the type of label\n",
    "        ## 5. Feed the output and label to loss_fn\n",
    "        ## (~3 lines of code)\n",
    "        optimizer.zero_grad()\n",
    "        op=model(batch)\n",
    "        train_op=op[is_labeled]\n",
    "        #train_labels=batch.y[is_labeled]  #Warning: 答案这里用了view，我参考了下面的y.view，就把view加上了，其实我不知道具体为什么要加\n",
    "        train_labels=batch.y[is_labeled].view(-1)\n",
    "        #loss=loss_fn(train_op,train_labels)  #RuntimeError: result type Float can't be cast to the desired output type Long\n",
    "        loss=loss_fn(train_op.float(),train_labels.float())  #https://pytorch.org/docs/stable/tensors.html#torch.Tensor.float\n",
    "\n",
    "        #########################################\n",
    "\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "ztPHXq_Gzn7U"
   },
   "outputs": [],
   "source": [
    "# The evaluation function\n",
    "def eval(model, device, loader, evaluator):\n",
    "    model.eval()\n",
    "    y_true = []\n",
    "    y_pred = []\n",
    "\n",
    "    for step, batch in enumerate(tqdm(loader, desc=\"Iteration\")):\n",
    "        batch = batch.to(device)\n",
    "\n",
    "        if batch.x.shape[0] == 1:\n",
    "            pass\n",
    "        else:\n",
    "            with torch.no_grad():\n",
    "                pred = model(batch)\n",
    "\n",
    "            y_true.append(batch.y.view(pred.shape).detach().cpu())\n",
    "            y_pred.append(pred.detach().cpu())\n",
    "\n",
    "    y_true = torch.cat(y_true, dim = 0).numpy()\n",
    "    y_pred = torch.cat(y_pred, dim = 0).numpy()\n",
    "\n",
    "    input_dict = {\"y_true\": y_true, \"y_pred\": y_pred}\n",
    "\n",
    "    return evaluator.eval(input_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "MR1wQ4hMZeMw"
   },
   "outputs": [],
   "source": [
    "model = GCN_Graph(args['hidden_dim'],\n",
    "            dataset.num_tasks, args['num_layers'],\n",
    "            args['dropout']).to(device)\n",
    "evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "qJGTNZiuZy0A",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34a35c121ea24735a36138779fb32f6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c98f58d807444e2a880f5bc9c16018a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4d034358e6c4a94a54b4e50167dcdb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a2e6de8c8e04e5496749c9135b8bfbe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.0767, Train: 72.64%, Valid: 64.67% Test: 68.98%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92cb0423464c4089a6ce9be6119ebbc5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f99abcf1d29431b95cf7e25c5e2c80a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "692c75a8c20d4a268c4a093ad867a835",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a67cd7e1f554fc880e69e13bebc5564",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0715, Train: 75.41%, Valid: 72.44% Test: 73.03%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2e53ca7a65e41c1842b5b27454ec680",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df5fd620e83b4b6dafdfebd17ff8be4c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "503a413f0b854e8baee669cc812de6bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce74a38a0f364575af81090e7734b3ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.8831, Train: 74.01%, Valid: 73.45% Test: 69.98%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62ef7bca5ece47ca82ef78f406553d5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7176e0c0ac3e4dafa98887053ec2ce68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2c2186badc549129b9c1e26a0b954bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a301491f2da4f698e1ac4d35f712464",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.7005, Train: 75.95%, Valid: 75.60% Test: 72.63%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "490bffe2867c4e798c2ac2900563a352",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7364c24373743f784e6bb73467a97cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c62acfadea24d6ca876d66bffa4b48e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b9c03442f76447386ea9182a8aa5c4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.9696, Train: 77.58%, Valid: 75.84% Test: 71.50%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a31a2fd6eb4f4ea6b7c48dd9bfc63d73",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a997f06cd6fa4c69979bd4e48d4e2f81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef4f8c7366e3471f94d6a9dcba76f9c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f219d90e5104adfab9110b3f95210d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.0319, Train: 78.03%, Valid: 74.54% Test: 67.91%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "711cc61bdb664a5b9752090eb4a48e95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "90fdc878135948c7a394eb262842af72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9463791e4cf64b839b84f715378b796e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "180172a99cfd422fa3359cfb14aac88d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.9489, Train: 76.78%, Valid: 73.17% Test: 70.58%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87e66a506e17465092aebd595432d3bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c286355d9d894fa7b26d756c4bde775d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e399e693f6f43febfc7a45bf5e02eca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0894413d633f4437a812cf8bb04b3a72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0496, Train: 78.61%, Valid: 76.33% Test: 70.07%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "541dade930bb45cd96a79371726e1fd2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef6f1487ede04a3abe1c183315aae122",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96886fc2dee7400a91a2a1e4e830dea7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f01e3eeaf2734ae0b77c8edf1bc131f4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.6267, Train: 79.32%, Valid: 74.91% Test: 72.06%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "63da42ec87d0447fb49e1d76f7367f18",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9f668a27d4d4467ac9835de3506d2a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "490e786fdac4480cb2c1c9896836f4fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b79b9f0e0fb74019a632a81e951cb16c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0494, Train: 78.93%, Valid: 74.99% Test: 71.26%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e362e56ff37142f1a81dcd9d2ecd634e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29081906a43045b8bc13119c6f9127e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "970b26a3715c4e4a871fc6e95f1d98c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14c098a49a6841fcb45d57dfe61128d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0247, Train: 79.27%, Valid: 71.67% Test: 69.82%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af089e1a4b86456083e2a36502a9d45b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "203a9a74a60a40869fb46e5240a968c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81397c0d552d48108ec0299f177fb4d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b80359543aa44fbc9542a0b31d041f34",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0416, Train: 80.28%, Valid: 76.14% Test: 73.35%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47fb277e968c4a87b789390cb5d43478",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da1a52f8a2fe42528a8f360e22a0185b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d855345f45d4762ab20291222884cd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adc874ff043042d383de8be423e3eab0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0214, Train: 80.34%, Valid: 78.13% Test: 71.27%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9fa39e8b7f694c5197a5e28598acd8c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9544baa505774ca3b2a5ba55bd194c8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e784cf78c8984462bdf3993fefe9db70",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32d73e077b754c98b7f83539c13bc2fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.7701, Train: 79.76%, Valid: 78.75% Test: 71.26%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed423ba557ca4d788a12b4233686aba4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6b84654cf59477e9ae0fdc59a03b6f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "493d45fa47144c4180edfc49eb78ae27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2ab0edbbd5240689471773ef29d9cd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0464, Train: 80.66%, Valid: 73.61% Test: 72.47%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a4bbe7b5fb34609a164ee8f4c78dc12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "801cb5f1b8e94099846dc5bd686e7fba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e11084f50134ee0a7217f454f235340",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c51d1e7d9b3489ca49d48e59c51e119",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.0347, Train: 80.90%, Valid: 77.93% Test: 71.84%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9392930247a441fbefa006ca829cee1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09003d10f1c14cc2aa240bcfe127994c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d59027d226b4a39a6e41a5ac9f30eaa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0352b2b72d494d0fbf9df1c42ba00503",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.3408, Train: 81.16%, Valid: 78.28% Test: 70.69%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3d24e63177d453ab2e9aff1989598a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3799a94c5f26415fb12724498bae0d52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bfc0066face43868dd518db6d656031",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8beab0493da844009580fca2a0c0cc3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.0169, Train: 81.09%, Valid: 75.24% Test: 72.52%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c04d834b622d4d04b3158d98677f7a25",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c2f90c7b0a54f5ab6337ae6ca5e88ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db298629f313458ea40b2e5483aca5b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c36a9abab7a4375ac8396bcaa0927ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0454, Train: 80.81%, Valid: 77.86% Test: 72.04%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbb2e73e647d47caafc7afeaf2569638",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c07e9030898742a8a15dcce42d4f1f22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a0f9b97a8d9441f9efc73850b7a0cef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "784953bb822e47f98783a73a6de7e60c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0259, Train: 82.56%, Valid: 76.30% Test: 73.01%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "50b3ff508a744fc1887a2b591ccd6d71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4daddd80aa4741bdaaddc09d7d2ad711",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41ec9322980a47598f10e9507e373356",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "659b0104986e4dcfb73ea3dab6eed17f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.0447, Train: 82.79%, Valid: 77.15% Test: 73.05%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bcc589bc8d8a430594dd6610cde984cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "474fd27874f3417e9aaf986c9f175123",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "399bd40f4ab94149a3d2aa1121cdfc1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62b228c2c9ca465284c071d3db98c83f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0141, Train: 82.16%, Valid: 74.03% Test: 73.44%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b70b4aa5e6004c5398e8422552cc819f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "113c79e08b5c45e4bd5f12686346f001",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c6de8e8005648578d7719cb4d366d19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2398f6ddabce41dda84b3aa62e2e0116",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0253, Train: 82.23%, Valid: 79.39% Test: 73.94%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b4f1c1be8db40c793c7de4f9b593aae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07c4d4f126904537abbc0d4154448aff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d13d9d1b1130430d98c84f4975e3b17b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60e861d0c6a54f6684cf43537365f28a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.4823, Train: 82.83%, Valid: 77.74% Test: 73.83%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48eeeacb9f7a4d7ea6cfe6ba64b41bca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7248dc4ebf44a118af910b6d077c2b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6678de6f04e4279a25f1085fe8b7d23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bb0e0ae21c2497cb47db8cf54fc8313",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.0309, Train: 82.96%, Valid: 78.03% Test: 73.94%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34624d42393f4447a2c12f111d0b4b52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "066de2fad50a4f208dd968667e55f343",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80df46e3097047a1be4767f4d781bd0c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "45e5589bc22142e7912df39c7cc70fb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0210, Train: 82.65%, Valid: 80.57% Test: 75.48%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3e0bfe2508f4b82a326ea4366db8103",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b34d9d40f7545048285979a49059c65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "288f80ea49aa4954b6244a2a2b4b215f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82283b62af4943a28bc0bd14309674b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0365, Train: 83.20%, Valid: 77.83% Test: 74.73%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af1be2d28a8e440d9d456232ae030846",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f522eed72dd44ec4b333864731c6edf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c65d4b0f4144a509d97d461e041adc9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11d72aabf950439095709ba3fd1e0f12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.0221, Train: 84.29%, Valid: 77.70% Test: 74.01%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e47e5f3ac774308acf03db7b9c0e813",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d466f58ed8504c35883cc829631a59cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d355d10c9bb74a29867c6b9aabb56418",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3dbac9467304b62833da917e88d2cd1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.1128, Train: 83.47%, Valid: 79.37% Test: 75.42%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9715e8e6b294003a07793bca8c271d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8e82012333d487e9a37524bb91bf8b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64c59844d38949ab943fe2376333cd6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e225b4d851948e5b0bc743e94f92820",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.0360, Train: 83.72%, Valid: 80.27% Test: 75.23%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "  print('Training...')\n",
    "  loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "  print('Evaluating...')\n",
    "  train_result = eval(model, device, train_loader, evaluator)\n",
    "  val_result = eval(model, device, valid_loader, evaluator)\n",
    "  test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "  train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "  if valid_acc > best_valid_acc:\n",
    "      best_valid_acc = valid_acc\n",
    "      best_model = copy.deepcopy(model)\n",
    "  print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "Oq5QaG21dOOO"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7210c29f6a854d04a7c488ed458fa629",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "630cee79f28b4f80966b0fa518547d4d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95d1fd22838446638f54c93e37096346",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 82.65%, Valid: 80.57% Test: 75.48%\n"
     ]
    }
   ],
   "source": [
    "train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "valid_acc = eval(best_model, device, valid_loader, evaluator)[dataset.eval_metric]\n",
    "test_acc = eval(best_model, device, test_loader, evaluator)[dataset.eval_metric]\n",
    "\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-uKs6j6t1ah3"
   },
   "source": [
    "## Question 6: What are your `best_model` validation and test ROC-AUC score? Please report them on Gradescope. For example, for an ROC-AUC score such as 50.01%, just report 50.01 and please don't include the percent sign. (20 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0be8429022bd4f9f81ebfc9723061795",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rocauc': 0.8056964040760337}\n"
     ]
    }
   ],
   "source": [
    "#PRW\n",
    "valid_acc = eval(best_model, device, valid_loader, evaluator)\n",
    "print(valid_acc)\n",
    "#这个结果就是rocauc，为什么上上个cell非要管它叫acc？应该是原notebook文件代码的问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#PRW对OGB的evaluation的理解\n",
    "#图任务的evaluation：\n",
    "#For evaluation metric, we closely follow [2]. Specifically, for ogbg-molhiv, we use ROC-AUC for evaluation.\n",
    "#（上一行来自：https://ogb.stanford.edu/docs/graphprop/）\n",
    "\n",
    "#节点任务的evaluation则是acc（见上面那个我用以理解evaluation的cell）\n",
    "print(evaluator.expected_output_format) \n",
    "#上面那个cell的输出（如果按顺序跑是In[20]）是：\n",
    "\"\"\"\n",
    "==== Expected output format of Evaluator for ogbn-arxiv\n",
    "{'acc': acc}\n",
    "- acc (float): Accuracy score averaged across 1 task(s)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gBi_t8n0iZ4P"
   },
   "source": [
    "## Question 7 (Optional): Experiment with other two global pooling layers other than mean pooling in Pytorch Geometric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "taxEEWyh1_jq"
   },
   "source": [
    "PRW  \n",
    "https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers  \n",
    "global_add_pool  \n",
    "global_mean_pool  \n",
    "global_max_pool  \n",
    "global_sort_pool  \n",
    "GlobalAttention  \n",
    "Set2Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#选做就不做了，以后有缘了再做"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "CS224W - Colab 2.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}