unit09_neural/lab_music_partial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdynRPcZHTSA"
      },
      "source": [
        "# Lab:  Neural Networks for Music Classification\n",
        "\n",
        "In addition to the concepts in the [MNIST neural network demo](./demo2_mnist_neural.ipynb), in this lab, you will learn to:\n",
        "* Load a file from a URL\n",
        "* Extract simple features from audio samples for machine learning tasks such as speech recognition and classification\n",
        "* Build a simple neural network for music classification using these features\n",
        "* Use a callback to store the loss and accuracy history in the training process\n",
        "* Optimize the learning rate of the neural network\n",
        "\n",
        "To illustrate the basic concepts, we will look at a relatively simple music classification problem.  Given a sample of music, we want to determine which instrument (e.g. trumpet, violin, piano) is playing.  This dataset was generously supplied by [Prof. Juan Bello](http://steinhardt.nyu.edu/faculty/Juan_Pablo_Bello) at NYU Stenihardt  and his former PhD student Eric Humphrey (now at Spotify).  They have a complete website dedicated to deep learning methods in music informatics:\n",
        "\n",
        "http://marl.smusic.nyu.edu/wordpress/projects/feature-learning-deep-architectures/deep-learning-python-tutorial/\n",
        "\n",
        "You can also check out Juan's <a href=\"http://www.nyu.edu/classes/bello/ACA.html\">course</a>.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iu0qPk4KHTSC"
      },
      "source": [
        "We first load the standard packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "juHRiZjTHTSD"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohlGTFBgHTSD"
      },
      "source": [
        "## Audio Feature Extraction with Librosa\n",
        "\n",
        "The key to audio classification is to extract the correct features. In addition to `pytorch`, we will need the `librosa` package.  The `librosa` package in python has a rich set of methods extracting the features of audio samples commonly used in machine learning tasks such as speech recognition and sound classification.\n",
        "\n",
        "Installation instructions and complete documentation for the package are given on the [librosa main page](https://librosa.github.io/librosa/).  On most systems, you should be able to simply use:\n",
        "\n",
        "    pip install -u librosa\n",
        "    \n",
        "If you are using Google colab, the packages are already installed.\n",
        "After you have installed the package, try to import it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "IU8id75mHTSD"
      },
      "outputs": [],
      "source": [
        "import librosa\n",
        "import librosa.display\n",
        "import librosa.feature"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yhwPUHizHTSD"
      },
      "source": [
        "In this lab, we will use a set of music samples from the website:\n",
        "\n",
        "http://theremin.music.uiowa.edu\n",
        "\n",
        "This website has a great set of samples for audio processing.  Look on the web for how to use the `requests.get` and `file.write` commands to load the file at the URL provided into your working directory.\n",
        "\n",
        "You can play the audio sample by copying the file to your local machine and playing it on any media player.  If you listen to it you will hear a soprano saxaphone (with vibrato) playing four notes (C, C#, D, Eb)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "EVaXGSlcHTSD"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "fn = \"SopSax.Vib.pp.C6Eb6.aiff\"\n",
        "url = \"http://theremin.music.uiowa.edu/sound files/MIS/Woodwinds/sopranosaxophone/\"+fn\n",
        "\n",
        "# TODO:  Load the file from url and save it in a file under the name fn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeuAAiLAHTSD"
      },
      "source": [
        "Next, use `librosa` command `librosa.load` to read the audio file with filename `fn` and get the samples `y` and sample rate `sr`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Lw4dxhQfHTSD"
      },
      "outputs": [],
      "source": [
        "# TODO\n",
        "# y, sr = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vOqbE0CtHTSD"
      },
      "source": [
        "Extracting features from audio files is an entire subject on its own right.  A commonly used set of features are called the Mel Frequency Cepstral Coefficients (MFCCs).  These are derived from the so-called mel spectrogram which is something like a regular spectrogram, but the power and frequency are represented in log scale, which more naturally aligns with human perceptual processing.  You can run the code below to display the mel spectrogram from the audio sample.\n",
        "\n",
        "You can easily see the four notes played in the audio track.  You also see the 'harmonics' of each notes, which are other tones at integer multiples of the fundamental frequency of each note."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Da_mKC9_HTSD"
      },
      "outputs": [],
      "source": [
        "S = librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128, fmax=8000)\n",
        "librosa.display.specshow(librosa.amplitude_to_db(S),\n",
        "                         y_axis='mel', fmax=8000, x_axis='time')\n",
        "plt.colorbar(format='%+2.0f dB')\n",
        "plt.title('Mel spectrogram')\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jYHXiVPHTSE"
      },
      "source": [
        "## Downloading the Data\n",
        "\n",
        "Using the MFCC features described above, Eric Humphrey and Juan Bellow have created a complete data set that can used for instrument classification.  Essentially, they collected a number of data files from the website above.  For each audio file, they then segmented the track into notes and then extracted 120 MFCCs for each note.  The goal is to recognize the instrument from the 120 MFCCs.  The process of feature extraction is quite involved.  So, we will just use their processed data provided at:\n",
        "\n",
        "https://github.com/marl/dl4mir-tutorial/blob/master/README.md\n",
        "\n",
        "Note the password.  Load the four files into some directory, say  `instrument_dataset`.  Then, load them with the commands."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data_dir = 'instrument_dataset/'\n",
        "import os\n",
        "fn = os.path.join(data_dir, 'uiowa_train_data.npy')\n",
        "Xtr = np.load(fn)\n",
        "fn = os.path.join(data_dir, 'uiowa_train_labels.npy')\n",
        "ytr = np.load(fn)\n",
        "fn = os.path.join(data_dir, 'uiowa_test_data.npy')\n",
        "Xts = np.load(fn)\n",
        "fn = os.path.join(data_dir, 'uiowa_test_labels.npy')\n",
        "yts = np.load(fn)"
      ],
      "metadata": {
        "id": "hJha9MSsIpDu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xECVvt48HTSE"
      },
      "source": [
        "Looking at the data files:\n",
        "* What are the number of training and test samples?\n",
        "* What is the number of features for each sample?\n",
        "* How many classes (i.e. instruments) are there per class?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iuJzuN8pHTSE"
      },
      "outputs": [],
      "source": [
        "# TODO\n",
        "#  ntr = ...\n",
        "#  nts = ...\n",
        "#  nfeatures = ...\n",
        "#  nclasses = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vww8-EyeHTSE"
      },
      "source": [
        "Before continuing, you must scale the training and test data, `Xtr` and `Xts`.  Compute the mean and std deviation of each feature in `Xtr` and create a new training data set, `Xtr_scale`, by subtracting the mean and dividing by the std deviation.  Also compute a scaled test data set, `Xts_scale` using the mean and std deviation learned from the training data set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r_3lBu4bHTSE"
      },
      "outputs": [],
      "source": [
        "# TODO Scale the training and test matrices\n",
        "# Xtr_scale = ...\n",
        "# Xts_scale = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "ML8UmK20HTSE"
      },
      "source": [
        "## Building a Neural Network Classifier\n",
        "\n",
        "Following the example in [MNIST neural network demo](./mnist_neural.ipynb).   create a neural network `model` with:\n",
        "* `nfeatures` inputs\n",
        "* `nh=256` hidden units\n",
        "* `sigmoid` activation\n",
        "* fully connected layer with `nclasses` outputs\n",
        "* `nn.Softmax` output\n",
        "* select the input and output shapes correctly\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "# Create the neural network\n",
        "#    class Model(nn.Module):\n",
        "#       def __init__(...):\n",
        "#           ...\n",
        "#       def forward(...):\n",
        "#           ...\n",
        "\n",
        "\n",
        "# TODO:  Create the model\n",
        "#  nhid = 256\n",
        "#  model = Model(...)"
      ],
      "metadata": {
        "id": "f54WS-pRJVnD"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now print a summary of the model.  You can use the `torchsummary.summary` method as follows:"
      ],
      "metadata": {
        "id": "1PutxzLhyYQU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torchsummary import summary\n",
        "\n",
        "# TODO\n",
        "#   summary(model, ...)  # put in the input shape"
      ],
      "metadata": {
        "id": "pFgv2Ymkygcl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, create `pytorch` `Dataloader` classes from the numpy arrays.  Note that you must use the scaled data `Xtr_scale` and `Xts_scale`."
      ],
      "metadata": {
        "id": "ONS8EcfqyyET"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YT_KR5nCHTSE"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "\n",
        "# TODO:  Convert your data to PyTorch tensors.\n",
        "# Xtr_torch = ...  (use dtype=torch.float32)\n",
        "# Xts_torch = ...  (use dtype=torch.float32)\n",
        "# ytr_torch = ...  (use dtype=torch.int32)\n",
        "# yts_torch = ...  (use dtype=torch.int32)\n",
        "\n",
        "# TODO:  Create a TensorDataset\n",
        "#   train_dataset = TensorDataset(...)\n",
        "#   test_dataset = TensorDataset(...)\n",
        "\n",
        "# TODO:  Create a DataLoader\n",
        "#    train_loader = ...\n",
        "#    test_loader = ...\n",
        "batch_size = 100\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0NTMruyyHTSE"
      },
      "source": [
        "Create the loss and an optimizer.  For the loss, use `nn.CrossEntropyLoss()`.  For the optimizer, use the `optim.Adam` optimizer with a learning rate of 0.001"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "GvLDuO73HTSE"
      },
      "outputs": [],
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "# TODO: Define loss function and optimizer\n",
        "#   criterion = ...\n",
        "#   optimizer = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YNGNb5JSHTSF"
      },
      "source": [
        "Fit the model for 20 epochs using the scaled data for both the training and validation.  In each epoch, save the:\n",
        "*  average training loss\n",
        "*  training accuracy\n",
        "*  test accuracy\n",
        "\n",
        "Your final test accuracy should be >95%."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "mzku_NNCHTSF"
      },
      "outputs": [],
      "source": [
        "# Lists to store training and testing accuracy for each epoch\n",
        "train_accuracy_history = []\n",
        "test_accuracy_history = []\n",
        "loss_history = []\n",
        "\n",
        "# Training loop\n",
        "epochs = 20\n",
        "\n",
        "# Training loop\n",
        "for epoch in range(epochs):\n",
        "  # TODO:  Train over epoch\n",
        "  #   model.train()\n",
        "  #   for (data,target) in train_loader:\n",
        "  #       ...\n",
        "  #   loss_history.append(...)\n",
        "  #   train_accuracy_history.append(...)\n",
        "\n",
        "  # TODO:  Test over epoch\n",
        "  #   model.eval()\n",
        "  #   for (data,target) in test_loader:\n",
        "  #       ...\n",
        "  #   test_accuracy_history.append(...)\n",
        "\n",
        "\n",
        "  # TODO:  Print accuracy\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjl2SyG_HTSF"
      },
      "source": [
        "Plot the validation accuracy saved in `test_accuracy_history`.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0mX0388eHTSF"
      },
      "outputs": [],
      "source": [
        "# TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI3UcDtnHTSF"
      },
      "source": [
        "Plot the loss values saved in the `loss_history`.  You should see that the loss is steadily decreasing.  Use the `semilogy` plot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d_FNTVkVHTSF"
      },
      "outputs": [],
      "source": [
        "# TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELxIUxC6HTSF"
      },
      "source": [
        "## Optimizing the Learning Rate\n",
        "\n",
        "One challenge in training neural networks is the selection of the learning rate.  Rerun the above code, trying four learning rates as shown in the vector `rates`.  For each learning rate:\n",
        "* construct the network\n",
        "* select the optimizer.  Use the Adam optimizer with the appropriate learrning rate.\n",
        "* train the model for 20 epochs\n",
        "* Save the accuracy and losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "keOtl2W1HTSF"
      },
      "outputs": [],
      "source": [
        "rates = [1e-4, 1e-3, 1e-2]\n",
        "test_acc_lr = []\n",
        "loss_hist_lr = []\n",
        "\n",
        "# TODO\n",
        "for lr in rates:\n",
        "\n",
        "    # TODO:  Create the model\n",
        "    # model = Model(...)\n",
        "    model = Model(nfeatures, nh, nclasses)\n",
        "\n",
        "\n",
        "    # TODO: Define loss function and optimizer\n",
        "    #   criterion = ...\n",
        "    #   optimizer = ...\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "\n",
        "    # Lists to store training and testing accuracy for each epoch\n",
        "    train_accuracy_history = []\n",
        "    test_accuracy_history = []\n",
        "    loss_history = []\n",
        "\n",
        "    # Number of epochs\n",
        "    epochs = 20\n",
        "\n",
        "    # Training loop\n",
        "    for epoch in range(epochs):\n",
        "\n",
        "      # TODO:  Train over epoch\n",
        "      #   model.train()\n",
        "      #   for (data,target) in train_loader:\n",
        "      #       ...\n",
        "      #   loss_history.append(...)\n",
        "      #   train_accuracy_history.append(...)\n",
        "\n",
        "      # TODO:  Test over epoch\n",
        "      #   model.eval()\n",
        "      #   for (data,target) in test_loader:\n",
        "      #       ...\n",
        "      #   test_accuracy_history.append(...)\n",
        "\n",
        "\n",
        "      # TODO:  Print accuracy\n",
        "\n",
        "      # TODO:  Save the loss history\n",
        "      #   loss_hist_lr.append(...)\n",
        "      #   test_acc_hist_lr.append(...)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mVc6RqwxHTSF"
      },
      "source": [
        "Plot the loss funciton vs. the epoch number for all three learning rates on one graph.  You should see that the lower learning rates are more stable, but converge slower.  The higher learning rate either plateaus at a lower validation accuracy or is unstable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Or9kQZaHTSF"
      },
      "outputs": [],
      "source": [
        "# TODO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6TG9kOI8HTSF"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4KWDUSwEHTSG"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}