FairVIC_Code.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myeA3QRk9I3N"
      },
      "source": [
        "# FairVIC Supplementary Code\n",
        "In this notebook you can find all of the supplementary code behind our method, FairVIC."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EfZcEn5o9XTh"
      },
      "source": [
        "## Misc (Imports, Misc. functions, preprocessing)\n",
        "This section is full of various imports, generic functions, or preprocessing steps for the data that is used. It is unimportant but skim through if you would like."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "io-pDBGv_T_H"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Shw9dLuK8hS3"
      },
      "outputs": [],
      "source": [
        "!pip install ucimlrepo\n",
        "!pip install aif360"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VOzWjMQx_Ob7"
      },
      "outputs": [],
      "source": [
        "# initialisations\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "sns.set(\n",
        "    palette=\"Paired\",\n",
        "    #style='whitegrid',\n",
        "    color_codes=True,\n",
        "    rc={\"figure.figsize\": (12,8)}\n",
        ")\n",
        "\n",
        "# fairness metrics\n",
        "from aif360.datasets import BinaryLabelDataset\n",
        "from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric\n",
        "\n",
        "# model building\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.metrics import confusion_matrix, accuracy_score, roc_auc_score, precision_score, recall_score, f1_score\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Dropout, BatchNormalization, Activation, Dropout\n",
        "from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.regularizers import L2\n",
        "from tensorflow.keras.metrics import AUC, Precision, Recall, F1Score\n",
        "from tensorflow.keras.losses import binary_crossentropy\n",
        "\n",
        "# save results\n",
        "import pickle\n",
        "\n",
        "# suppress warnings\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "import logging\n",
        "logging.getLogger('matplotlib.font_manager').disabled = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__moCeRi_V1g"
      },
      "source": [
        "### Import the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "301zwZru_PDz"
      },
      "outputs": [],
      "source": [
        "from ucimlrepo import fetch_ucirepo\n",
        "\n",
        "# fetch dataset\n",
        "adult = fetch_ucirepo(id=2)\n",
        "\n",
        "# data (as pandas dataframes)\n",
        "X = adult.data.features\n",
        "y = adult.data.targets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SZrvsERY_Pv7"
      },
      "outputs": [],
      "source": [
        "# check to see if the data has imported correctly\n",
        "X.head(4)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ubovq8Gf_P8i"
      },
      "outputs": [],
      "source": [
        "# sort the target variables to be correct\n",
        "y['income'] = y['income'].replace({'<=50K.': '<=50K', '>50K.': '>50K'})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4pIop-MT_1kd"
      },
      "source": [
        "### Preprocessing data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KSmMTxxG_QIy"
      },
      "outputs": [],
      "source": [
        "# feature selection\n",
        "X = X[['age', 'workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country']]\n",
        "X.dtypes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oE5ogWlQ_QYj"
      },
      "outputs": [],
      "source": [
        "X['workclass'] = X['workclass'].astype('category').cat.codes\n",
        "X['education'] = X['education'].astype('category').cat.codes\n",
        "X['marital-status'] = X['marital-status'].astype('category').cat.codes\n",
        "X['occupation'] = X['occupation'].astype('category').cat.codes\n",
        "X['relationship'] = X['relationship'].astype('category').cat.codes\n",
        "X['race'] = X['race'].astype('category').cat.codes\n",
        "X['sex'] = X['sex'].astype('category').cat.codes\n",
        "X['native-country'] = X['native-country'].astype('category').cat.codes\n",
        "y['income'] = y['income'].astype('category').cat.codes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MV0B0vJY_QkC"
      },
      "outputs": [],
      "source": [
        "X = X.astype('float32')\n",
        "y = y.astype('float32')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RZYXDI4P_QvZ"
      },
      "outputs": [],
      "source": [
        "y = y['income']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H8mEVyTn9rDQ"
      },
      "source": [
        "## FairVIC\n",
        "Here you will find the code behind our method, FairVIC, please read throught the comments for explanations 😀"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afs0qc8KC-9Z"
      },
      "source": [
        "### FairVIC class\n",
        "Here is all the code, in one class, for training a custom neural network on our new loss function FairVIC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DKZj9W9y9pa_"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "Our fairness loss function, FairVIC\n",
        "Returns:\n",
        "  trained model -> a trained model that aims to be as fair as possible\n",
        "\"\"\"\n",
        "class FairModel():\n",
        "  def __init__(self):\n",
        "    self.model = self.create_model()\n",
        "    self.protected_attribute = None\n",
        "    self.privileged_groups = None\n",
        "    self.unprivileged_groups = None\n",
        "    self.favorable_label = None\n",
        "    self.unfavorable_label = None\n",
        "\n",
        "  def configure_fairness_evaluation(self, protected_attribute, privileged_groups, unprivileged_groups, favorable_label, unfavorable_label):\n",
        "    self.protected_attribute = protected_attribute\n",
        "    self.privileged_groups = privileged_groups\n",
        "    self.unprivileged_groups = unprivileged_groups\n",
        "    self.favorable_label = favorable_label\n",
        "    self.unfavorable_label = unfavorable_label\n",
        "\n",
        "  def fairvic_wrapper(self, protected_attribute, lambda_binary, lambda_cov, lambda_var, lambda_inv, eps=1e-4):\n",
        "      def fairvic_loss(y_true, y_pred):\n",
        "        y_pred_squeezed = tf.squeeze(y_pred, axis=1)\n",
        "\n",
        "        binary_loss = binary_crossentropy(y_true, y_pred_squeezed)\n",
        "\n",
        "        protected = tf.reshape(protected_attribute, (-1, 1))\n",
        "\n",
        "        variance_loss = tf.reduce_mean(tf.maximum(0., 1. - tf.math.sqrt(tf.reduce_mean(tf.square(protected - tf.reduce_mean(protected, axis=0)), axis=0) + eps)))\n",
        "\n",
        "        invariance_loss = tf.reduce_mean(tf.square(protected - tf.reduce_mean(protected, axis=0)))\n",
        "\n",
        "        y_pred_reshaped = tf.reshape(y_pred, (-1, 1))\n",
        "        cov_matrix = tf.matmul(tf.transpose(y_pred_reshaped - tf.reduce_mean(y_pred_reshaped, axis=0)), protected)\n",
        "        covariance_loss = tf.sqrt(tf.reduce_sum(tf.square(cov_matrix))) / tf.cast(tf.shape(y_pred_reshaped)[0], tf.float32)\n",
        "\n",
        "        total_loss = (lambda_binary * binary_loss +\n",
        "                      lambda_var * variance_loss +\n",
        "                      lambda_inv * invariance_loss +\n",
        "                      lambda_cov * covariance_loss)\n",
        "\n",
        "        return total_loss\n",
        "      return fairvic_loss\n",
        "\n",
        "  def create_model(self):\n",
        "    model = Sequential([\n",
        "        Dense(256, input_shape=(X_train.shape[1],), kernel_regularizer=L2(1e-4), kernel_initializer='he_normal'),\n",
        "        BatchNormalization(),\n",
        "        Activation('relu'),\n",
        "        Dropout(0.25),\n",
        "\n",
        "        Dense(128, kernel_regularizer=L2(1e-4), kernel_initializer='he_normal'),\n",
        "        BatchNormalization(),\n",
        "        Activation('relu'),\n",
        "        Dropout(0.25),\n",
        "\n",
        "        Dense(64, kernel_regularizer=L2(1e-4), kernel_initializer='he_normal'),\n",
        "        BatchNormalization(),\n",
        "        Activation('relu'),\n",
        "        Dropout(0.25),\n",
        "\n",
        "        Dense(32, kernel_regularizer=L2(1e-4), kernel_initializer='he_normal'),\n",
        "        BatchNormalization(),\n",
        "        Activation('relu'),\n",
        "\n",
        "        Dense(1, activation='sigmoid')\n",
        "    ])\n",
        "    return model\n",
        "\n",
        "  @tf.function\n",
        "  def train_step(self, inputs, labels, protected, optimizer, lambda_binary, lambda_cov, lambda_var, lambda_inv):\n",
        "      with tf.GradientTape() as tape:\n",
        "          predictions = self.model(inputs, training=True)\n",
        "          loss = self.fairvic_wrapper(protected, lambda_binary, lambda_cov, lambda_var, lambda_inv)(labels, predictions)\n",
        "      gradients = tape.gradient(loss, self.model.trainable_variables)\n",
        "      optimizer.apply_gradients(zip(gradients, self.model.trainable_variables))\n",
        "      return loss\n",
        "\n",
        "  @tf.function\n",
        "  def val_step(self, inputs, labels, protected, lambda_binary, lambda_cov, lambda_var, lambda_inv):\n",
        "      predictions = self.model(inputs, training=False)\n",
        "      loss = self.fairvic_wrapper(protected, lambda_binary, lambda_cov, lambda_var, lambda_inv)(labels, predictions)\n",
        "      return loss\n",
        "\n",
        "\n",
        "  def train(self, epochs, batch_size, X_train, y_train, X_train_protected, X_val, y_val, X_val_protected, lambda_binary, lambda_cov, lambda_var, lambda_inv):\n",
        "      train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train, X_train_protected)).batch(batch_size)\n",
        "      val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val, X_val_protected)).batch(batch_size)\n",
        "\n",
        "      optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)\n",
        "      best_val_loss = np.inf\n",
        "      epoch_train_losses = []\n",
        "      epoch_val_losses = []\n",
        "\n",
        "      for epoch in range(epochs):\n",
        "          print(f\"Epoch {epoch+1}/{epochs}\")\n",
        "          train_losses_batch = []\n",
        "          for batch, (inputs, labels, protected) in enumerate(train_dataset):\n",
        "              loss = self.train_step(inputs, labels, protected, optimizer, lambda_binary, lambda_cov, lambda_var, lambda_inv)\n",
        "              train_losses_batch.append(loss.numpy())\n",
        "          train_loss = np.mean(train_losses_batch)\n",
        "\n",
        "          val_losses_batch = []\n",
        "          for batch, (inputs, labels, protected) in enumerate(val_dataset):\n",
        "              loss = self.val_step(inputs, labels, protected, lambda_binary, lambda_cov, lambda_var, lambda_inv)\n",
        "              val_losses_batch.append(loss.numpy())\n",
        "          val_loss = np.mean(val_losses_batch)\n",
        "\n",
        "          print(f\"Epoch {epoch+1}, Loss: {train_loss}, Val Loss: {val_loss}, Lr: {optimizer.learning_rate.numpy()}\")\n",
        "          epoch_train_losses.append(train_loss)\n",
        "          epoch_val_losses.append(val_loss)\n",
        "\n",
        "      self.plot_losses(epoch_train_losses, epoch_val_losses)\n",
        "\n",
        "  # plot the loss history for sanity checks\n",
        "  def plot_losses(self, train_losses, val_losses):\n",
        "    plt.title('Learning Curves')\n",
        "    plt.xlabel('Epoch')\n",
        "    plt.ylabel('Loss')\n",
        "    plt.plot(train_losses, label='train')\n",
        "    plt.plot(val_losses, label='val')\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "  # evaluate the trained model for accuracy metrics\n",
        "  def evaluate(self, X_test, y_test):\n",
        "    predictions = self.model.predict(X_test)\n",
        "    predictions = (predictions > 0.5).astype(int)\n",
        "\n",
        "    acc = accuracy_score(y_test, predictions)\n",
        "    auc = roc_auc_score(y_test, predictions)\n",
        "    precision = precision_score(y_test, predictions)\n",
        "    recall = recall_score(y_test, predictions)\n",
        "    f1 = f1_score(y_test, predictions)\n",
        "\n",
        "    return {'acc': acc, 'auc': auc, 'precision': precision, 'recall': recall, 'f1': f1, 'y_test': y_test, 'y_pred': predictions}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E94F1E9NDCXz"
      },
      "source": [
        "### Using FairVIC\n",
        "Please follow the walkthrough below on how to use FairVIC"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P-DaIHCgDXG3"
      },
      "source": [
        "Firstly, we want to split out data in training, validation, and test datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2dNFC3xiDQO2"
      },
      "outputs": [],
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiCJerVSDfut"
      },
      "source": [
        "Next, we want to extract the pritected characteristic column from the dataset. In this psuedo-example, we extract only the 'sex' column"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L0_Dhe4BDQHV"
      },
      "outputs": [],
      "source": [
        "protected_attribute_column_name = 'sex'\n",
        "X_train_protected = X.loc[X_train.index, protected_attribute_column_name]\n",
        "X_test_protected = X.loc[X_test.index, protected_attribute_column_name]\n",
        "X_val_protected = X.loc[X_val.index, protected_attribute_column_name]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hc-ChxmSDqjH"
      },
      "source": [
        "We then can do some standard scaling of the data which is typical for most neural net training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uqggUB04DoeA"
      },
      "outputs": [],
      "source": [
        "scaler = MinMaxScaler()\n",
        "X_train = scaler.fit_transform(X_train)\n",
        "X_test = scaler.transform(X_test)\n",
        "X_val = scaler.transform(X_val)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3l4jGP0kD3Nw"
      },
      "source": [
        "Next, we need to define the model. Here we can just call a new instance of the class and it will instantiate a new neural network that has been created using the 'create_model()' function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KEpLjAJRDoVJ"
      },
      "outputs": [],
      "source": [
        "fair_model = FairModel()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbjNvwvVEPlY"
      },
      "source": [
        "Now we have everything we need to train the model. In the class you can change the weights of the loss function multiplers by tweaking the\n",
        "\n",
        "```\n",
        "lambda_binary=0.1, lambda_cov=0.1, lambda_var=0.1, lambda_inv=0.7\n",
        "```\n",
        "variables. 'lambda_binary' in this case is simply our accuracy loss that is used for this example, this can be replaced by any other accuracy loss function. We will use 200 training epochs and a batch size of 256 right now.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lambda_binary = 0.1\n",
        "lambda_var = 0.1\n",
        "lambda_inv = 0.1\n",
        "lambda_cov = 0.7"
      ],
      "metadata": {
        "id": "-0qL8q9qlL_x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cB0Mt76LDoP2"
      },
      "outputs": [],
      "source": [
        "fair_model.train(200, 256,\n",
        "                 X_train, y_train, X_train_protected, X_val, y_val, X_val_protected,\n",
        "                 lambda_binary=lambda_binary, lambda_cov=lambda_cov,\n",
        "                 lambda_var=lambda_var, lambda_inv=lambda_inv)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zHZDp2vZE-OG"
      },
      "source": [
        "To evaluate the model's performance in standard accuracy metrics, we have provided a function below to do so. This can obviously be changed to allow for any custom metrics needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E9svpEecEwxb"
      },
      "outputs": [],
      "source": [
        "eval_metrics = fair_model.evaluate(X_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6JhF-Z26FQaF"
      },
      "source": [
        "To evaluate the model's performance in fairness metrics, we use this little code snippet below. All that is needed is to specifiy the 'protected attribute' as well as the 'priviledged' and 'unprivileged' groups."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fKF7qDHlUlum"
      },
      "outputs": [],
      "source": [
        "X_test = pd.DataFrame(data=X_test, columns=['age', 'workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country'])\n",
        "predictions = eval_metrics['y_pred']\n",
        "\n",
        "test_df = pd.concat([X_test.reset_index(drop=True), y_test.reset_index(drop=True)], axis=1)\n",
        "test_df.columns = list(X_test.columns) + ['label']\n",
        "\n",
        "test_df = test_df.astype(int)\n",
        "\n",
        "test_bld = BinaryLabelDataset(df=test_df,\n",
        "                              label_names=['label'],\n",
        "                              protected_attribute_names=['sex'],\n",
        "                              favorable_label=1,\n",
        "                              unfavorable_label=0)\n",
        "\n",
        "predictions_bld = test_bld.copy()\n",
        "predictions_bld.labels = predictions.reshape(-1, 1)\n",
        "\n",
        "classification_metric = ClassificationMetric(test_bld,\n",
        "                                              predictions_bld,\n",
        "                                              unprivileged_groups=[{'sex': 0}],\n",
        "                                              privileged_groups=[{'sex': 1}])\n",
        "\n",
        "fairness_metrics = {\n",
        "        'equalized_odds_difference': classification_metric.equalized_odds_difference(),\n",
        "        'average_abs_odds_difference': classification_metric.average_abs_odds_difference(),\n",
        "        'disparate_impact': classification_metric.disparate_impact(),\n",
        "        'demographic_parity_difference': classification_metric.mean_difference(),\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oAhLhx2RHxmW"
      },
      "source": [
        "Finally, to see our results for both accuracy and fairness, we print every metric here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9cmwNYp_Fd9U"
      },
      "outputs": [],
      "source": [
        "print(f\"Results\")\n",
        "print(\"============================================================================================\")\n",
        "print(f\"Accuracy: {eval_metrics['acc']}\")\n",
        "print(f\"AUC: {eval_metrics['auc']}\")\n",
        "print(f\"Precision: {eval_metrics['precision']}\")\n",
        "print(f\"Recall: {eval_metrics['recall']}\")\n",
        "print(f\"F1 Score: {eval_metrics['f1']}\")\n",
        "print(\"---------------------------------------------\")\n",
        "print(\"Fairness Metrics:\")\n",
        "print(f\"Equalized Odds Difference: {fairness_metrics['equalized_odds_difference']}\")\n",
        "print(f\"Average Absolute Odds Difference: {fairness_metrics['average_abs_odds_difference']}\")\n",
        "print(f\"Disparate Impact: {fairness_metrics['disparate_impact']}\")\n",
        "print(f\"Demographic Parity (Mean) Difference: {fairness_metrics['demographic_parity_difference']}\")\n",
        "print(\"============================================================================================\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "auShRw0MH5Al"
      },
      "source": [
        "That is how you can incorporate FairVIC into a neural network's training. It is elegant yet efficient. We encourage you to mess around with the weights! 😀"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "io-pDBGv_T_H",
        "__moCeRi_V1g",
        "4pIop-MT_1kd"
      ],
      "gpuType": "L4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}