Merge pull request #18 from ruivieira/main

Add documentation for data drift integration tutorial

docs/modules/ROOT/nav.adoc

@@ -7,8 +7,9 @@
 ** xref:installing-opendatahub.adoc[]
 ** xref:bias-monitoring.adoc[]
 ** xref:data-drift-monitoring.adoc[]
+** xref:accessing-service-from-python.adoc[]
 * Components
 ** xref:trustyai-service.adoc[]
 ** xref:trustyai-operator.adoc[]
 ** xref:python-trustyai.adoc[]
-** xref:trustyai-core.adoc[]
+** xref:trustyai-core.adoc[]

docs/modules/ROOT/pages/accessing-service-from-python.adoc

@@ -0,0 +1,267 @@
+= Accessing Service from Python
+
+Data drift occurs when a machine model's performance declines or is different on unseen data compared to its training data due distribution changes in the data over time. In this notebook, we explore and visualize data drift on a simple XGBoost model, that predicts credit card acceptance based on an applicant's age, credit score, years of education, and years in employment. This tutorial is a counterpart to the xref:data-drift-monitoring.adoc[data drift monitoring tutorial].
+
+== Prerequisites
+
+Follow the instructions within the xref:installing-opendatahub.adoc[Installing Open Data Hub] section. Additionally, follow the instructions in the xref:data-drift-monitoring.adoc#deploy-model[Deploy Model] section in the Data Drift tutorial. Before proceeding, check that you have the following:
+
+. ODH installation
+. A TrustyAI Operator
+. A model-namespace project containing an instance of the TrustyAI Service
+. A model storage container
+. A Seldon MLServer serving runtime
+. The delpoyed credit model
+
+== Imports
+
+[source,python]
+----
+import os
+import subprocess
+import warnings
+warnings.filterwarnings("ignore")
+
+import matplotlib.pyplot as plt
+
+from trustyai.utils.api.api import TrustyAIApi
+from trustyai.utils.extras.metrics_service import TrustyAIMetricsService
+----
+
+== Clone the Data Drift Repository
+
+For the purposes of recreating the data drift demo, we will be reusing the data in that repository.
+
+[source,shell]
+----
+git clone https://github.com/trustyai-explainability/odh-trustyai-demos.git
+cd odh-trustyai-demos/3-DataDrift
+----
+
+== Initialize Metrics Service
+
+In order to use the metrics service, we first have to initialize it using our OpenShift login token and model namespace.
+
+[source,python]
+----
+TOKEN = os.environ.get("TOKEN", "None")
+trustyService = TrustyAIMetricsService(
+    token = TOKEN,
+    namespace="model-namespace",
+    verify=False
+)
+----
+
+== Upload Model Training Data To TrustyAI
+
+[source,python]
+----
+trustyService.upload_payload_data(
+    json_file="data/training_data.json"
+)
+----
+
+== Label Data Fields
+
+[source,python]
+----
+name_mapping = {
+    "modelId": "gaussian-credit-model",
+    "inputMapping":
+      {
+        "credit_inputs-0": "Age",
+        "credit_inputs-1": "Credit Score",
+        "credit_inputs-2": "Years of Education",
+        "credit_inputs-3": "Years of Employment"
+      },
+    "outputMapping": {
+      "predict-0": "Acceptance Probability"
+    }
+}
+
+trustyService.label_data_fields(payload=name_mapping)
+----
+
+== Examining TrustyAI's Model Metadata
+
+[source,python]
+----
+trustyService.get_model_metadata()
+----
+
+[source,text]
+----
+
+
+[{'metrics': {'scheduledMetadata': {'metricCounts': {}}},
+  'data': {'inputSchema': {'items': {'credit_inputs-2': {'type': 'DOUBLE',
+      'name': 'credit_inputs-2',
+      'values': None,
+      'index': 2},
+     'credit_inputs-3': {'type': 'DOUBLE',
+      'name': 'credit_inputs-3',
+      'values': None,
+      'index': 3},
+     'credit_inputs-0': {'type': 'DOUBLE',
+      'name': 'credit_inputs-0',
+      'values': None,
+      'index': 0},
+     'credit_inputs-1': {'type': 'DOUBLE',
+      'name': 'credit_inputs-1',
+      'values': None,
+      'index': 1}},
+    'remapCount': 2,
+    'nameMapping': {'credit_inputs-0': 'Age',
+     'credit_inputs-1': 'Credit Score',
+     'credit_inputs-2': 'Years of Education',
+     'credit_inputs-3': 'Years of Employment'},
+    'nameMappedItems': {'Years of Education': {'type': 'DOUBLE',
+      'name': 'credit_inputs-2',
+      'values': None,
+      'index': 2},
+     'Age': {'type': 'DOUBLE',
+      'name': 'credit_inputs-0',
+      'values': None,
+      'index': 0},
+     'Years of Employment': {'type': 'DOUBLE',
+      'name': 'credit_inputs-3',
+      'values': None,
+      'index': 3},
+     'Credit Score': {'type': 'DOUBLE',
+      'name': 'credit_inputs-1',
+      'values': None,
+      'index': 1}}},
+   'outputSchema': {'items': {'predict-0': {'type': 'FLOAT',
+      'name': 'predict-0',
+      'values': None,
+      'index': 4}},
+    'remapCount': 2,
+    'nameMapping': {'predict-0': 'Acceptance Probability'},
+    'nameMappedItems': {'Acceptance Probability': {'type': 'FLOAT',
+      'name': 'predict-0',
+      'values': None,
+      'index': 4}}},
+   'observations': 1000,
+   'modelId': 'gaussian-credit-model'}}
+----
+
+== Register Drift Monitoring
+
+[source,python]
+----
+drift_monitoring = {
+    "modelId": "gaussian-credit-model",
+    "referenceTag": "TRAINING"
+}
+trustyService.get_metric_request(
+    payload=drift_monitoring,
+    metric="drift/meanshift", reoccuring=True
+)
+----
+
+[source,text]
+----
+'{"requestId":"709174f5-a3f4-4ae9-8f7e-a56b708836ff","timestamp":"2024-03-06T14:23:17.740+00:00"}'
+----
+
+== Check the Metrics
+
+Let's get the meanshift values for the training data we just uploaded to the TrustyAI service for the past 5 minutes.
+
+[source,python]
+----
+train_df = trustyService.get_metric_data(
+    metric="trustyai_meanshift",
+    time_interval="[5m]"
+)
+display(train_df.head())
+----
+
+[options="header"]
+|===
+| timestamp | Age | Credit Score | Years of Education | Years of Employment
+
+| 2024-03-06 09:23:18 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:22 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:26 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:30 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:34 | 1.0 | 1.0 | 1.0 | 1.0
+|===
+
+Let's also visualize the meanshift in a plot similar to the one displayed in ODH Observe -> Metrics tab. We will define a helper function so that we can use it again for the unseen data.
+
+[source,python]
+----
+def plot_meanshift(df):
+    """
+    :param df: A pandas DataFrame returned by the TrustyAIMetricsService().get_metric_request
+               function with columns corresponding to the timestamp and name of the metric
+    returns a scatterplot with the timestamp on the x-axis and the specific metric on the y-axis
+    """
+    plt.figure(figsize=(12,5))
+    for col in df.columns[1:]:
+        plt.plot(
+            df["timestamp"],
+            df[col]
+        )
+    plt.xlabel("timestamp")
+    plt.ylabel("meanshift")
+    plt.xticks(rotation=45)
+    plt.legend(df.columns[1:])
+    plt.tight_layout()
+    plt.show()
+
+plot_meanshift(train_df)
+----
+
+image::python-service-01.png[Mean Shift plot]
+
+== Collect "Real-World" Inferences
+
+[source,python]
+----
+model_name = "gaussian-credit-model"
+model_route = TrustyAIApi().get_service_route(
+    name=model_name,
+    namespace=trustyService.namespace
+)
+
+for batch in list(range(0, 596, 5)):
+    trustyService.upload_data_to_model(
+        model_route=f"{model_route}/v2/models/gaussian-credit-model",
+        json_file=f"data/data_batches/{batch}.json"
+    )
+----
+
+== Observe Drift
+
+Let's check if our model is behaving differently on the unseen data.
+
+[source,python]
+----
+test_df = trustyService.get_metric_data(
+    metric="trustyai_meanshift",
+    time_interval="[5m]"
+    )
+display(test_df.head())
+----
+
+[options="header"]
+|===
+| timestamp | Age | Credit Score | Years of Education | Years of Employment
+
+| 2024-03-06 09:23:18 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:22 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:26 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:30 | 1.0 | 1.0 | 1.0 | 1.0
+| 2024-03-06 09:23:34 | 1.0 | 1.0 | 1.0 | 1.0
+|===
+
+[source,python]
+----
+plot_meanshift(test_df)
+----
+
+image::python-service-02.png[Mean Shift plot]
+
+As observed, the meanshift values for each of the features have changed drastically from the training to test data, dropping below 1.0. In particular, Age and Credit Score are significantly different according to a p-value of 0.05. Thus, it is clear that our model suffers from data drift.

docs/modules/ROOT/pages/data-drift-monitoring.adoc

@@ -17,6 +17,7 @@ Follow the instructions within the xref:installing-opendatahub.adoc[Installing o
 you should have an ODH installation, a TrustyAI Operator, and a `model-namespace` project containing
 an instance of the TrustyAI Service.
 
+[#deploy-model]
 == Deploy Model
 
 . Navigate to the `model-namespace` created in the setup section:
@@ -97,7 +98,7 @@ include::example$model_gaussian_credit.yaml[]
 +
 . `data_tag`: A string tag to reference this particular set of data. Here, we choose `"TRAINING"`
 . `request`: A https://kserve.github.io/website/0.8/modelserving/inference_api/#inference-request-json-object[KServe Inference Request], as if you were sending this data directly to the model server's `/infer` endpoint. 
-. `response`: (Optionally) the https://kserve.github.io/website/0.8/modelserving/inference_api/#inference-response-json-objectt[KServe Inference Response] that is returned from sending the above request to the model. 
+. `response`: (Optionally) the https://kserve.github.io/website/0.8/modelserving/inference_api/#inference-response-json-objectt[KServe Inference Response] that is returned from sending the above request to the model.
 
 == Examining TrustyAI's Model Metadata
 
@@ -144,7 +145,7 @@ curl -sk -H 'Authorization: Bearer ${TOKEN}' \
 You should see the message `Feature and output name mapping successfully applied.`
 
 The payload of the request is a simple set of `original-name : new-name` pairs, assigning new meaningful names to the input and output
-features of our model. 
+features of our model.
 
 == Register the Drift Monitoring
 
@@ -174,7 +175,7 @@ the reference distribution.
 +
 image::data_drift_meanshift_initial.png[Initial Meanshift Chart]
 +
-. You'll notice that a metric is emitted for each of the four features and the single output, making for five measurements in total. All metric values should equal 1 (no drift), which makes sense: we _only_ have the training data, which can't drift from itself. 
+. You'll notice that a metric is emitted for each of the four features and the single output, making for five measurements in total. All metric values should equal 1 (no drift), which makes sense: we _only_ have the training data, which can't drift from itself.
 
 == Collect "Real-World" Inferences
 
@@ -200,7 +201,7 @@ done
 image::data_drift_meanshift_post.png[Post-deployment metrics]
 
 Navigating back to the Observe -> Metrics page in the OpenShift console, we can see the MeanShift metric
-values for the various features changes. Notably, the values for `Credit Score`, `Age`, and `Acceptance Probability` have all dropped to 0, indicating there is a statistically very high likelihood that the values of these fields in the inference data come from a different distribution than that of the training data. Meanwhile, the `Years of Employment` and `Years of Education` scores have dropped to 0.34 and 0.82 respectively, indicating that there is a little drift, but not enough to be particularly concerning. Remember, the Mean-Shift metric scores are p-values, so only values < 0.05 indicate statistical significance. 
+values for the various features changes. Notably, the values for `Credit Score`, `Age`, and `Acceptance Probability` have all dropped to 0, indicating there is a statistically very high likelihood that the values of these fields in the inference data come from a different distribution than that of the training data. Meanwhile, the `Years of Employment` and `Years of Education` scores have dropped to 0.34 and 0.82 respectively, indicating that there is a little drift, but not enough to be particularly concerning. Remember, the Mean-Shift metric scores are p-values, so only values < 0.05 indicate statistical significance.
 
 == A Peek Behind The Curtain
 To better understand the what these metrics tell us, let's take a look behind the curtain at the actual data I generated for this example, and look at the real distributions of the training and inference
@@ -211,7 +212,7 @@ image::data_drift_gaussian_credit_model_distributions.png[Real Data Distribution
 In red are each features' distributions in the training set, while the blue shows the distribution
 seen in the "real-world" inference data. We can clearly see that the `Age` and `Credit Score` data
 are drawn from two different distributions, while `Years of Education` and `Years of Employment` look
-to be the same distribution, and this exactly aligns with the metric values we observed in the previous section. Naturally, the differing input distributions also cause the output `Acceptance Probability` distribution to shift as well. 
+to be the same distribution, and this exactly aligns with the metric values we observed in the previous section. Naturally, the differing input distributions also cause the output `Acceptance Probability` distribution to shift as well.
 
 == Conclusion
 

docs/modules/ROOT/pages/python-trustyai.adoc

@@ -6,12 +6,20 @@ Python TrustyAI is a Python library for explainable AI. It is a wrapper around t
 
 The library is available on PyPi and can be installed with pip.
 
-```bash
+[source,shell]
+----
 pip install trustyai
-```
+----
 
 To install additional experimental features, the following command can be used:
 
-```bash
+[source,shell]
+----
 pip install trustyai[extras]
-```
+----
+
+== Service integration
+
+The Python TrustyAI library allows for interaction with a running xref:trustyai-service.adoc[TrustyAI service].
+
+For an example, please consult the xref:accessing-service-from-python.adoc[] tutorial.