GX Community Demo - 2024-08-28.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e2d37938",
   "metadata": {},
   "source": [
    "# Welcome to GX Core 1.0!\n",
    "\n",
    "\n",
    "### Install the Great Expectations package\n",
    "\n",
    "Start by installing the Great Expectations package using a pip command. If you have specific dependencies, you can specify them here. Learn more about additional dependencies in our [docs](https://docs.greatexpectations.io/docs/core/set_up_a_gx_environment/install_additional_dependencies). In this example, we'll be using Postgres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e6c5a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install 'great_expectations[sqlalchemy]'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7649a7e",
   "metadata": {},
   "source": [
    "Import the `great_expectations` and `expectations` modules and instantiate your Data Context. In 1.0, Expectations are top-level classes namespaced to `gxe`. A Data Context defines the storage location for metadata, such as your configurations for Data Sources, Expectation Suites, Checkpoints, and Data Docs. It also contains your Validation Results and the metrics associated with them, and it provides access to those objects in Python, along with other helper functions for the GX Python API. Learn more [here](https://docs.greatexpectations.io/docs/core/set_up_a_gx_environment/create_a_data_context)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2836172e-4948-4fb7-bf5b-7b44d76d7da3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import great_expectations as gx\n",
    "import great_expectations.expectations as gxe\n",
    "\n",
    "context = gx.get_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3d97f3e",
   "metadata": {},
   "source": [
    "### Connect to Data\n",
    "\n",
    "Create your Data Source and Data Asset. In this example, we'll use a publicly available Postgres data source that GX has set up for all to use and test with. It contains contains New York City (NYC) taxi data from January 2019. The [NYC Taxi data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) is a popular set of open source data that contains records of completed taxi cab trips in NYC, including information such as pick up and drop off times, the number of passengers, the fare collected, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba3019e-a441-41df-a661-dd263d5b0d29",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = context.data_sources.add_postgres(\"Data Source\", connection_string=\"postgresql+psycopg2://example_user:workshop_example_password@postgres.workshops.greatexpectations.io/gx_example_db\")\n",
    "asset = ds.add_table_asset(name=\"Data Asset\", table_name=\"nyc_taxi_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60013f5c",
   "metadata": {},
   "source": [
    "A new concept (that we'll touch on in greater detail later with partitioning) is the \"Batch Definition\", which describes what data will be validated in each run. In this example, we'll use our whole asset with the \"whole table\" Batch Definition. We'll use our Batch Definition to get a concrete Batch we can validate expectations against."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e945041-25b5-4c9a-a9d1-c385084e7d56",
   "metadata": {},
   "outputs": [],
   "source": [
    "bd = asset.add_batch_definition_whole_table(\"Batch Definition\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e339c041",
   "metadata": {},
   "source": [
    "### Create an Expectation \n",
    "\n",
    "Let's create our first gx 1.0 Expectation!\n",
    "Expectation classes are exposed directly in gx 1.0, and are statically typed using [Pydantic](https://docs.pydantic.dev/latest/). Inellisense will show valid arguments and their types. In this example, we'll create an Expectation that says that our `passenger_count` column should never be greater than 4, since taxis in NYC only have 4 seats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39f47e29",
   "metadata": {},
   "outputs": [],
   "source": [
    "expectation = gxe.ExpectColumnMaxToBeBetween(column=\"passenger_count\", max_value=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "920faf9e",
   "metadata": {},
   "source": [
    "### Run a validation\n",
    "\n",
    "Now that we have completed the configuration, we can validate our data! All we have to do is get the specific batch of data using our Batch Definition and then calling the `batch()` method and passing in our newly-created Expectation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e991f723",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = bd.get_batch()\n",
    "print(batch.validate(expectation))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84daa48",
   "metadata": {},
   "source": [
    "### Edit the Expectation \n",
    "\n",
    "You'll see that while our validation executed as expected, the expectation itself failed. This is because some taxis in NYC are minivans and can hold up to 6 passengers. In fact, the results show that there are observed values of 6. Fortunately, editing the expectation is quite simple. After we do so, we can re-run the validation and see that it is now passing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1a1c2d-4559-46c9-b3ad-345f3230282b",
   "metadata": {},
   "outputs": [],
   "source": [
    "expectation.max_value = 6\n",
    "print(batch.validate(expectation))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "820d0dcc",
   "metadata": {},
   "source": [
    "### Create an Expectation Suite\n",
    "\n",
    "So far we have run a single Expectation on a batch of data. However, there may often be times where you will want run a Suite of Expectations. Doing so is simple. Create an Expectation Suite and then add the Expectations you wish to run. Finally, run the `validate` method on your batch, passing in the Suite instead of the single Expectation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9971d158-39d1-42c0-a8f4-a70c711ef30b",
   "metadata": {},
   "outputs": [],
   "source": [
    "suite = context.suites.add(gx.ExpectationSuite(name=\"Expectation Suite\"))\n",
    "suite.add_expectation(expectation)\n",
    "suite.add_expectation(gxe.ExpectColumnMinToBeBetween(column=\"passenger_count\", min_value=1))\n",
    "\n",
    "print(batch.validate(suite))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18e21013",
   "metadata": {},
   "source": [
    "### Create a Validation Definition\n",
    "\n",
    "Validation definitions are a new, greatly simplified, concept in GX 1.0. They represent an explicit way to tie data, via a Batch Definition, to the expectation on that data. These are at the center of the reworked Checkpoints API. Creating a Validation Definition is simple. Give it a name, link it to the batch definition defined above, and pass in the suite of expectations we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b33e7d-26ad-4990-8087-8488ba63089b",
   "metadata": {},
   "outputs": [],
   "source": [
    "vd = gx.ValidationDefinition(\n",
    "    name=\"Validation Definition\",\n",
    "    data=bd,\n",
    "    suite=suite\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d64489",
   "metadata": {},
   "source": [
    "You can simply run the validation definition as shown below, or you can use it in a Checkpoint, which we will see next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64309125-53ad-4088-80bb-9ae31588a707",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vd.run())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd932d8c",
   "metadata": {},
   "source": [
    "### Create and run a Checkpoint\n",
    "\n",
    "Validation Definitions can be tied to actions, such as slack notifications and building data docs, via Checkpoints. Checkpoints also provide an interface to run multiple Validation Definitions together. In this example, we'll use the validation definition created above and trigger the `UpdateDataDocsAction`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48e29381-d5df-4ca3-8f91-d7360b899b2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from great_expectations.checkpoint.actions import UpdateDataDocsAction\n",
    "\n",
    "checkpoint = context.checkpoints.add(gx.Checkpoint(\n",
    "    name=\"Checkpoint\",\n",
    "    validation_definitions=[\n",
    "        vd\n",
    "    ],\n",
    "    actions = [\n",
    "        UpdateDataDocsAction(name=\"update_data_docs\")\n",
    "    ]\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cb3dd38",
   "metadata": {},
   "source": [
    "We can now run the Checkpoint and open the data docs to see our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c756273c-bd69-418a-83c5-10517b503919",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint.run()\n",
    "context.open_data_docs()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f00e644",
   "metadata": {},
   "source": [
    "### Create monthly batch definitions\n",
    "\n",
    "We previously created \"whole table\" Batch Definitions. Now, we'll create monthly batch definitions. When checkpoints use these, they will run expectation suites against the last batch defined by the Batch Definition. We'll create a new Validation Definition since this Batch Definition is new, and then run it. By default, running this validation will use the latest set of data available within the Data Asset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "415c73e1-e916-4634-a147-2382980e8c71",
   "metadata": {},
   "outputs": [],
   "source": [
    "monthlybd = asset.add_batch_definition_monthly(\"Monthly BD\", column=\"pickup\")\n",
    "vd = gx.ValidationDefinition(name=\"Monthly VD\", data=monthlybd, suite=suite)\n",
    "\n",
    "print(vd.run())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adec8811",
   "metadata": {},
   "source": [
    "Alternatively, you can specify which month of data you wish to use by including `batch_parameters` in the `run` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea09a3a1-d0b8-47cc-beed-ca9e13b33b83",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vd.run(batch_parameters={\"year\": 2019, \"month\":1}))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}